Dynamic Instrumentation

Overview

Dynamic instrumentation allows you to add instrumentation into your running production systems without any restarts and at any location in your application’s code, including third-party libraries. You can add or modify telemetry for logs, metrics, spans, and corresponding tagging, from the Datadog UI. Dynamic Instrumentation has low overhead and has no side effects on your system.

If you are interested in trying out the latest user experience improvements for Dynamic Instrumentation, consider opting into the Symbol Database private beta.

Getting started

Prerequisites

Dynamic Instrumentation requires the following:

Datadog Agent 7.44.0 or higher is installed alongside your service.
Remote Configuration is enabled in that Agent.
For Java applications, tracing library dd-trace-java 1.24.0 or higher.
For Python applications, tracing library dd-trace-py 2.2.0 or higher.
For .NET applications, tracing library dd-trace-dotnet 2.42.0 or higher.
Unified Service Tagging tags service, env, and version are applied to your deployment.
Optionally, Source Code Integration is set up for your service.
The Dynamic Instrumentation Read Configuration (debugger_read) permission is required to access the Dynamic Instrumentation page
The Dynamic Instrumentation Write Configuration (debugger_write) permission is required to create or modify instrumentations.
The Dynamic Instrumentation Capture Variables (debugger_capture_variables) permission is required to use the Capture method parameters and local variables option.

For more information about roles and on how to assign roles to users, see Role Based Access Control.

Create a logs index

Dynamic Instrumentation creates “dynamic logs” that are sent to Datadog and appear alongside your regular application logs.

If you use Exclusion filters, ensure Dynamic Instrumentation logs are not filtered:

Create a logs index and configure it to the desired retention with no sampling.
Set the filter to match on the source:dd_debugger tag. All Dynamic Instrumentation logs have this source.
Ensure that the new index takes precedence over any other with filters that match that tag, because the first match wins.

Enable Dynamic Instrumentation

To enable Dynamic Instrumentation on a service, go to the in-app setup page.

For more detailed instructions, select your runtime below:

Explore Dynamic Instrumentation

Dynamic Instrumentation can help you understand what your application is doing at runtime. By adding a Dynamic Instrumentation probe you are exporting additional data from your application, without the need to change code or redeploy it.

Using probes

A probe allows you to collect data from specific points in your code without halting the execution of the program.

Think of using probes as enhancing your observability by adding dynamic logs, metrics, and spans to a running application without needing to change code, deploy it, or restart a service. You can gather data immediately without disturbing the user experience or requiring lengthy deployments.

As a developer, you can also think of a probe as a “non-breaking breakpoint”. In traditional debugging, a breakpoint is a point in the program where the execution stops, allowing the developer to inspect the state of the program at that point. However, in real-world production environments, it’s not practical or even possible to stop the execution of the program. Probes fill in this gap by allowing you to inspect variable state in production environments in a non-intrusive way.

Creating a probe

All probe types require the same initial setup:

Go to the Dynamic Instrumentation page.
Click Create Probe in the top right, or click the three-dot menu on a service and select Add a probe for this service.
If they are not prefilled, choose service, runtime, environment, and version.
In the source code, specify where to set the probe by selecting either a class and method or a source file and line. If you set up Source Code Integration for your service, autocomplete shows suggestions for the selecting a file, and displays the file’s code so you can choose the line.

See the individual probe types below for specific creation steps for each probe type.

Alternatively, you can create a probe from these other contexts:

Profiling: On a profiler flame graph, you can create a probe for a method by selecting Instrument this frame with a probe from the frame’s context menu.
Error Tracking: On a stack trace, mouse over a stack frame and click Instrument. This prefills the probe creation form with the Issue context.

Creating log probes

A log probe emits a log when it executes.

To create a log probe:

Select Log as the probe type.
Complete the generic probe setup (choose service, environment, version, and probe location).
Define a log message template. You can use the Dynamic Instrumentation expression language to reference values from the execution context.
Optionally enable extra data capturing from the probe. (Beta)
Optionally define a condition using the Dynamic Instrumentation expression language. The log is emitted when the expression evaluates to true.

Log probes are enabled by default on all service instances that match the specified environment and version. They are rate-limited to execute at most 5000 times per second, on each instance of your service.

You must set a log message template on every log probe. The template supports embedding expressions inside curly brackets. For example: User {user.id} purchased {count(products)} products.

You can also set a condition on a log probe using the expression language. The expression must evaluate to a Boolean. The probe executes if the expression is true, and does not capture or emit any data if the expression is false.

Creating a Dynamic Instrumentation log probe

Beta: If you enable Capture method parameters and local variables on the log probe, all execution context is added the log event:

Method arguments, local variables, and fields, with the following default limits:
- Follow references three levels deep (configurable in the UI).
- The first 100 items inside collections.
- The first 255 characters for string values.
- 20 fields inside objects. Static fields are not collected.
Call stack trace.
Caught and uncaught exceptions.

Probes with this setting enabled are rate-limited to one hit per second.

Warning: The captured data may contain sensitive information, including personal data, passwords, and secrets such as AWS keys.

To ensure this information is properly redacted:

Datadog Dynamic Instrumentation employs several techniques to redact sensitive information. To learn more about the default mechanisms or how to extend the it to meet your needs, read Sensitive Data Scrubbing.
Turn off the Capture method parameters and local variables option and explicitly select the variables you want to include in the log message template. Doing so ensures that log probes contain only data related to the variables that you specifically identify, thus reducing the risk of unintentional sensitive data leaks.
If you are the Administrator of your Datadog account and would like to prevent other users from being able to use the Capture method parameters and local variables option, you can revoke their Dynamic Instrumentation Capture Variables (debugger_capture_variables) permission.

Alternatively, if you need to log this data but want to mitigate the risk associated with it being accessible in the Datadog product, you can limit which users in your organization can view the captured data by setting up a Restriction query on source:dd_debugger.

Creating metric probes

A metric probe emits a metric when it executes.

To create a metric probe:

Select Metric as the probe type.
Complete the generic probe setup (choose service, environment, version, and probe location).
Specify a name for the metric, which will be prefixed with dynamic.instrumentation.metric.probe..
Select a metric type (count, gauge, or histogram).
Choose the value of the metric using the Dynamic Instrumentation expression language. You can use any numeric value you’d like from the execution context, such as a method parameter, local variable, a class field, or an expression that yields a numeric value. For count metrics this is optional, and if you omit it, every invocation increments the count by one.

Creating a Dynamic Instrumentation metric probe

Metric probes are automatically enabled on all service instances that match the configured environment and version. Metric probes are not rate limited and execute every time the method or line is invoked.

Dynamic Instrumentation metric probes support the following metric types:

Count: Counts how many times a given method or line is executed. Can be combined with metric expressions to use the value of a variable to increment the count.
Gauge: Generates a gauge based on the last value of a variable. This metric requires a metric expression.
Histogram: Generates a statistical distribution of a variable. This metric requires a metric expression.

Creating span probes

A span probe emits a span when a method is executed.

To create a span probe:

Select Span as the probe type.
Complete the generic probe setup (choose service, environment, version, and probe location).

Creating a Dynamic Instrumentation span probe

You can use a span probe as an alternative to creating new spans with Custom Instrumentation. If the method throws an exception, the details of the exception are associated with the newly created span’s error tag.

Creating span tag probes

A span tag probe adds a tag value to an existing span. You can add a tag either to the active span or to the service entry span. Keep in mind that internal spans are not indexed by default and so might not be searchable in APM.

To create a span tag probe:

Select Span Tag as the probe type.
Complete the generic probe setup (choose service, environment, version, and probe location).
Specify a name for the tag.
Specify the value of the tag using the Dynamic Instrumentation expression language.
Optionally define a condition using the Dynamic Instrumentation expression language. The tag will only be added when the expression evaluates to true.
Optionally add additional tags, each with their own name, expression, and optional condition.