Overview

Dynamic Instrumentation lets you add metrics, spans, and span tags to running production systems without restarts or code changes, including in third-party libraries.

To use autocomplete and search, opt in to the preview.

Note: Dynamic Instrumentation is not compatible with Azure App Services or serverless environments.

Requirements

Datadog SDK

Datadog configuration

• Install the Datadog Agent, version 7.49.0 or higher.
• Configure Unified Service Tagging with service, env, and version tags.
• Enable Remote Configuration in the Agent.
• (Recommended) Set up Source Code Integration.

Permissions

The following permissions are required to use Dynamic Instrumentation:

• Dynamic Instrumentation Read Configuration (debugger_read): Required to access the Dynamic Instrumentation page.
• One of the following write permissions:
  • Dynamic Instrumentation Write Configuration (debugger_write): Required to create or modify instrumentations in any environment.
  • Dynamic Instrumentation Write Pre-Prod (debugger_write_preprod): Required to create or modify instrumentations in known pre-production environments only (such as staging or QA).

For more information about roles and how to assign roles to users, see Role Based Access Control.

Enable Dynamic Instrumentation

Configure Dynamic Instrumentation

Configure Dynamic Instrumentation using the following environment variables:

Explore Dynamic Instrumentation

Dynamic Instrumentation can help you understand what your application is doing at runtime. By adding an instrumentation at a specific code location, you can capture additional telemetry from your application without the need to change code or redeploy it.

Using instrumentations

An instrumentation allows you to collect additional telemetry from specific points in your code without halting the execution of the program.

Dynamic spans, span tags, and metrics are a UI-based alternative to adding custom instrumentation directly to your source code. Datadog receives the instrumentation configurations you define and dynamically applies them to the running service without requiring restarts.

Datadog captures and processes spans, tags, and metrics generated by Dynamic Instrumentation like other telemetry from the running application. Unlike manual custom instrumentation, Dynamic Instrumentation does not require code changes, deployments, or service restarts. To stop collecting data, disable the instrumentation in Datadog.

To redact sensitive data from instrumentations, see Sensitive data scrubbing.

Creating an instrumentation

All instrumentation types require the same initial setup:

1. Go to the Dynamic Instrumentation page.
2. Click Create Instrumentation in the top right, or click the three-dot menu on a service and select Add an instrumentation for this service.
3. If the fields are not prefilled, choose service, runtime, environment, and version (optional).
4. Specify where to set the instrumentation in the source code by selecting either a class and method, or a specific line of code in a file. When autocomplete and search are available, use them to find files, methods, or symbols.
   Specify where to set the instrumentation in the source code by selecting a class and method. When autocomplete and search are available, use them to find methods or symbols.

For the best experience, set up Source Code Integration to view code directly in Datadog and select instrumentation locations as you would with breakpoints in an IDE.

For creation steps specific to each instrumentation type, see the following sections.

Creating dynamic metrics

A dynamic metric emits a metric when it executes. To create a dynamic metric:

1. Select Metric as the instrumentation type.
2. Complete the generic instrumentation setup (choose service, environment, and location).
3. Specify a name for the metric, which is prefixed with dynamic.instrumentation.metric.probe..
4. Select a metric type (count, gauge, or histogram).
5. Choose the value of the metric using the Dynamic Instrumentation expression language. You can use any numeric value from the execution context, such as a method parameter, local variable, a class field, or an expression that yields a numeric value. For count metrics, this is optional, and if you omit it, every invocation increments the count by one.

Metric instrumentations are automatically enabled on all service instances that match the configured environment and version. Metric instrumentations are not rate limited and execute every time the method

Dynamic Instrumentation supports the following metric types:

• Count: Counts how many times a given method
  or line
  is executed. Can be combined with metric expressions to use the value of a variable to increment the count.
• Gauge: Generates a gauge based on the last value of a variable. This metric requires a metric expression.
• Histogram: Generates a statistical distribution of a variable. This metric requires a metric expression.

Creating dynamic spans

A dynamic span emits a span when a method is executed. To create a dynamic span:

1. Select Span as the instrumentation type.
2. Complete the generic instrumentation setup (choose service, environment, version, and location).

You can use a dynamic span as an alternative to creating spans with Custom Instrumentation. If the method throws an exception, the details of the exception are associated with the newly created span's error tag.

Creating dynamic span tags

A dynamic span tag adds a tag value to an existing span. You can add a tag to either the active span or the service entry span.

Note: Internal spans are not indexed by default and so might not be searchable in APM.

To create a dynamic span tag:

1. Select Span Tag as the instrumentation type.
2. Complete the generic instrumentation setup (choose service, environment, version, and location).
3. Specify a name for the tag.
4. Specify the value of the tag using the Dynamic Instrumentation expression language.
5. Optionally define a condition using the Dynamic Instrumentation expression language. The tag is added only when the expression evaluates to true.
6. Optionally add additional tags, each with their own name, expression, and optional condition.

You can use a dynamic span tag as an alternative to using Custom Instrumentation to add tags in code.

Further reading