--- title: Dynamic Instrumentation description: >- Add instrumentation to your running production systems without restarts to collect metrics, spans, and tags from any location in your code. breadcrumbs: Docs > APM > Application Instrumentation > Dynamic Instrumentation --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Dynamic Instrumentation {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} Dynamic Instrumentation lets you add metrics, spans, and span tags to running production systems without restarts or code changes, including in third-party libraries. To use autocomplete and search, opt in to the [preview](https://docs.datadoghq.com/dynamic_instrumentation/symdb.md). **Note**: Dynamic Instrumentation is not compatible with Azure App Services or serverless environments. ## Requirements{% #requirements %} ### Datadog SDK{% #datadog-sdk %} {% section displayed-if="Language is Java" %} This section only applies to users who meet the following criteria: Language is Java - **[Datadog Java SDK](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/java.md)** (JDK 8 or higher) - Minimum for in-app enablement: 1.48.0 - Minimum for manual enablement: 1.34.0 - Does not support Kotlin coroutines {% /section %} {% section displayed-if="Language is Python" %} This section only applies to users who meet the following criteria: Language is Python - **[Datadog Python SDK (`ddtrace`)](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/python.md)** - Minimum for in-app enablement: 3.10.0 - Minimum for manual enablement: 2.2.0 {% /section %} {% section displayed-if="Language is .NET" %} This section only applies to users who meet the following criteria: Language is .NET - **Datadog .NET SDK** ([.NET Framework](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/dotnet-framework.md) or [.NET Core](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/dotnet-core.md)) - Minimum for in-app enablement: 3.29.0 - Minimum for manual enablement: 2.54.0 {% /section %} {% section displayed-if="Language is PHP" %} This section only applies to users who meet the following criteria: Language is PHP - **[Datadog PHP SDK (`dd-trace-php`)](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/php.md)**, minimum version 1.5.0 - File and line instrumentations are not supported {% /section %} ### Datadog configuration{% #datadog-configuration %} - Install the [Datadog Agent](https://docs.datadoghq.com/agent.md), version 7.49.0 or higher. - Configure [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md) with `service`, `env`, and `version` tags. - Enable [Remote Configuration](https://docs.datadoghq.com/tracing/guide/remote_config.md) in the Agent. - (Recommended) Set up [Source Code Integration](https://docs.datadoghq.com/integrations/guide/source-code-integration.md). ### Permissions{% #permissions %} The following permissions are required to use Dynamic Instrumentation: - **Dynamic Instrumentation Read Configuration** (`debugger_read`): Required to access the Dynamic Instrumentation page. - One of the following write permissions: - **Dynamic Instrumentation Write Configuration** (`debugger_write`): Required to create or modify instrumentations in any environment. - **Dynamic Instrumentation Write Pre-Prod** (`debugger_write_preprod`): Required to create or modify instrumentations in known pre-production environments only (such as staging or QA). For more information about roles and how to assign roles to users, see [Role Based Access Control](https://docs.datadoghq.com/account_management/rbac/permissions.md#apm). ## Enable Dynamic Instrumentation{% #enable-dynamic-instrumentation %} {% alert level="info" %} Dynamic Instrumentation and [Live Debugger](https://docs.datadoghq.com/tracing/live_debugger.md) share the same enablement state per service and environment: enabling or disabling one also enables or disables the other. The two products have separate permissions and Settings pages. {% /alert %} {% section displayed-if="The selected value for Language is included in the given list: 'Java, Python, .NET'" %} This section only applies to users who meet the following criteria: The selected value for Language is included in the given list: 'Java, Python, .NET' You can enable Dynamic Instrumentation in-app or manually. ### Enable in-app (recommended){% #enable-in-app %} Enable Dynamic Instrumentation in-app in one of two ways: - On the [Dynamic Instrumentation Settings page](https://app.datadoghq.com/dynamic-instrumentation/settings), enable the service and environment. - Create an instrumentation. Dynamic Instrumentation is enabled automatically on the selected service and environment. ### Enable manually{% #enable-manually %} Follow these steps if your SDK is below the in-app minimum, or if you want to manage enablement outside the Datadog UI. {% /section %} {% section displayed-if="Language is Java" %} This section only applies to users who meet the following criteria: Language is Java 1. If you don't already have APM enabled, set `DD_APM_ENABLED=true` and configure the Agent to listen on port `8126/TCP`. 1. Download `dd-java-agent.jar`: {% tab title="Wget" %} ``` wget -O dd-java-agent.jar 'https://dtdg.co/latest-java-tracer' ``` {% /tab %} {% tab title="cURL" %} ``` curl -Lo dd-java-agent.jar 'https://dtdg.co/latest-java-tracer' ``` {% /tab %} {% tab title="Dockerfile" %} ``` ADD 'https://dtdg.co/latest-java-tracer' dd-java-agent.jar ``` {% /tab %} 1. Start your service with Dynamic Instrumentation enabled. The `-javaagent` argument must come before `-jar`: {% tab title="Command arguments" %} ``` java \ -javaagent:dd-java-agent.jar \ -Ddd.service= \ -Ddd.env= \ -Ddd.version= \ -Ddd.dynamic.instrumentation.enabled=true \ -jar .jar ``` {% /tab %} {% tab title="Environment variables" %} ``` export DD_SERVICE= export DD_ENV= export DD_VERSION= export DD_DYNAMIC_INSTRUMENTATION_ENABLED=true java \ -javaagent:dd-java-agent.jar \ -jar .jar ``` {% /tab %} **Note**: On JDK 18 and earlier, classes compiled with the `-parameters` flag (default in Spring 6+, Spring Boot 3+, and Scala) may fail to instrument with the error `Method Parameters detected`. 1. After you start your service, open the [Dynamic Instrumentation page](https://app.datadoghq.com/dynamic-instrumentation). {% /section %} {% section displayed-if="Language is Python" %} This section only applies to users who meet the following criteria: Language is Python 1. If you don't already have APM enabled, set `DD_APM_ENABLED=true` and configure the Agent to listen on port `8126/TCP`. 1. Install `ddtrace`: ``` pip install ddtrace ``` 1. Enable Dynamic Instrumentation with environment variables or in code: {% tab title="Environment variables" %} ``` export DD_SERVICE= export DD_ENV= export DD_VERSION= export DD_DYNAMIC_INSTRUMENTATION_ENABLED=true ddtrace-run python -m myapp.py ``` {% /tab %} {% tab title="In code" %} ``` import os os.environ["DD_DYNAMIC_INSTRUMENTATION_ENABLED"] = "true" import ddtrace.auto # this must be imported as soon as possible ``` {% /tab %} 1. After you start your service, open the [Dynamic Instrumentation page](https://app.datadoghq.com/dynamic-instrumentation). {% /section %} {% section displayed-if="Language is .NET" %} This section only applies to users who meet the following criteria: Language is .NET 1. If you don't already have APM enabled, set `DD_APM_ENABLED=true` and configure the Agent to listen on port `8126/TCP`. 1. Start your service with the following environment variables set: ``` DD_SERVICE= DD_ENV= DD_VERSION= DD_DYNAMIC_INSTRUMENTATION_ENABLED=true ``` 1. After you start your service, open the [Dynamic Instrumentation page](https://app.datadoghq.com/dynamic-instrumentation). {% /section %} {% section displayed-if="Language is PHP" %} This section only applies to users who meet the following criteria: Language is PHP 1. If you don't already have APM enabled, set `DD_APM_ENABLED=true` and configure the Agent to listen on port `8126/TCP`. 1. Start your service with the following environment variables set: ``` DD_SERVICE= DD_ENV= DD_VERSION= DD_DYNAMIC_INSTRUMENTATION_ENABLED=true ``` 1. After you start your service, open the [Dynamic Instrumentation page](https://app.datadoghq.com/dynamic-instrumentation). {% /section %} ## Configure Dynamic Instrumentation{% #configure-dynamic-instrumentation %} Configure Dynamic Instrumentation using the following environment variables: | Environment variable | Type | Description | | ------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------- | | `DD_DYNAMIC_INSTRUMENTATION_ENABLED` | Boolean | Set to `true` to enable Dynamic Instrumentation. | | `DD_SERVICE` | String | The [service](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md) name, for example, `web-backend`. | | `DD_ENV` | String | The [environment](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md) name, for example, `production`. | | `DD_VERSION` | String | The [version](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md) of your service. | | `DD_TAGS` | String | Tags to apply to produced data. Must be a list of `:` separated by commas such as: `layer:api,team:intake`. | {% section displayed-if="Language is Java" %} This section only applies to users who meet the following criteria: Language is Java **Note**: You can also set `-Ddd.dynamic.instrumentation.enabled=true` instead of `DD_DYNAMIC_INSTRUMENTATION_ENABLED`. {% /section %} ## Explore Dynamic Instrumentation{% #explore-dynamic-instrumentation %} Dynamic Instrumentation can help you understand what your application is doing at runtime. By adding an instrumentation at a specific code location, you can capture additional telemetry from your application without the need to change code or redeploy it. ### Using instrumentations{% #using-instrumentations %} An instrumentation allows you to collect additional telemetry from specific points in your code without halting the execution of the program. Dynamic spans, span tags, and metrics are a UI-based alternative to adding custom instrumentation directly to your source code. Datadog receives the instrumentation configurations you define and dynamically applies them to the running service without requiring restarts. Datadog captures and processes spans, tags, and metrics generated by Dynamic Instrumentation like other telemetry from the running application. Unlike manual custom instrumentation, Dynamic Instrumentation does not require code changes, deployments, or service restarts. To stop collecting data, disable the instrumentation in Datadog. To redact sensitive data from instrumentations, see [Sensitive data scrubbing](https://docs.datadoghq.com/dynamic_instrumentation/sensitive-data-scrubbing.md). ### Creating an instrumentation{% #creating-an-instrumentation %} All instrumentation types require the same initial setup: 1. Go to the [Dynamic Instrumentation page](https://app.datadoghq.com/dynamic-instrumentation). 1. Click Create Instrumentation in the top right, or click the three-dot menu on a service and select Add an instrumentation for this service. 1. If the fields are not prefilled, choose service, runtime, environment, and version (optional). 1. {% section displayed-if="The selected value for Language is included in the given list: 'Java, Python, .NET'" %} This section only applies to users who meet the following criteria: The selected value for Language is included in the given list: 'Java, Python, .NET' Specify where to set the instrumentation in the source code by selecting either a class and method, or a specific line of code in a file. When autocomplete and search are available, use them to find files, methods, or symbols. {% /section %} {% section displayed-if="Language is PHP" %} This section only applies to users who meet the following criteria: Language is PHP Specify where to set the instrumentation in the source code by selecting a class and method. When autocomplete and search are available, use them to find methods or symbols. {% /section %} For the best experience, set up [Source Code Integration](https://docs.datadoghq.com/integrations/guide/source-code-integration.md) to view code directly in Datadog and select instrumentation locations as you would with breakpoints in an IDE. For creation steps specific to each instrumentation type, see the following sections. {% alert level="info" %} Dynamic log instrumentations are supported in [Live Debugger](https://docs.datadoghq.com/tracing/live_debugger.md). Use Live Debugger to capture logs and variable snapshots in real time from running applications. {% /alert %} ### Creating dynamic metrics{% #creating-dynamic-metrics %} A dynamic metric emits a metric when it executes. To create a dynamic metric: 1. Select Metric as the instrumentation type. 1. Complete the generic instrumentation setup (choose service, environment, and location). 1. Specify a name for the metric, which is prefixed with `dynamic.instrumentation.metric.probe.`. 1. Select a metric type (count, gauge, or histogram). 1. Choose the value of the metric using the [Dynamic Instrumentation expression language](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md). You can use any numeric value from the execution context, such as a method parameter, local variable, a class field, or an expression that yields a numeric value. For count metrics, this is optional, and if you omit it, every invocation increments the count by one. Metric instrumentations are automatically enabled on all service instances that match the configured environment and version. Metric instrumentations are not rate limited and execute every time the method {% section displayed-if="The selected value for Language is included in the given list: 'Java, Python, .NET'" %} This section only applies to users who meet the following criteria: The selected value for Language is included in the given list: 'Java, Python, .NET' or line{% /section %} is invoked. Dynamic Instrumentation supports the following metric types: - Count: Counts how many times a given method {% section displayed-if="The selected value for Language is included in the given list: 'Java, Python, .NET'" %} This section only applies to users who meet the following criteria: The selected value for Language is included in the given list: 'Java, Python, .NET' or line {% /section %} is executed. Can be combined with [metric expressions](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md) to use the value of a variable to increment the count. - Gauge: Generates a gauge based on the last value of a variable. This metric requires a [metric expression](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md). - Histogram: Generates a statistical distribution of a variable. This metric requires a [metric expression](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md). ### Creating dynamic spans{% #creating-dynamic-spans %} A dynamic span emits a span when a method is executed. To create a dynamic span: 1. Select Span as the instrumentation type. 1. Complete the generic instrumentation setup (choose service, environment, version, and location). You can use a dynamic span as an alternative to [creating spans with Custom Instrumentation](https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/server-side.md). If the method throws an exception, the details of the exception are associated with the newly created span's `error` tag. ### Creating dynamic span tags{% #creating-dynamic-span-tags %} A dynamic span tag adds a tag value to an existing span. You can add a tag to either the *active* span or the *service entry* span. **Note**: Internal spans are not indexed by default and so might not be searchable in APM. To create a dynamic span tag: 1. Select Span Tag as the instrumentation type. 1. Complete the generic instrumentation setup (choose service, environment, version, and location). 1. Specify a name for the tag. 1. Specify the value of the tag using the [Dynamic Instrumentation expression language](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md). 1. Optionally define a condition using the Dynamic Instrumentation expression language. The tag is added only when the expression evaluates to true. 1. Optionally add additional tags, each with their own name, expression, and optional condition. You can use a *dynamic span tag* as an alternative to [using Custom Instrumentation to add tags in code](https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/server-side.md). ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Learn more about the Dynamic Instrumentation Expression Language](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md) - [Removing sensitive information from your Dynamic Instrumentation data](https://docs.datadoghq.com/dynamic_instrumentation/sensitive-data-scrubbing.md) - [Learn more about how to instrument your application](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries.md) - [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md) - [Discover and catalog the services reporting to Datadog](https://docs.datadoghq.com/internal_developer_portal/catalog.md) - [Learn more about Metrics](https://docs.datadoghq.com/metrics.md)