---
title: Dynamic Instrumentation
description: >-
  Add instrumentation to your running production systems without restarts to
  collect metrics, spans, and tags from any location in your code.
breadcrumbs: Docs > APM > Application Instrumentation > Dynamic Instrumentation
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Dynamic Instrumentation

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Dynamic Instrumentation lets you add metrics, spans, and span tags to running production systems without restarts or code changes, including in third-party libraries.

If you are interested in trying out the latest user experience improvements for Dynamic Instrumentation, consider opting into the [autocomplete and search Preview](https://docs.datadoghq.com/dynamic_instrumentation/symdb.md).

## Getting started{% #getting-started %}

### Prerequisites{% #prerequisites %}

Dynamic Instrumentation supports Java, Python, .NET, and PHP. It requires the following:

- [Datadog Agent](https://docs.datadoghq.com/agent.md) 7.49.0 or higher is installed alongside your service.
- [Remote Configuration](https://docs.datadoghq.com/tracing/guide/remote_config.md) is enabled in that Agent.
- A supported Datadog SDK is installed and up to date. See the Enable Dynamic Instrumentation section for version requirements.
- [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md) tags `service`, `env`, and `version` are applied to your deployment.
- Recommended: [Source Code Integration](https://docs.datadoghq.com/integrations/guide/source-code-integration.md) is set up for your service.

### Permissions{% #permissions %}

The following permissions are required to use Dynamic Instrumentation:

- **Dynamic Instrumentation Read Configuration** (`debugger_read`): Required to access the Dynamic Instrumentation page.
- One of the following write permissions:
  - **Dynamic Instrumentation Write Configuration** (`debugger_write`): Required to create or modify instrumentations in any environment.
  - **Dynamic Instrumentation Write Pre-Prod** (`debugger_write_preprod`): Required to create or modify instrumentations in known pre-production environments only (such as staging or QA).

For more information about roles and how to assign roles to users, see [Role Based Access Control](https://docs.datadoghq.com/account_management/rbac/permissions.md#apm).

### Enable Dynamic Instrumentation{% #enable-dynamic-instrumentation %}

{% alert level="info" %}
Dynamic Instrumentation and [Live Debugger](https://docs.datadoghq.com/tracing/live_debugger.md) share the same enablement state per service and environment: enabling or disabling one also enables or disables the other. The two products have separate permissions and Settings pages.
{% /alert %}

#### (Recommended) In-app enablement{% #in-app-enablement %}

Manage Dynamic Instrumentation for each service and environment from the [Dynamic Instrumentation Settings page](https://app.datadoghq.com/dynamic-instrumentation/settings). In-app enablement is supported on the following minimum SDK versions:

- [Java](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/java.md) ≥ 1.48.0
- [Python](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/python.md) ≥ 3.10.0
- [.NET](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/dotnet-core.md) ≥ 3.29.0

If your SDK meets the minimum version and all prerequisites are met, Datadog automatically attempts to enable the service the first time you create an instrumentation for it.

#### Manual enablement{% #manual-enablement %}

Manual enablement is required for PHP and for older SDK versions of Java, Python, and .NET. You can also choose manual enablement on supported SDK versions if you prefer to manage enablement through environment variables (for example, to enable Dynamic Instrumentation in bulk across many services).

Select your runtime for manual enablement instructions:

- [java](https://docs.datadoghq.com/dynamic_instrumentation/enabling/java.md)
- [python](https://docs.datadoghq.com/dynamic_instrumentation/enabling/python.md)
- [dotnet](https://docs.datadoghq.com/dynamic_instrumentation/enabling/dotnet.md)
- [dotnet](https://docs.datadoghq.com/dynamic_instrumentation/enabling/dotnet.md)
- [php](https://docs.datadoghq.com/dynamic_instrumentation/enabling/php.md)

### Limitations{% #limitations %}

- Dynamic Instrumentation is not compatible with Azure App Services or serverless environments.
- Not all instrumentation types are supported in every language. See the language-specific enabling pages linked from Enable Dynamic Instrumentation for supported features and limitations.
- The Java SDK does not support Kotlin coroutines.

## Explore Dynamic Instrumentation{% #explore-dynamic-instrumentation %}

Dynamic Instrumentation can help you understand what your application is doing at runtime. By adding an instrumentation at a specific code location, you can capture additional telemetry from your application without the need to change code or redeploy it.

### Using instrumentations{% #using-instrumentations %}

An instrumentation allows you to collect additional telemetry from specific points in your code without halting the execution of the program.

Dynamic spans, span tags, and metrics are a UI-based alternative to adding custom instrumentation directly to your source code. Datadog receives the instrumentation configurations you define and dynamically applies them to the running service without requiring restarts.

Datadog captures and processes spans, tags, and metrics generated by Dynamic Instrumentation like other telemetry from the running application. Unlike manual custom instrumentation, Dynamic Instrumentation does not require code changes, deployments, or service restarts. To stop collecting data, disable the instrumentation in Datadog.

### Creating an instrumentation{% #creating-an-instrumentation %}

All instrumentation types require the same initial setup:

1. Go to the [Dynamic Instrumentation page](https://app.datadoghq.com/dynamic-instrumentation).
1. Click Create Instrumentation in the top right, or click the three-dot menu on a service and select Add an instrumentation for this service.
1. If they are not prefilled, choose service, runtime, environment, and version (optional).
1. Specify where to set the instrumentation in the source code by selecting either a class and method, or a specific line of code in a file. When autocomplete and search are available, use them to find files, methods, or symbols.

For the best experience, set up [Source Code Integration](https://docs.datadoghq.com/integrations/guide/source-code-integration.md) to view code directly in Datadog and select instrumentation locations as you would with breakpoints in an IDE.

For creation steps specific to each instrumentation type, see the following sections.

{% alert level="info" %}
Dynamic log instrumentations are supported in [Live Debugger](https://docs.datadoghq.com/tracing/live_debugger.md). Use Live Debugger to capture logs and variable snapshots in real time from running applications.
{% /alert %}

### Creating dynamic metrics{% #creating-dynamic-metrics %}

A dynamic metric emits a metric when it executes. To create a dynamic metric:

1. Select Metric as the instrumentation type.
1. Complete the generic instrumentation setup (choose service, environment, and location).
1. Specify a name for the metric, which is prefixed with `dynamic.instrumentation.metric.probe.`.
1. Select a metric type (count, gauge, or histogram).
1. Choose the value of the metric using the [Dynamic Instrumentation expression language](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md). You can use any numeric value you'd like from the execution context, such as a method parameter, local variable, a class field, or an expression that yields a numeric value. For count metrics this is optional, and if you omit it, every invocation increments the count by one.

Metric instrumentations are automatically enabled on all service instances that match the configured environment and version. Metric instrumentations are not rate limited and execute every time the method or line is invoked.

Dynamic Instrumentation supports the following metric types:

- Count: Counts how many times a given method or line is executed. Can be combined with [metric expressions](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md) to use the value of a variable to increment the count.
- Gauge: Generates a gauge based on the last value of a variable. This metric requires a [metric expression](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md).
- Histogram: Generates a statistical distribution of a variable. This metric requires a [metric expression](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md).

### Creating dynamic spans{% #creating-dynamic-spans %}

A dynamic span emits a span when a method is executed. To create a dynamic span:

1. Select Span as the instrumentation type.
1. Complete the generic instrumentation setup (choose service, environment, version, and location).

You can use a dynamic span as an alternative to [creating spans with Custom Instrumentation](https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/java.md#adding-spans). If the method throws an exception, the details of the exception are associated with the newly created span's `error` tag.

### Creating dynamic span tags{% #creating-dynamic-span-tags %}

A dynamic span tag adds a tag value to an existing span. You can add a tag to either the *active* span or the *service entry* span.

**Note**: Internal spans are not indexed by default and so might not be searchable in APM.

To create a dynamic span tag:

1. Select Span Tag as the instrumentation type.
1. Complete the generic instrumentation setup (choose service, environment, version, and location).
1. Specify a name for the tag.
1. Specify the value of the tag using the [Dynamic Instrumentation expression language](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md).
1. Optionally define a condition using the Dynamic Instrumentation expression language. The tag is added only when the expression evaluates to true.
1. Optionally add additional tags, each with their own name, expression, and optional condition.

You can use a *dynamic span tag* as an alternative to [using Custom Instrumentation to add tags in code](https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/java.md#adding-tags).

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Learn more about the Dynamic Instrumentation Expression Language](https://docs.datadoghq.com/dynamic_instrumentation/expression-language.md)
- [Removing sensitive information from your Dynamic Instrumentation data](https://docs.datadoghq.com/dynamic_instrumentation/sensitive-data-scrubbing.md)
- [Learn more about how to instrument your application](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries.md)
- [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md)
- [Discover and catalog the services reporting to Datadog](https://docs.datadoghq.com/internal_developer_portal/catalog.md)
- [Learn more about Metrics](https://docs.datadoghq.com/metrics.md)