This documentation covers Agent v6 only, to know how to set up APM tracing with Agent v5, refer to the dedicated APM with Agent v5 doc.
With Datadog’s infrastructure monitoring, metrics are sent to the Agent, which then forwards them to Datadog. Similarly, tracing metrics are also sent to the Agent: the application code instrumentation flushes to the Agent every second (see here for the Python client for instance) and the Agent flushes to the Datadog API every 10 seconds.
To start tracing your application:
Install the Datadog Agent: Install and configure the latest Datadog Agent. (On macOS, install and run the Trace Agent in addition to the Datadog Agent. See the macOS Trace Agent documentation for more information).
Enable trace collection for the Datadog Agent. See below dedicated instructions.
Configure your environment: An environment is a mandatory primary tag used to scope a whole Datadog APM application. A common use case is to disaggregate metrics from stage environments such as production, staging, and pre-production. Learn how to configure primary tags.
Instrument your application:
Select one of the following supported languages:
To instrument an application written in a language that does not yet have official library support, visit the list of community tracing libraries.
To enable trace collection for your Agent, update the
apm_config key in your Agent
datadog.yaml main configuration file:
apm_config: enabled: true
Find below the list of all available parameters for your
datadog.yaml configuration file:
||boolean||When set to
||string||Datadog API endpoint where traces are sent.|
||string||Default environment to which traces should be registered under (e.g. staging, production, etc..).|
||float||Use this setting to adjust the trace sample rate. The value should be a float between
||float||Maximum number of traces to sample per second. Set to
||list||A list of resources that the Agent should ignore.|
||string||Location of the log file.|
||list||A list of tag replacement rules. Read more about scrubbing sensitive data with replace rules.|
||number||Port that the Datadog Agent’s trace receiver listen on. Default value is
||float||Maximum memory that the Agent is allowed to occupy. When this is exceeded the process is killed.|
||float||Maximum CPU percentage that the Agent should use. The Agent automatically adjusts its pre-sampling rate to stay below this number.|
To get a an overview of all the possible settings for APM, take a look at the Agent’s
datadog.example.yaml configuration file.
For more information about the Datadog Agent, see the dedicated doc page or refer to the
To enable trace search, services must be flowing into Datadog. Once services are set up, navigate to the Trace Search & Analytics docs page to find a list of each of the services running within your infrastructure.
In Datadog, every automatically instrumented service has an operation name, which is used to set the type of request being traced. For example, if you’re tracing a Python Flask application, you might have a
flask.request as your operation name. In a Node application using Express, you would have
express.request ask your operation name.
Replace both the
<OPERATION_NAME> in your configuration with the service name and operation name of the traces you want to add to Trace Search.
For example, if you have a Python service named
python-api, and it’s running Flask (operation name
<SERVICE_NAME> would be
python-api, and your
<OPERATION_NAME> would be
The Trace Search & Analytics docs page populates with a list of your services and resource names available for usage in Trace Search:
servicesto extract APM events from.
apm_config. For example:
apm_config: analyzed_spans: <SERVICE_NAME_1>|<OPERATION_NAME_1>: 1 <SERVICE_NAME_2>|<OPERATION_NAME_2>: 1
[trace.analyzed_spans]. For example:
[trace.analyzed_spans] <SERVICE_NAME_1>|<OPERATION_NAME_1>: 1 <SERVICE_NAME_2>|<OPERATION_NAME_2>: 1
DD_APM_ANALYZED_SPANS to the Agent container environment (compatible with version 12.6.5250+). Format should be a comma-separated regular expressions without spaces. For example:
There are several dimensions available to scope an entire Datadog APM application. These include aggregate statistics (such as requests/second, latency, error rate, Apdex score) and visible traces. These dimensions are set up through primary tags that allow you to get an even finer view of your application’s behavior. Use cases for primary tags include environment, availability zone, datacenter, etc.
Primary tags must follow a different set of rules from those of conventional Datadog tags.
The default and mandatory primary tag is the environment your traces are collected from. Its tag key is
env, and its default value for un-tagged data is
There are several ways to specify an environment when reporting data:
Use a host tag with the format
env:<ENVIRONMENT> to tag all traces from that Agent accordingly.
Override the default tag used by the Agent in the Agent configuration file. This tags all traces coming through the Agent, overriding the host tag value.
apm_config: env: <ENVIRONMENT>
When submitting a single trace, specify an environment by tagging one of its spans with the metadata key
env. This overrides the Agent configuration and the host tag’s value (if any). Consult the trace tagging documentation to learn how to assign a tag to your traces.
Environments appear at the top of APM pages. Use the dropdown to scope the data displayed on the current page.
If you added a host tag other than
env:<ENVIRONMENT> to your traces, it can be set as a primary tag along with the environment tag. Go to the APM Settings page to define, change, or remove your primary tags.
If you change a previously set primary tag, be aware of the following:
Primary tags appear at the top of APM pages. Use these selectors to slice the data displayed on the current page. To view all data independent of a primary tag, choose
<TAG_NAME>:* from the dropdown (as in the image below).