This documentation covers Agent v6 only, to know how to set up APM tracing with Agent v5, refer to the dedicated APM with Agent v5 doc.
With our infrastructure monitoring, metrics are sent to the Agent, which then forwards them to Datadog. Similarly, tracing metrics are also sent to the Agent: the application code instrumentation flushes to the Agent every 1 s (see here for the Python client for instance) and the Agent flushes to the Datadog API every 10s.
To start tracing your application:
Install the Trace Agent:
On Linux and Windows, the Trace Agent is pre-packaged with the standard Datadog Agent and no extra configuration is needed. See the Linux Trace Agent and Windows Trace Agent documentation for more information.
On macOS, install and run the Trace Agent in addition to the Datadog Agent. See the macOS Trace Agent documentation for more information.
On Docker, enable the Trace Agent in the
datadog/agent container by passing
DD_APM_ENABLED=true as an environment variable. See the APM and Docker documentation for more information.
On Heroku, Deploy the Trace Agent via the Datadog Heroku Buildpack.
Configure your environment:
An environment is a first class dimension used to scope a whole Datadog APM application. A common use case is to disaggregate metrics from stage environments such as production, staging, and pre-production. Learn how to configure environments.
Note: if you do not configure your own environments, all data will default to
Instrument your application: Select one of the following supported languages:
To instrument an application written in a language that does not yet have official library support, visit our list of community tracing libraries.
The APM Agent (also known as Trace Agent) is shipped by default with the
Agent 6 in the Linux, MacOS, and Windows packages. The APM Agent is enabled by default on Linux. To enable the check on other platforms or disable it on Linux, update the
apm_config key in your
apm_config: enabled: true
|File setting||Type||Environment variable||Description|
||When set to
||Datadog API endpoint where traces are sent.|
||string||-||Default environment to which traces should be registered under (e.g. staging, production, etc..).|
||float||-||Use this setting to adjust the trace sample rate. The value should be a float between
||float||-||Maximum number of traces to sample per second. Set to
||A list of resources that the agent should ignore. If using the environment variable, this should be a comma-separated list.|
||string||-||Location of the log file.|
||list||A list of tag replacement rules. See the Scrubbing sensitive information section.|
||Port that the Datadog Agent’s trace receiver listen on. Default value is
||Allows the agent to receive outside connections. It then listen on all interfaces.|
||float||-||Maximum memory that the agent is allowed to occupy. When this is exceeded the process is killed.|
||float||-||Maximum CPU percentage that the agent should use. The agent automatically adjusts its pre-sampling rate to stay below this number.|
||number||-||Maximum number of network connections that the agent is allowed to use. When this is exceeded the process is killed.|
To get a an overview of all the possible settings for APM, take a look at the Trace Agent’s
datadog.example.yaml configuration file.
For more information about the Datadog Agent, see the dedicated doc page or refer to the
Automatic scrubbing is available for some services, such as ElasticSearch, MongoDB, Redis, Memcached, and HTTP server and client request URLs. Below is an example configuration snippet documenting all the available options.
apm_config: # Defines obfuscation rules for sensitive data. Disabled by default. obfuscation: # ElasticSearch obfuscation rules. Applies to spans of type "elasticsearch". # More specifically, to the "elasticsearch.body" tag. elasticsearch: enabled: true # Values for the keys listed here will not be obfuscated. keep_values: - client_id - product_id # MongoDB obfuscation rules. Applies to spans of type "mongodb". # More specifically, to the "mongodb.query" tag. mongodb: enabled: true # Values for the keys listed here will not be obfuscated. keep_values: - document_id - template_id # HTTP obfuscation rules for "http.url" tags in spans of type "http". http: # If true, query strings in URLs will be obfuscated. remove_query_string: true # If true, path segments in URLs containing digits will be replaced by "?" remove_paths_with_digits: true # When enabled, stack traces will be removed (replaced by "?"). remove_stack_traces: true # Obfuscation rules for spans of type "redis". Applies to the "redis.raw_command" tags. redis: enabled: true # Obfuscation rules for spans of type "memcached". Applies to the "memcached.command" tag. memcached: enabled: true
To scrub sensitive data from your span’s tags, use the
replace_tags setting. It is a list containing one or more groups of parameters that describe how to perform replacements of sensitive data within your tags. These parameters are:
name: The key of the tag to replace. To match all tags, use
*. To match the resource, use
pattern: The regexp pattern to match against.
repl: The replacement string.
apm_config: replace_tags: # Replace all numbers following the `token/` string in the tag "http.url" with "?": - name: "http.url" pattern: "token/(.*)" repl: "?" # Replace all the occurrences of "foo" in any tag with "bar": - name: "*" pattern: "foo" repl: "bar" # Remove all "error.stack" tag's value. - name: "error.stack" pattern: "(?s).*"
Additional helpful documentation, links, and articles: