Logging is here!

APM Setup

This documentation covers Agent v6 only, to know how to set up APM tracing with Agent v5, refer to the dedicated APM with Agent v5 doc.

Setup process

With our infrastructure monitoring, metrics are sent to the Agent, which then forwards them to Datadog. Similarly, tracing metrics are also sent to the Agent: the application code instrumentation flushes to the Agent every 1 s (see here for the Python client for instance) and the Agent flushes to the Datadog API every 10s.

To start tracing your application:

  1. Install the Datadog Agent: Install and configure the latest Datadog Agent. For additional information, reference the getting started guide.

  2. Install the Trace Agent:

    • On Linux and Windows, the Trace Agent is pre-packaged with the standard Datadog Agent and no extra configuration is needed. See the Linux Trace Agent and Windows Trace Agent documentation for more information.

    • On macOS, install and run the Trace Agent in addition to the Datadog Agent. See the macOS Trace Agent documentation for more information.

    • On Docker, enable the Trace Agent in the datadog/agent container by passing DD_APM_ENABLED=true as an environment variable. See the APM and Docker documentation for more information.

    • On Heroku, Deploy the Trace Agent via the Datadog Heroku Buildpack.

  3. Configure your environment: An environment is a first class dimension used to scope a whole Datadog APM application. A common use case is to disaggregate metrics from stage environments such as production, staging, and pre-production. Learn how to configure environments. Note: if you do not configure your own environments, all data will default to env:none.

  4. Instrument your application: Select one of the following supported languages:

    To instrument an application written in a language that does not yet have official library support, visit our list of community tracing libraries.

  5. Start monitoring your app’s performance: Within a few minutes of running APM, you will start to see your services appear in the APM home page. See Using the APM UI to learn more.

Agent configuration

The APM Agent (also known as Trace Agent) is shipped by default with the Agent 6 in the Linux, MacOS, and Windows packages. The APM Agent is enabled by default on Linux. To enable the check on other platforms or disable it on Linux, update the apm_config key in your datadog.yaml:

apm_config:
  enabled: true
File setting Type Environment variable Description
enabled               boolean   DD_APM_ENABLED           When set to true, the Datadog Agent accepts trace metrics. Default value is true.                                                          
apm_dd_url           string   DD_APM_DD_URL           Datadog API endpoint where traces are sent.                                                                                                      
env string - Default environment to which traces should be registered under (e.g. staging, production, etc..).
extra_sample_rate float - Use this setting to adjust the trace sample rate. The value should be a float between 0 (no sampling) and 1 (normal sampling). Default value is 1.
max_traces_per_second float - Maximum number of traces to sample per second. Set to 0 to disable the limit (not recommended). The default value is 10.
ignore_resources     list     DD_IGNORE_RESOURCE       A list of resources that the agent should ignore. If using the environment variable, this should be a comma-separated list.                                                    
log_file string - Location of the log file.
replace_tags list A list of tag replacement rules. See the Scrubbing sensitive information section.
receiver_port         number   DD_RECEIVER_PORT         Port that the Datadog Agent’s trace receiver listen on. Default value is 8126.                                                                
apm_non_local_traffic boolean   DD_APM_NON_LOCAL_TRAFFIC Allows the agent to receive outside connections. It then listen on all interfaces.                                                                              
max_memory float - Maximum memory that the agent is allowed to occupy. When this is exceeded the process is killed.
max_cpu_percent float - Maximum CPU percentage that the agent should use. The agent automatically adjusts its pre-sampling rate to stay below this number.
max_connections number - Maximum number of network connections that the agent is allowed to use. When this is exceeded the process is killed.

For more information about the Datadog Agent, see the dedicated doc page or refer to the datadog.yaml templates.

Reference the dedicated documentation to setup tracing with Docker.

Scrubbing sensitive information

To scrub sensitive data from your span’s tags, use the replace_tags setting. It is a list containing one or more groups of parameters that describe how to perform replacements of sensitive data within your tags. These parameters are:

  • name: The key of the tag to replace. To match all tags, use *. To match the resource, use resource.name.
  • pattern: The regexp pattern to match against.
  • repl: The replacement string.

For example:

apm_config:
  replace_tags:
    # Replace all numbers following the `token/` string in the tag "http.url" with "?":
    - name: "http.url"
      pattern: "token/(.*)"
      repl: "?"
    # Replace all the occurrences of "foo" in any tag with "bar":
    - name: "*"
      pattern: "foo"
      repl: "bar"
    # Remove all "error.stack" tag's value.
    - name: "error.stack"
      pattern: "(?s).*"

Further Reading