The Service Map for APM is here!

APM Setup

This documentation covers Agent v6 only, to know how to set up APM tracing with Agent v5, refer to the dedicated APM with Agent v5 doc.

Setup process

With our infrastructure monitoring, metrics are sent to the Agent, which then forwards them to Datadog. Similarly, tracing metrics are also sent to the Agent: the application code instrumentation flushes to the Agent every 1 s (see here for the Python client for instance) and the Agent flushes to the Datadog API every 10s.

To start tracing your application:

  1. Install the Datadog Agent: Install and configure the latest Datadog Agent. For additional information, reference the getting started guide.

  2. Install the Trace Agent:

    • On Linux and Windows, the Trace Agent is pre-packaged with the standard Datadog Agent and no extra configuration is needed. See the Linux Trace Agent and Windows Trace Agent documentation for more information.

    • On macOS, install and run the Trace Agent in addition to the Datadog Agent. See the macOS Trace Agent documentation for more information.

    • On Docker, enable the Trace Agent in the datadog/agent container by passing DD_APM_ENABLED=true as an environment variable. See the APM and Docker documentation for more information.

    • On Heroku, Deploy the Trace Agent via the Datadog Heroku Buildpack.

  3. Configure your environment: An environment is a first class dimension used to scope a whole Datadog APM application. A common use case is to disaggregate metrics from stage environments such as production, staging, and pre-production. Learn how to configure environments. Note: if you do not configure your own environments, all data will default to env:none.

  4. Instrument your application: Select one of the following supported languages:

    To instrument an application written in a language that does not yet have official library support, visit our list of community tracing libraries.

  5. Start monitoring your app’s performance: Within a few minutes of running APM, you will start to see your services appear in the APM home page. See Using the APM UI to learn more.

Agent configuration

The APM Agent (also known as Trace Agent) is shipped by default with the Agent 6 in the Linux, MacOS, and Windows packages. The APM Agent is enabled by default on Linux. To enable the check on other platforms or disable it on Linux, update the apm_config key in your datadog.yaml:

apm_config:
  enabled: true
File setting Type Environment variable Description
enabled boolean DD_APM_ENABLED When set to true, the Datadog Agent accepts trace metrics. Default value is true.
apm_dd_url string DD_APM_DD_URL Datadog API endpoint where traces are sent.
env string - Default environment to which traces should be registered under (e.g. staging, production, etc..).
extra_sample_rate float - Use this setting to adjust the trace sample rate. The value should be a float between 0 (no sampling) and 1 (normal sampling). Default value is 1.
max_traces_per_second float - Maximum number of traces to sample per second. Set to 0 to disable the limit (not recommended). The default value is 10.
ignore_resources list DD_IGNORE_RESOURCE A list of resources that the agent should ignore. If using the environment variable, this should be a comma-separated list.
log_file string - Location of the log file.
replace_tags list A list of tag replacement rules. See the Scrubbing sensitive information section.
receiver_port number DD_RECEIVER_PORT Port that the Datadog Agent’s trace receiver listen on. Default value is 8126.
apm_non_local_traffic boolean DD_APM_NON_LOCAL_TRAFFIC Allows the agent to receive outside connections. It then listen on all interfaces.
max_memory float - Maximum memory that the agent is allowed to occupy. When this is exceeded the process is killed.
max_cpu_percent float - Maximum CPU percentage that the agent should use. The agent automatically adjusts its pre-sampling rate to stay below this number.
max_connections number - Maximum number of network connections that the agent is allowed to use. When this is exceeded the process is killed.

To get a an overview of all the possible settings for APM, take a look at the Trace Agent’s datadog.example.yaml configuration file. For more information about the Datadog Agent, see the dedicated doc page or refer to the datadog.yaml templates.

Reference the dedicated documentation to setup tracing with Docker.

Scrubbing sensitive information

Automatic scrubbing

Automatic scrubbing is available for some services, such as ElasticSearch, MongoDB, Redis, Memcached, and HTTP server and client request URLs. Below is an example configuration snippet documenting all the available options.

apm_config:
  # Defines obfuscation rules for sensitive data. Disabled by default.
  obfuscation:
    # ElasticSearch obfuscation rules. Applies to spans of type "elasticsearch".
    # More specifically, to the "elasticsearch.body" tag.
    elasticsearch:
      enabled: true
      # Values for the keys listed here will not be obfuscated.
      keep_values:
        - client_id
        - product_id

    # MongoDB obfuscation rules. Applies to spans of type "mongodb".
    # More specifically, to the "mongodb.query" tag.
    mongodb:
      enabled: true
      # Values for the keys listed here will not be obfuscated.
      keep_values:
        - document_id
        - template_id

    # HTTP obfuscation rules for "http.url" tags in spans of type "http".
    http:
      # If true, query strings in URLs will be obfuscated.
      remove_query_string: true
      # If true, path segments in URLs containing digits will be replaced by "?"
      remove_paths_with_digits: true

    # When enabled, stack traces will be removed (replaced by "?").
    remove_stack_traces: true

    # Obfuscation rules for spans of type "redis". Applies to the "redis.raw_command" tags.
    redis:
      enabled: true

    # Obfuscation rules for spans of type "memcached". Applies to the "memcached.command" tag.
    memcached:
      enabled: true

Replace rules

To scrub sensitive data from your span’s tags, use the replace_tags setting. It is a list containing one or more groups of parameters that describe how to perform replacements of sensitive data within your tags. These parameters are:

  • name: The key of the tag to replace. To match all tags, use *. To match the resource, use resource.name.
  • pattern: The regexp pattern to match against.
  • repl: The replacement string.

For example:

apm_config:
  replace_tags:
    # Replace all numbers following the `token/` string in the tag "http.url" with "?":
    - name: "http.url"
      pattern: "token/(.*)"
      repl: "?"
    # Replace all the occurrences of "foo" in any tag with "bar":
    - name: "*"
      pattern: "foo"
      repl: "bar"
    # Remove all "error.stack" tag's value.
    - name: "error.stack"
      pattern: "(?s).*"

Further Reading