Tracing Docker Applications

Tracing Docker Applications

Enable the Trace Agent in the gcr.io/datadoghq/agent container by passing DD_APM_ENABLED=true as an environment variable.

Tracing from the host

Tracing is available on port 8126/tcp from your host only by adding the option -p 127.0.0.1:8126:8126/tcp to the docker run command.

To make it available from any host, use -p 8126:8126/tcp instead.

For example, the following command allows the Agent to receive traces from your host only:

    docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              gcr.io/datadoghq/agent:latest


    docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              gcr.io/datadoghq/agent:latest

    Where <DATADOG_SITE> is , so that the Agent sends data to the right Datadog location.

    docker run -d -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              gcr.io/datadoghq/agent:latest


    docker run -d -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              gcr.io/datadoghq/agent:latest

    Where <DATADOG_SITE> is , so that the Agent sends data to the right Datadog location.

    Docker APM Agent environment variables

    List of all environment variables available for tracing within the Docker Agent:

    Environment variableDescription
    DD_API_KEYDatadog API Key
    DD_PROXY_HTTPSSet up the URL for the proxy to use.
    DD_APM_REPLACE_TAGSScrub sensitive data from your span’s tags.
    DD_APM_FILTER_TAGS_REQUIREDefines required tags that traces must have in order to be sent to Datadog.
    DD_APM_FILTER_TAGS_REJECTDefines rejection tags. The Agent drops traces that have these tags.
    DD_HOSTNAMEManually set the hostname to use for metrics if autodection fails, or when running the Datadog Cluster Agent.
    DD_DOGSTATSD_PORTSet the DogStatsD port.
    DD_APM_RECEIVER_SOCKETCollect your traces through a Unix Domain Sockets and takes priority over hostname and port configuration if set. Off by default, when set it must point to a valid sock file.
    DD_BIND_HOSTSet the StatsD & receiver hostname.
    DD_LOG_LEVELSet the logging level. (trace/debug/info/warn/error/critical/off)
    DD_APM_ENABLEDWhen set to true, the Datadog Agent accepts trace metrics.
    DD_APM_CONNECTION_LIMITSets the maximum connection limit for a 30 second time window. The default limit is 2000 connections.
    DD_APM_DD_URLSet the Datadog API endpoint where your traces are sent: https://trace.agent.. Defaults to https://trace.agent.datadoghq.com.
    DD_APM_RECEIVER_PORTPort that the Datadog Agent’s trace receiver listens on. Default value is 8126.
    DD_APM_NON_LOCAL_TRAFFICAllow non-local traffic when tracing from other containers.
    DD_APM_IGNORE_RESOURCESConfigure resources for the Agent to ignore. Format should be comma separated, regular expressions. Example: GET /ignore-me,(GET|POST) /and-also-me.
    DD_APM_ANALYZED_SPANSConfigure the spans to analyze for transactions. Format should be comma separated instances of <SERVICE_NAME>|;<OPERATION_NAME>=1. i.e. my-express-app|;express.request=1,my-dotnet-app|;aspnet_core_mvc.request=1. You can also enable it automatically with the configuration parameter in the Tracing Client.
    DD_APM_MAX_EPSSets the maximum Indexed Spans per second. Default is 200 events per second.

    Tracing from other containers

    As with DogStatsD, traces can be submitted to the Agent from other containers either using Docker networks or with the Docker host IP.

    Docker network

    As a first step, create a user-defined bridge network:

    docker network create <NETWORK_NAME>

    Then start the Agent and the application container, connected to the network previously created:

      # Datadog Agent
docker run -d --name datadog-agent \
              --network <NETWORK_NAME> \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_APM_NON_LOCAL_TRAFFIC=true \
              gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
              --network <NETWORK_NAME> \
              -e DD_AGENT_HOST=datadog-agent \
              company/app:latest


      # Datadog Agent
docker run -d --name datadog-agent \
              --network <NETWORK_NAME> \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              -e DD_APM_NON_LOCAL_TRAFFIC=true \
              gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
              --network <NETWORK_NAME> \
              -e DD_AGENT_HOST=datadog-agent \
              company/app:latest

      Where <DATADOG_SITE> is , so that the Agent sends data to the right Datadog location.

      # Datadog Agent
docker run -d --name datadog-agent \
              --network "<NETWORK_NAME>" \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_APM_NON_LOCAL_TRAFFIC=true \
              gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
              --network "<NETWORK_NAME>" \
              -e DD_AGENT_HOST=datadog-agent \
              company/app:latest


      # Datadog Agent
docker run -d --name datadog-agent \
              --network "<NETWORK_NAME>" \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              -e DD_APM_NON_LOCAL_TRAFFIC=true \
              gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
              --network "<NETWORK_NAME>" \
              -e DD_AGENT_HOST=datadog-agent \
              company/app:latest

      Where <DATADOG_SITE> is , so that the Agent sends data to the right Datadog location.

      This exposes the hostname datadog-agent in your app container. If you’re using docker-compose, <NETWORK_NAME> parameters are the ones defined under the networks section of your docker-compose.yml.

      Your application tracers must be configured to submit traces to this address. Set environment variables with the DD_AGENT_HOST as the Agent container name, and DD_TRACE_AGENT_PORT as the Agent Trace port in your application containers. The example above uses host datadog-agent and port 8126 (the default value so you don’t have to set it).

      Alternately, see the examples below to set the Agent host manually in each supported language:

      Either update the Java Agent configuration via environment variables:

      DD_AGENT_HOST=datadog-agent \
DD_TRACE_AGENT_PORT=8126 \
java -javaagent:/path/to/the/dd-java-agent.jar -jar /your/app.jar

      or via system properties:

      java -javaagent:/path/to/the/dd-java-agent.jar \
     -Ddd.agent.host=datadog-agent \
     -Ddd.agent.port=8126 \
     -jar /your/app.jar

      from ddtrace import tracer

tracer.configure(
    hostname='datadog-agent',
    port=8126,
)

      Datadog.configure do |c|
  c.tracer hostname: 'datadog-agent',
           port: 8126
end

      package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start(tracer.WithAgentAddr("datadog-agent:8126"))
    defer tracer.Stop()
}

      const tracer = require('dd-trace').init({
    hostname: 'datadog-agent',
    port: 8126
});

      Set the environment variables before running your instrumented app:

      # Environment variables
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
export CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
export DD_INTEGRATIONS=/opt/datadog/integrations.json
export DD_DOTNET_TRACER_HOME=/opt/datadog

# For containers
export DD_AGENT_HOST=datadog-agent
export DD_TRACE_AGENT_PORT=8126

# Start your application
dotnet example.dll

      Docker host IP

      Agent container port 8126 should be linked to the host directly. Configure your application tracer to report to the default route of this container (determine this using the ip route command).

      The following is an example for the Python Tracer, assuming 172.17.0.1 is the default route:

      from ddtrace import tracer

tracer.configure(hostname='172.17.0.1', port=8126)

      Further Reading

      Additional helpful documentation, links, and articles:

      Source codeGITHUB more more Trace your ECS applicationsDOCUMENTATION more more Collect your application logsDOCUMENTATION more more Collect automatically your applications metrics and logsDOCUMENTATION more more Limit data collection to a subset of containers onlyDOCUMENTATION more more Assign tags to all data emitted by a containerDOCUMENTATION more more