Tracing Docker Applications

As of Agent 6.0.0, the Trace Agent is enabled by default. If it has been turned off, you can re-enable it in the gcr.io/datadoghq/agent container by passing DD_APM_ENABLED=true as an environment variable.

Tracing from the host

Tracing is available on port 8126/tcp from your host only by adding the option -p 127.0.0.1:8126:8126/tcp to the docker run command.

To make it available from any host, use -p 8126:8126/tcp instead.

For example, the following command allows the Agent to receive traces from your host only:

docker run -d --cgroupns host \
              --pid host \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              gcr.io/datadoghq/agent:latest

docker run -d -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              gcr.io/datadoghq/agent:latest

Docker APM Agent environment variables

List of all environment variables available for tracing within the Docker Agent:

Tracing from other containers

As with DogStatsD, traces can be submitted to the Agent from other containers either using Docker networks or with the Docker host IP.

Docker network

As a first step, create a user-defined bridge network:

docker network create <NETWORK_NAME>

Then start the Agent and the application container, connected to the network previously created:

# Datadog Agent
docker run -d --name datadog-agent \
              --network <NETWORK_NAME> \
              --cgroupns host \
              --pid host \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              -e DD_APM_NON_LOCAL_TRAFFIC=true \
              gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
              --network <NETWORK_NAME> \
              -e DD_AGENT_HOST=datadog-agent \
              company/app:latest

# Datadog Agent
docker run -d --name datadog-agent \
              --cgroupns host \
              --pid host \
              --network "<NETWORK_NAME>" \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_APM_ENABLED=true \
              -e DD_SITE=<DATADOG_SITE> \
              -e DD_APM_NON_LOCAL_TRAFFIC=true \
              gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
              --network "<NETWORK_NAME>" \
              -e DD_AGENT_HOST=datadog-agent \
              company/app:latest

This exposes the hostname datadog-agent in your app container. If you’re using docker-compose, <NETWORK_NAME> parameters are the ones defined under the networks section of your docker-compose.yml.

Your application tracers must be configured to submit traces to this address. Set environment variables with the DD_AGENT_HOST as the Agent container name, and DD_TRACE_AGENT_PORT as the Agent Trace port in your application containers. The example above uses host datadog-agent and port 8126 (the default value so you don’t have to set it).

Alternately, see the examples below to set the Agent host manually in each supported language:

DD_AGENT_HOST=datadog-agent \
DD_TRACE_AGENT_PORT=8126 \
java -javaagent:/path/to/the/dd-java-agent.jar -jar /your/app.jar

java -javaagent:/path/to/the/dd-java-agent.jar \
     -Ddd.agent.host=datadog-agent \
     -Ddd.agent.port=8126 \
     -jar /your/app.jar

from ddtrace import tracer

tracer.configure(
    hostname='datadog-agent',
    port=8126,
)

Datadog.configure do |c|
  c.agent.host = 'datadog-agent'
  c.agent.port = 8126
end

package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start(tracer.WithAgentAddr("datadog-agent:8126"))
    defer tracer.Stop()
}

const tracer = require('dd-trace').init({
    hostname: 'datadog-agent',
    port: 8126
});

# Environment variables
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
export CORECLR_PROFILER_PATH=<SYSTEM_DEPENDENT_PATH>
export DD_DOTNET_TRACER_HOME=/opt/datadog

# For containers
export DD_AGENT_HOST=datadog-agent
export DD_TRACE_AGENT_PORT=8126

# Start your application
dotnet example.dll

Docker host IP

Agent container port 8126 should be linked to the host directly. Configure your application tracer to report to the default route of this container (determine this using the ip route command).

The following is an example for the Python Tracer, assuming 172.17.0.1 is the default route:

from ddtrace import tracer

tracer.configure(hostname='172.17.0.1', port=8126)

Further Reading

Additional helpful documentation, links, and articles:

Source codeGITHUB Trace your ECS applicationsDOCUMENTATION Collect your application logsDOCUMENTATION Collect automatically your applications metrics and logsDOCUMENTATION Limit data collection to a subset of containers onlyDOCUMENTATION Assign tags to all data emitted by a containerDOCUMENTATION