Tracing Docker Applications
As of Agent 6.0.0, the Trace Agent is enabled by default. If it has been turned off, you can re-enable it in the gcr.io/datadoghq/agent
container by passing DD_APM_ENABLED=true
as an environment variable.
The CLI commands on this page are for the Docker runtime. Replace docker
with nerdctl
for the containerd runtime, or podman
for the Podman runtime.
If you are collecting traces from a containerized app (your Agent and app running in separate containers), as an alternative to the following instructions, you can automatically inject the tracing library into your application. Read
Injecting Libraries for instructions.
Tracing from the host
Tracing is available on port 8126/tcp
from your host only by adding the option -p 127.0.0.1:8126:8126/tcp
to the docker run
command.
To make it available from any host, use -p 8126:8126/tcp
instead.
For example, the following command allows the Agent to receive traces from your host only:
docker run -d --cgroupns host \
--pid host \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-p 127.0.0.1:8126:8126/tcp \
-e DD_API_KEY=<DATADOG_API_KEY> \
-e DD_APM_ENABLED=true \
-e DD_SITE=<DATADOG_SITE> \
gcr.io/datadoghq/agent:latest
Where your <DATADOG_SITE>
is
(defaults to datadoghq.com
).
docker run -d -p 127.0.0.1:8126:8126/tcp \
-e DD_API_KEY=<DATADOG_API_KEY> \
-e DD_APM_ENABLED=true \
-e DD_SITE=<DATADOG_SITE> \
gcr.io/datadoghq/agent:latest
Where your <DATADOG_SITE>
is
(defaults to datadoghq.com
).
Docker APM Agent environment variables
List of all environment variables available for tracing within the Docker Agent:
Environment variable | Description |
---|
DD_API_KEY | Datadog API Key |
DD_PROXY_HTTPS | Set up the URL for the proxy to use. |
DD_APM_REPLACE_TAGS | Scrub sensitive data from your span’s tags. |
DD_APM_FILTER_TAGS_REQUIRE | Defines required tags that traces must have in order to be sent to Datadog. |
DD_APM_FILTER_TAGS_REJECT | Defines rejection tags. The Agent drops traces that have these tags. |
DD_HOSTNAME | Manually set the hostname to use for metrics if autodetection fails, or when running the Datadog Cluster Agent. |
DD_DOGSTATSD_PORT | Set the DogStatsD port. |
DD_APM_RECEIVER_SOCKET | Collect your traces through a Unix Domain Sockets and takes priority over hostname and port configuration if set. Off by default, when set it must point to a valid sock file. |
DD_BIND_HOST | Set the StatsD & receiver hostname. |
DD_LOG_LEVEL | Set the logging level. (trace /debug /info /warn /error /critical /off ) |
DD_APM_ENABLED | When set to true (the default), the Datadog Agent accepts traces and trace metrics. |
DD_APM_CONNECTION_LIMIT | Sets the maximum connection limit for a 30 second time window. The default limit is 2000 connections. |
DD_APM_DD_URL | Set the Datadog API endpoint where your traces are sent: https://trace.agent. . Defaults to https://trace.agent.datadoghq.com . |
DD_APM_RECEIVER_PORT | Port that the Datadog Agent’s trace receiver listens on. Default value is 8126 . |
DD_APM_NON_LOCAL_TRAFFIC | Allow non-local traffic when tracing from other containers. |
DD_APM_IGNORE_RESOURCES | Configure resources for the Agent to ignore. Format should be comma separated, regular expressions. Example: GET /ignore-me,(GET|POST) /and-also-me . |
Tracing from other containers
As with DogStatsD, traces can be submitted to the Agent from other containers either using Docker networks or with the Docker host IP.
Docker network
As a first step, create a user-defined bridge network:
docker network create <NETWORK_NAME>
The CLI commands on this page are for the Docker runtime. Replace docker
with nerdctl
for the containerd runtime, or podman
for the Podman runtime.
Then start the Agent and the application container, connected to the network previously created:
# Datadog Agent
docker run -d --name datadog-agent \
--network <NETWORK_NAME> \
--cgroupns host \
--pid host \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-e DD_API_KEY=<DATADOG_API_KEY> \
-e DD_APM_ENABLED=true \
-e DD_SITE=<DATADOG_SITE> \
-e DD_APM_NON_LOCAL_TRAFFIC=true \
gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
--network <NETWORK_NAME> \
-e DD_AGENT_HOST=datadog-agent \
company/app:latest
Where your <DATADOG_SITE>
is
(defaults to datadoghq.com
).
# Datadog Agent
docker run -d --name datadog-agent \
--cgroupns host \
--pid host \
--network "<NETWORK_NAME>" \
-e DD_API_KEY=<DATADOG_API_KEY> \
-e DD_APM_ENABLED=true \
-e DD_SITE=<DATADOG_SITE> \
-e DD_APM_NON_LOCAL_TRAFFIC=true \
gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
--network "<NETWORK_NAME>" \
-e DD_AGENT_HOST=datadog-agent \
company/app:latest
Where your <DATADOG_SITE>
is
(defaults to datadoghq.com
).
This exposes the hostname datadog-agent
in your app
container.
If you’re using docker-compose
, <NETWORK_NAME>
parameters are the ones defined under the networks
section of your docker-compose.yml
.
Your application tracers must be configured to submit traces to this address. Set environment variables with the DD_AGENT_HOST
as the Agent container name, and DD_TRACE_AGENT_PORT
as the Agent Trace port in your application containers. The example above uses host datadog-agent
and port 8126
(the default value so you don’t have to set it).
Alternately, see the examples below to set the Agent host manually in each supported language:
Either update the Java Agent configuration with environment variables:
DD_AGENT_HOST=datadog-agent \
DD_TRACE_AGENT_PORT=8126 \
java -javaagent:/path/to/the/dd-java-agent.jar -jar /your/app.jar
or through system properties:
java -javaagent:/path/to/the/dd-java-agent.jar \
-Ddd.agent.host=datadog-agent \
-Ddd.agent.port=8126 \
-jar /your/app.jar
from ddtrace import tracer
tracer.configure(
hostname='datadog-agent',
port=8126,
)
Datadog.configure do |c|
c.agent.host = 'datadog-agent'
c.agent.port = 8126
end
package main
import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
func main() {
tracer.Start(tracer.WithAgentAddr("datadog-agent:8126"))
defer tracer.Stop()
}
const tracer = require('dd-trace').init({
hostname: 'datadog-agent',
port: 8126
});
Set the environment variables before running your instrumented app:
# Environment variables
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
export CORECLR_PROFILER_PATH=<SYSTEM_DEPENDENT_PATH>
export DD_DOTNET_TRACER_HOME=/opt/datadog
# For containers
export DD_AGENT_HOST=datadog-agent
export DD_TRACE_AGENT_PORT=8126
# Start your application
dotnet example.dll
The value for the CORECLR_PROFILER_PATH
environment variable varies based on the system where the application is running:
Operating System and Process Architecture | CORECLR_PROFILER_PATH Value |
---|
Alpine Linux x64 | <APP_DIRECTORY>/datadog/linux-musl-x64/Datadog.Trace.ClrProfiler.Native.so |
Linux x64 | <APP_DIRECTORY>/datadog/linux-x64/Datadog.Trace.ClrProfiler.Native.so |
Linux ARM64 | <APP_DIRECTORY>/datadog/linux-arm64/Datadog.Trace.ClrProfiler.Native.so |
Windows x64 | <APP_DIRECTORY>\datadog\win-x64\Datadog.Trace.ClrProfiler.Native.dll |
Windows x86 | <APP_DIRECTORY>\datadog\win-x86\Datadog.Trace.ClrProfiler.Native.dll |
In the table above, <APP_DIRECTORY>
refers to the directory containing the application’s .dll
files.
Docker host IP
Agent container port 8126
should be linked to the host directly.
Configure your application tracer to report to the default route of this container (determine this using the ip route
command).
The following is an example for the Python Tracer, assuming 172.17.0.1
is the default route:
from ddtrace import tracer
tracer.configure(hostname='172.17.0.1', port=8126)
Unix Domain Socket (UDS)
To submit traces via socket, the socket should be mounted to the Agent container and your application container.
# Datadog Agent
docker run -d --name datadog-agent \
--network <NETWORK_NAME> \
--cgroupns host \
--pid host \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-v /var/run/datadog/:/var/run/datadog/ \
-e DD_API_KEY=<DATADOG_API_KEY> \
-e DD_APM_ENABLED=true \
-e DD_SITE=<DATADOG_SITE> \
-e DD_APM_NON_LOCAL_TRAFFIC=true \
-e DD_APM_RECEIVER_SOCKET=/var/run/datadog/apm.socket \
gcr.io/datadoghq/agent:latest
# Application
docker run -d --name app \
--network <NETWORK_NAME> \
-v /var/run/datadog/:/var/run/datadog/ \
-e DD_TRACE_AGENT_URL=unix:///var/run/datadog/apm.socket \
company/app:latest
Refer to the language-specific APM instrumentation docs for tracer settings.
Further Reading
Additional helpful documentation, links, and articles: