Kubernetes trace collection
Datadog's Research Report: The State of Serverless Report: The State of Serverless

Kubernetes trace collection

In order to start collecting your application traces you must be running the Datadog Agent in your Kubernetes cluster.

Setup

To enable trace collection with your Agent, follow the instructions below:

  1. Configure the Datadog Agent to accept traces:

    • If you haven’t already, install the Helm chart.
    • Update your datadog-values.yaml file with the following APM configuration:

      datadog:
        ## @param apm - object - required
        ## Enable apm agent and provide custom configs
        #
        apm:
          ## @param enabled - boolean - optional - default: false
          ## Enable this to enable APM and tracing, on port 8126
          #
          enabled: true
      • Then, upgrade your Datadog Helm chart using the following command : helm upgrade -f datadog-values.yaml <RELEASE NAME> stable/datadog (don’t forget to set the API key in the yaml file)

    To enable APM trace collection, open the Daemonset configuration file and edit the following:

    • Allow incoming data from port 8126 (forwarding traffic from the host to the agent):

       # (...)
            ports:
              # (...)
              - containerPort: 8126
                hostPort: 8126
                name: traceport
                protocol: TCP
       # (...)
    • If using an old agent version (7.17 or lower), in addition to the steps above, set the DD_APM_NON_LOCAL_TRAFFIC and DD_APM_ENABLED variable to true in your env section of the datadog.yaml Agent manifest:

       # (...)
            env:
              # (...)
              - name: DD_APM_ENABLED
                value: 'true'
              - name: DD_APM_NON_LOCAL_TRAFFIC
                value: "true"
       # (...)
    Note: On minikube, you may receive an Unable to detect the kubelet URL automatically error. In this case, set DD_KUBELET_TLS_VERIFY=false.

  2. Configure your application pods to pull the host IP in order to communicate with the Datadog Agent: Use the downward API to pull the host IP; the application container needs the DD_AGENT_HOST environment variable that points to status.hostIP.

        apiVersion: apps/v1
        kind: Deployment
         # ...
            spec:
              containers:
              - name: "<CONTAINER_NAME>"
                image: "<CONTAINER_IMAGE>"/"<TAG>"
                env:
                  - name: DD_AGENT_HOST
                    valueFrom:
                      fieldRef:
                        fieldPath: status.hostIP
  3. Configure your application tracers to emit traces: Point your application-level tracers to where the Datadog Agent host is using the environment variable DD_AGENT_HOST. Refer to the language-specific APM instrumentation docs for more examples.

Agent Environment Variables

List of all environment variables available for tracing within the Agent running in Kubernetes:

Environment variableDescription
DD_API_KEYDatadog API Key
DD_PROXY_HTTPSSet up the URL for the proxy to use.
DD_APM_REPLACE_TAGSScrub sensitive data from your span’s tags.
DD_HOSTNAMEManually set the hostname to use for metrics if autodection fails, or when running the Datadog Cluster Agent.
DD_DOGSTATSD_PORTSet the DogStatsD port.
DD_APM_RECEIVER_SOCKETCollect your traces through a Unix Domain Sockets and takes priority over hostname and port configuration if set. Off by default, when set it must point to a valid sock file.
DD_BIND_HOSTSet the StatsD & receiver hostname.
DD_LOG_LEVELSet the logging level. (trace/debug/info/warn/error/critical/off)
DD_APM_ENABLEDWhen set to true, the Datadog Agent accepts trace metrics. Default value is true (Agent 7.18+)
DD_APM_CONNECTION_LIMITSets the maximum connection limit for a 30 second time window.
DD_APM_DD_URLDatadog API endpoint where traces are sent. For Datadog EU site set DD_APM_DD_URL to https://trace.agent.datadoghq.eu
DD_APM_RECEIVER_PORTPort that the Datadog Agent’s trace receiver listens on. Default value is 8126.
DD_APM_NON_LOCAL_TRAFFICAllow non-local traffic when tracing from other containers. Default value is true (Agent 7.18+)
DD_APM_IGNORE_RESOURCESConfigure resources for the Agent to ignore. Format should be comma separated, regular expressions. i.e. GET /ignore-me,(GET|POST) /and-also-me.
DD_APM_ANALYZED_SPANSConfigure the spans to analyze for transactions. Format should be comma separated instances of <SERVICE_NAME>|;<OPERATION_NAME>=1. i.e. my-express-app|;express.request=1,my-dotnet-app|;aspnet_core_mvc.request=1. You can also enable it automatically with the configuration parameter in the Tracing Client.
DD_APM_ENVSets the default environment for your traces.
DD_APM_MAX_EPSSets the maximum Analyzed Spans per second.
DD_APM_MAX_TPSSets the maximum traces per second.

Further Reading

Additional helpful documentation, links, and articles: