Kubernetes Trace Collection
New announcements from Dash: Incident Management, Continuous Profiler, and more! New announcements from Dash!

Kubernetes Trace Collection

In order to start collecting your application traces you must be running the Datadog Agent in your Kubernetes cluster.

Setup

To enable trace collection with your Agent, follow the instructions below:

  1. Configure the Datadog Agent to accept traces:

    • If you haven’t already, install the Helm chart.

    • Update your values.yaml file with the following APM configuration:

      datadog:
        ## @param apm - object - required
        ## Enable apm agent and provide custom configs
        #
        apm:
          ## @param enabled - boolean - optional - default: false
          ## Enable this to enable APM and tracing, on port 8126
          #
          enabled: true
      
    • Set your operating system. Add targetSystem: linux or targetSystem: windows to the top of your values.yaml.

    • Then, upgrade your Datadog Helm chart using the following command: helm upgrade -f values.yaml <RELEASE NAME> datadog/datadog. Don’t forget to set the API key in the YAML file. If you did not set your operating system in values.yaml, add --set targetSystem=linux or --set targetSystem=windows to this command.

    To enable APM trace collection, open the DaemonSet configuration file and edit the following:

    • Allow incoming data from port 8126 (forwarding traffic from the host to the agent):

       # (...)
            ports:
              # (...)
              - containerPort: 8126
                hostPort: 8126
                name: traceport
                protocol: TCP
       # (...)
      
    • If using an old agent version (7.17 or lower), in addition to the steps above, set the DD_APM_NON_LOCAL_TRAFFIC and DD_APM_ENABLED variable to true in your env section of the datadog.yaml trace Agent manifest:

       # (...)
            env:
              # (...)
              - name: DD_APM_ENABLED
                value: 'true'
              - name: DD_APM_NON_LOCAL_TRAFFIC
                value: "true"
       # (...)
      

    Update your datadog-agent.yaml manifest with:

    agent:
      image:
        name: "datadog/agent:latest"
      apm:
        enabled: true
    

    See the sample manifest with APM and metrics collection enabled for a complete example.

    Then apply the new configuration:

    $ kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml
    
    Note: On minikube, you may receive an Unable to detect the kubelet URL automatically error. In this case, set DD_KUBELET_TLS_VERIFY=false.

  2. Configure your application pods to pull the host IP in order to communicate with the Datadog Agent:

  • Automatically with the Datadog Admission Controller, or

  • Manually using the downward API to pull the host IP; the application container needs the DD_AGENT_HOST environment variable that points to status.hostIP.

        apiVersion: apps/v1
        kind: Deployment
         # ...
            spec:
              containers:
              - name: "<CONTAINER_NAME>"
                image: "<CONTAINER_IMAGE>"/"<TAG>"
                env:
                  - name: DD_AGENT_HOST
                    valueFrom:
                      fieldRef:
                        fieldPath: status.hostIP
    
  1. Configure your application tracers to emit traces: Point your application-level tracers to where the Datadog Agent host is using the environment variable DD_AGENT_HOST. Refer to the language-specific APM instrumentation docs for more examples.

Agent Environment Variables

Note: As a best practice, Datadog recommends using unified service tagging when assigning tags. Unified service tagging ties Datadog telemetry together through the use of three standard tags: env, service, and version. To learn how to configure your environment with unified tagging, refer to the dedicated unified service tagging documentation.

List of all environment variables available for tracing within the Agent running in Kubernetes:

Environment variableDescription
DD_API_KEYDatadog API Key
DD_PROXY_HTTPSSet up the URL for the proxy to use.
DD_APM_REPLACE_TAGSScrub sensitive data from your span’s tags.
DD_HOSTNAMEManually set the hostname to use for metrics if autodection fails, or when running the Datadog Cluster Agent.
DD_DOGSTATSD_PORTSet the DogStatsD port.
DD_APM_RECEIVER_SOCKETCollect your traces through a Unix Domain Sockets and takes priority over hostname and port configuration if set. Off by default, when set it must point to a valid sock file.
DD_BIND_HOSTSet the StatsD & receiver hostname.
DD_LOG_LEVELSet the logging level. (trace/debug/info/warn/error/critical/off)
DD_APM_ENABLEDWhen set to true, the Datadog Agent accepts trace metrics. Default value is true (Agent 7.18+)
DD_APM_CONNECTION_LIMITSets the maximum connection limit for a 30 second time window.
DD_APM_DD_URLDatadog API endpoint where traces are sent. For Datadog EU site set DD_APM_DD_URL to https://trace.agent.datadoghq.eu
DD_APM_RECEIVER_PORTPort that the Datadog Agent’s trace receiver listens on. Default value is 8126.
DD_APM_NON_LOCAL_TRAFFICAllow non-local traffic when tracing from other containers. Default value is true (Agent 7.18+)
DD_APM_IGNORE_RESOURCESConfigure resources for the Agent to ignore. Format should be comma separated, regular expressions. i.e. GET /ignore-me,(GET|POST) /and-also-me.
DD_APM_ANALYZED_SPANSConfigure the spans to analyze for transactions. Format should be comma separated instances of <SERVICE_NAME>|;<OPERATION_NAME>=1. i.e. my-express-app|;express.request=1,my-dotnet-app|;aspnet_core_mvc.request=1. You can also enable it automatically with the configuration parameter in the Tracing Client.
DD_ENVSets the global env for all data emitted by the Agent. If env is not present in your trace data, this variable will be used. See APM environment setup for more details.
DD_APM_MAX_EPSSets the maximum Indexed Spans per second. Default is 200 events per second.
DD_APM_MAX_TPSSets the maximum traces per second. Default is 10 traces per second.

Operator environment variables

Environment variableDescription
agent.apm.enabledEnable this to enable APM and tracing, on port 8126. See the Datadog Docker documentation.
agent.apm.envThe Datadog Agent supports many environment variables.
agent.apm.hostPortNumber of port to expose on the host. If specified, this must be a valid port number, 0 < x < 65536. If HostNetwork is specified, this must match ContainerPort. Most containers do not need this.
agent.apm.resources.limitsLimits describes the maximum amount of compute resources allowed. For more info, see the Kubernetes documentation.
agent.apm.resources.requestsRequests describes the minimum amount of compute resources required. If requests is omitted for a container, it defaults to limits if that is explicitly specified, otherwise to an implementation-defined value. For more info, see the Kubernetes documentation.

Further Reading