Kubernetes Trace Collection

Kubernetes Trace Collection

In order to start collecting your application traces you must be running the Datadog Agent in your Kubernetes cluster.

Setup

You can configure the Agent to intake traces by using either UDP or Unix Domain Socket. To enable trace collection with your Agent:

  1. Configure the Datadog Agent to accept traces:

    • If you haven’t already, install the Helm chart.

    • Update your values.yaml file with the following APM configuration:

      datadog:
        ## @param apm - object - required
        ## Enable apm agent and provide custom configs
        #
        apm:
          ## @param enabled - boolean - optional - default: false
          ## Enable this to enable APM and tracing, on port 8126
          #
          enabled: true
      
    • Set your operating system. Add targetSystem: linux or targetSystem: windows to the top of your values.yaml.

    • Set the API key: apiKey: <DATADOG_API_KEY>

    • Set your Datadog site to: (defaults to site: datadoghq.com).

    • Then, upgrade your Datadog Helm chart using the following command: helm upgrade -f values.yaml <RELEASE NAME> datadog/datadog. If you did not set your operating system in values.yaml, add --set targetSystem=linux or --set targetSystem=windows to this command.

    • If you haven’t already, install the Helm chart.

    • Update your values.yaml file with the following APM configuration:

      datadog:
        ## @param apm - object - required
        ## Enable apm agent and provide custom configs
        #
        apm:
          # datadog.apm.socketEnabled -- Enable APM over Socket (Unix Socket or windows named pipe)
          ## ref: https://docs.datadoghq.com/agent/kubernetes/apm/
          socketEnabled: true
          # datadog.apm.socketPath -- Path to the trace-agent socket
          socketPath: /var/run/datadog/apm.socket
      
    • Set your operating system. Add targetSystem: linux or targetSystem: windows to the top of your values.yaml.

    • Set the API key: apiKey: <DATADOG_API_KEY>

    • Set your Datadog site to: (defaults to site: datadoghq.com).

    • Then, upgrade your Datadog Helm chart using the following command: helm upgrade -f values.yaml <RELEASE NAME> datadog/datadog. If you did not set your operating system in values.yaml, add --set targetSystem=linux or --set targetSystem=windows to this command.

    To enable APM trace collection, open the DaemonSet configuration file and edit the following:

    • Allow incoming data from port 8126 (forwarding traffic from the host to the agent):

       # (...)
            ports:
              # (...)
              - containerPort: 8126
                hostPort: 8126
                name: traceport
                protocol: TCP
       # (...)
      
    • Set your Datadog site to: (defaults to site: datadoghq.com).

    • If using an old agent version (7.17 or lower), in addition to the steps above, set the DD_APM_NON_LOCAL_TRAFFIC and DD_APM_ENABLED variable to true in your env section of the datadog.yaml trace Agent manifest:

       # (...)
            env:
              # (...)
              - name: DD_APM_ENABLED
                value: 'true'
              - name: DD_APM_NON_LOCAL_TRAFFIC
                value: "true"
       # (...)
      

    To enable APM trace collection, open the DaemonSet configuration file and edit the following:

    ```yaml
     # (...)
        env:
        - name: DD_APM_ENABLED
          value: "true"
        - name: DD_APM_RECEIVER_SOCKET
          value: "/var/run/datadog/apm.socket"
     # (...)
        volumeMounts:
        - name: apmsocket
          mountPath: /var/run/datadog/
      volumes:
      - hostPath:
          path: /var/run/datadog/
          type: DirectoryOrCreate
     # (...)
    ```
    

    This configuration creates a directory on the host and mounts it to the Agent. It creates a socket file in that directory with the DD_APM_RECEIVER_SOCKET value. This allows incoming data from the socket and makes it available to the application containers to access on the host.

    Update your datadog-agent.yaml manifest with the following:

    agent:
      image:
        name: "gcr.io/datadoghq/agent:latest"
      apm:
        enabled: true
        hostPort: 8126
    site: <DATADOG_SITE>
    

    Where your <DATADOG_SITE> is (defaults to datadoghq.com).

    See the sample manifest with APM and metrics collection enabled for a complete example.

    Then apply the new configuration:

    $ kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml
    

    Update your datadog-agent.yaml manifest with the following:

    agent:
      image:
        name: "gcr.io/datadoghq/agent:latest"
      agent:
        apm:
          enabled: true
          unixDomainSocket:
            enabled: true
        config:
            unixDomainSocket:
              enabled: true
    site: <DATADOG_SITE>
    

    Where your <DATADOG_SITE> is (defaults to datadoghq.com).

    See the sample manifest with APM and metrics collection enabled for a complete example.

    Then apply the new configuration:

    $ kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml
    

    Note: On minikube, you may receive an Unable to detect the kubelet URL automatically error. In this case, set DD_KUBELET_TLS_VERIFY=false.

  2. Configure your application pods in order to communicate with the Datadog Agent:

    If you are sending traces to the Agent by using UDP (<IP_ADDRESS>:8126), either automatically with the Datadog Admission Controller, or manually using the downward API to pull the host IP, the application container needs the DD_AGENT_HOST environment variable that points to status.hostIP:

        apiVersion: apps/v1
        kind: Deployment
          # ...
            spec:
              containers:
              - name: "<CONTAINER_NAME>"
                image: "<CONTAINER_IMAGE>"/"<TAG>"
                env:
                  - name: DD_AGENT_HOST
                    valueFrom:
                      fieldRef:
                        fieldPath: status.hostIP
    

    If you are sending traces to the Agent by using Unix Domain Socket, mount the directory the socket is in to the application container and specify the path with DD_TRACE_AGENT_URL:

        apiVersion: apps/v1
        kind: Deployment
        # ...
            spec:
              containers:
              - name: "<CONTAINER_NAME>"
                image: "<CONTAINER_IMAGE>"/"<TAG>"
                imagePullPolicy: Never
                volumeMounts:
                - name: apmsocketpath
                  mountPath: /var/run/datadog
                ports:
                - containerPort: 9390
                env:
                - name: DD_TRACE_AGENT_URL
                  value: 'unix:///var/run/datadog/apm.socket'
                # ...
              volumes:
                - hostPath:
                    path: /var/run/datadog/
                  name: apmsocketpath
    

  3. Configure your application tracers to emit traces:

    Point your application-level tracers to where the Datadog Agent host is using the environment variable DD_AGENT_HOST. Refer to the language-specific APM instrumentation docs for more examples.

    Point your application-level tracers to where the Datadog Agent host is using the environment variable DD_APM_RECEIVER_SOCKET or for Helm apm.socketPath. For Helm and Operator the default for this value is "/var/run/datadog/apm.socket".

    Some tracers do not support Unix Domain Socket. Refer to the language-specific APM instrumentation docs to see if this is supported for your tracer.

Agent environment variables

Note: As a best practice, Datadog recommends using unified service tagging when assigning tags. Unified service tagging ties Datadog telemetry together through the use of three standard tags: env, service, and version. To learn how to configure your environment with unified tagging, refer to the dedicated unified service tagging documentation.

List of all environment variables available for tracing within the Agent running in Kubernetes:

Environment variableDescription
DD_API_KEYDatadog API Key
DD_PROXY_HTTPSSet up the URL for the proxy to use.
DD_APM_REPLACE_TAGSScrub sensitive data from your span’s tags.
DD_HOSTNAMEManually set the hostname to use for metrics if autodection fails, or when running the Datadog Cluster Agent.
DD_DOGSTATSD_PORTSet the DogStatsD port.
DD_APM_RECEIVER_SOCKETCollect your traces through a Unix Domain Sockets and takes priority over hostname and port configuration if set. Off by default, when set it must point to a valid sock file.
DD_BIND_HOSTSet the StatsD & receiver hostname.
DD_LOG_LEVELSet the logging level. (trace/debug/info/warn/error/critical/off)
DD_APM_ENABLEDWhen set to true, the Datadog Agent accepts trace metrics. Default value is true (Agent 7.18+)
DD_APM_CONNECTION_LIMITSets the maximum connection limit for a 30 second time window.
DD_APM_DD_URLSet the Datadog API endpoint where your traces are sent: https://trace.agent.. Defaults to https://trace.agent.datadoghq.com.
DD_APM_RECEIVER_PORTPort that the Datadog Agent’s trace receiver listens on. Default value is 8126.
DD_APM_NON_LOCAL_TRAFFICAllow non-local traffic when tracing from other containers. Default value is true (Agent 7.18+)
DD_APM_IGNORE_RESOURCESConfigure resources for the Agent to ignore. Format should be comma separated, regular expressions. Like GET /ignore-me,(GET|POST) /and-also-me.
DD_ENVSets the global env for all data emitted by the Agent. If env is not present in your trace data, this variable is used. See APM environment setup for more details.

Operator environment variables

Environment variableDescription
agent.apm.enabledEnable this to enable APM and tracing, on port 8126. See the Datadog Docker documentation.
agent.apm.envThe Datadog Agent supports many environment variables.
agent.apm.hostPortNumber of port to expose on the host. If specified, this must be a valid port number, 0 < x < 65536. If HostNetwork is specified, this must match ContainerPort. Most containers do not need this.
agent.apm.resources.limitsLimits describes the maximum amount of compute resources allowed. For more info, see the Kubernetes documentation.
agent.apm.resources.requestsRequests describes the minimum amount of compute resources required. If requests is omitted for a container, it defaults to limits if that is explicitly specified, otherwise to an implementation-defined value. For more info, see the Kubernetes documentation.

Further Reading