Kubernetes Trace Collection

In order to start collecting your application traces you must be running the Datadog Agent in your Kubernetes cluster.

Setup

To enable trace collection with your Agent, follow the instructions below:

1. Configure the Datadog Agent to accept traces:

   • Dropdown

   • If you haven’t already, install the Helm chart.

   • Update your values.yaml file with the following APM configuration:

     datadog:
       ## @param apm - object - required
       ## Enable apm agent and provide custom configs
       #
       apm:
         ## @param enabled - boolean - optional - default: false
         ## Enable this to enable APM and tracing, on port 8126
         #
         enabled: true
     

   • Set your operating system. Add targetSystem: linux or targetSystem: windows to the top of your values.yaml.

   • Set the API key: apiKey: <DATADOG_API_KEY>

   • Set your site so that the Agent sends data to the right Datadog location: site:

   • Then, upgrade your Datadog Helm chart using the following command: helm upgrade -f values.yaml <RELEASE NAME> datadog/datadog. If you did not set your operating system in values.yaml, add --set targetSystem=linux or --set targetSystem=windows to this command.

   To enable APM trace collection, open the DaemonSet configuration file and edit the following:

   • Allow incoming data from port 8126 (forwarding traffic from the host to the agent):

      # (...)
           ports:
             # (...)
             - containerPort: 8126
               hostPort: 8126
               name: traceport
               protocol: TCP
      # (...)
     

   • Ensure the Agent sends data to the correct Datadog location by setting site:

   • If using an old agent version (7.17 or lower), in addition to the steps above, set the DD_APM_NON_LOCAL_TRAFFIC and DD_APM_ENABLED variable to true in your env section of the datadog.yaml trace Agent manifest:

      # (...)
           env:
             # (...)
             - name: DD_APM_ENABLED
               value: 'true'
             - name: DD_APM_NON_LOCAL_TRAFFIC
               value: "true"
      # (...)
     

   Update your datadog-agent.yaml manifest with:

   agent:
     image:
       name: "gcr.io/datadoghq/agent:latest"
     apm:
       enabled: true
       hostPort: 8126
   

   agent:
     image:
       name: "gcr.io/datadoghq/agent:latest"
     apm:
       enabled: true
       hostPort: 8126
   site: <DATADOG_SITE>
   

   Where <DATADOG_SITE> is , so that the Agent sends data to the right Datadog location.

   See the sample manifest with APM and metrics collection enabled for a complete example.

   Then apply the new configuration:

   $ kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml
   

   Note: On minikube, you may receive an Unable to detect the kubelet URL automatically error. In this case, set DD_KUBELET_TLS_VERIFY=false.

2. Configure your application pods to pull the host IP in order to communicate with the Datadog Agent:

• Automatically with the Datadog Admission Controller, or

• Manually using the downward API to pull the host IP; the application container needs the DD_AGENT_HOST environment variable that points to status.hostIP.

      apiVersion: apps/v1
      kind: Deployment
       # ...
          spec:
            containers:
            - name: "<CONTAINER_NAME>"
              image: "<CONTAINER_IMAGE>"/"<TAG>"
              env:
                - name: DD_AGENT_HOST
                  valueFrom:
                    fieldRef:
                      fieldPath: status.hostIP
  

3. Configure your application tracers to emit traces: Point your application-level tracers to where the Datadog Agent host is using the environment variable DD_AGENT_HOST. Refer to the language-specific APM instrumentation docs for more examples.

Agent environment variables

Note: As a best practice, Datadog recommends using unified service tagging when assigning tags. Unified service tagging ties Datadog telemetry together through the use of three standard tags: env, service, and version. To learn how to configure your environment with unified tagging, refer to the dedicated unified service tagging documentation.

List of all environment variables available for tracing within the Agent running in Kubernetes:

Operator environment variables

Further Reading

Additional helpful documentation, links, and articles:

Collect your application logsDOCUMENTATION Collect your Prometheus metricsDOCUMENTATION Collect automatically your applications metrics and logsDOCUMENTATION Limit data collection to a subset of containers onlyDOCUMENTATION Assign tags to all data emitted by a containerDOCUMENTATION