Kubernetes APM - Trace Collection

• If you haven’t already, install the Helm chart.

The default configuration creates a directory on the host and mounts it within the Agent. The Agent then creates and listens on a socket file /var/run/datadog/apm.socket. The application pods can then similarly mount this volume and write to this same socket. You can modify the path and socket with the datadog.apm.hostSocketPath and datadog.apm.socketPath configuration values.

This feature can be disabled with datadog.apm.socketEnabled.

Optional - Configure the Datadog Agent to accept traces over TCP

The Datadog Agent can also be configured to receive traces over TCP. To enable this feature:

• Update your values.yaml file with the following APM configuration:

  datadog:
    ## Enable apm agent and provide custom configs
    apm:
      # datadog.apm.portEnabled -- Enable APM over TCP communication (port 8126 by default)
      ## ref: https://docs.datadoghq.com/agent/kubernetes/apm/
      portEnabled: true
  

Then, upgrade your Datadog Helm chart using the following command: helm upgrade -f values.yaml <RELEASE NAME> datadog/datadog. If you did not set your operating system in values.yaml, add --set targetSystem=linux or --set targetSystem=windows to this command.

Warning: The datadog.apm.portEnabled parameter opens a port on your host. Make sure your firewall only allows access from your applications or trusted sources. If your network plugin doesn’t support hostPorts, add hostNetwork: true in your Agent pod specifications. This shares the network namespace of your host with the Datadog Agent. This also means that all ports opened on the container are opened on the host. If a port is used both on the host and in your container, they conflict (since they share the same network namespace) and the pod does not start. Some Kubernetes installations do not allow this.

To enable APM trace collection, open the DaemonSet configuration file and edit the following:

• Allow incoming data from port 8126 (forwarding traffic from the host to the agent) within the trace-agent container:

    # (...)
    containers:
      - name: trace-agent
        # (...)
        ports:
          - containerPort: 8126
            hostPort: 8126
            name: traceport
            protocol: TCP
    # (...)
  

• If using an old agent version (7.17 or lower), in addition to the steps above, set the DD_APM_NON_LOCAL_TRAFFIC and DD_APM_ENABLED variable to true in your env section of the datadog.yaml trace Agent manifest:

    # (...)
    containers:
      - name: trace-agent
        # (...)
        env:
          - name: DD_APM_ENABLED
            value: 'true'
          - name: DD_APM_NON_LOCAL_TRAFFIC
            value: "true"
          # (...)
  

Warning: The hostPort parameter opens a port on your host. Make sure your firewall only allows access from your applications or trusted sources. If your network plugin doesn’t support hostPorts, add hostNetwork: true in your Agent pod specifications. This shares the network namespace of your host with the Datadog Agent. This also means that all ports opened on the container are opened on the host. If a port is used both on the host and in your container, they conflict (since they share the same network namespace) and the pod does not start. Some Kubernetes installations do not allow this.

To enable APM trace collection, open the DaemonSet configuration file and edit the following:

  # (...)
  containers:
  - name: trace-agent
    # (...)
    env:
    - name: DD_APM_ENABLED
      value: "true"
    - name: DD_APM_RECEIVER_SOCKET
      value: "/var/run/datadog/apm.socket"
  # (...)
    volumeMounts:
    - name: apmsocket
      mountPath: /var/run/datadog/
  volumes:
  - hostPath:
      path: /var/run/datadog/
      type: DirectoryOrCreate
  # (...)

This configuration creates a directory on the host and mounts it within the Agent. The Agent then creates and listens on a socket file in that directory with the DD_APM_RECEIVER_SOCKET value of /var/run/datadog/apm.socket. The application pods can then similarly mount this volume and write to this same socket.

When APM is enabled, the default configuration creates a directory on the host and mounts it within the Agent. The Agent then creates and listens on a socket file /var/run/datadog/apm/apm.socket. The application pods can then similarly mount this volume and write to this same socket. You can modify the path and socket with the features.apm.unixDomainSocketConfig.path configuration value.

Optional - Configure the Datadog Agent to accept traces over TCP

The Datadog Agent can also be configured to receive traces over TCP. To enable this feature:

Update your DatadogAgent manifest with the following:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>
    site: <DATADOG_SITE>

  features:
    apm:
      enabled: true
      hostPortConfig:
        enabled: true

Where your <DATADOG_SITE> is (defaults to datadoghq.com).

See the sample manifest with APM and metrics collection enabled for a complete example.

Then apply the new configuration:

kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml

Warning: The hostPort parameter opens a port on your host. Make sure your firewall only allows access from your applications or trusted sources. If your network plugin doesn’t support hostPorts, add hostNetwork: true in your Agent pod specifications. This shares the network namespace of your host with the Datadog Agent. This also means that all ports opened on the container are opened on the host. If a port is used both on the host and in your container, they conflict (since they share the same network namespace) and the pod does not start. Some Kubernetes installations do not allow this.

The Datadog Admission Controller is a component of the Datadog Cluster Agent that simplifies your application pod configuration. Learn more by reading the Datadog Admission Controller documentation.

Use the Datadog Admission Controller to inject environment variables and mount the necessary volumes on new application pods, automatically configuring pod and Agent trace communication. Learn how to automatically configure your application to submit traces to Datadog Agent by reading the Injecting Libraries Using Admission Controller documentation.

If you are sending traces to the Agent by using Unix Domain Socket (UDS), mount the host directory the socket is in (that the Agent created) to the application container and specify the path to the socket with DD_TRACE_AGENT_URL:

apiVersion: apps/v1
kind: Deployment
#(...)
    spec:
      containers:
      - name: "<CONTAINER_NAME>"
        image: "<CONTAINER_IMAGE>/<TAG>"
        env:
        - name: DD_TRACE_AGENT_URL
          value: 'unix:///var/run/datadog/apm.socket'
        volumeMounts:
        - name: apmsocketpath
          mountPath: /var/run/datadog
        #(...)
      volumes:
        - hostPath:
            path: /var/run/datadog/
          name: apmsocketpath

Configure your application tracers to emit traces:

After configuring your Datadog Agent to collect traces and giving your application pods the configuration on where to send traces, install the Datadog tracer into your applications to emit the traces. Once this is done, the tracer sends the traces to the appropriate DD_AGENT_HOST (for IP:Port) or DD_TRACE_AGENT_URL (for UDS) endpoint.

If you are sending traces to the Agent by using TCP (<IP_ADDRESS>:8126) supply this IP address to your application pods—either automatically with the Datadog Admission Controller, or manually using the downward API to pull the host IP. The application container needs the DD_AGENT_HOST environment variable that points to status.hostIP:

apiVersion: apps/v1
kind: Deployment
#(...)
    spec:
      containers:
      - name: "<CONTAINER_NAME>"
        image: "<CONTAINER_IMAGE>/<TAG>"
        env:
          - name: DD_AGENT_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP

Note: This configuration requires the Agent to be configured to accept traces over TCP

Configure your application tracers to emit traces:

After configuring your Datadog Agent to collect traces and giving your application pods the configuration on where to send traces, install the Datadog tracer into your applications to emit the traces. Once this is done, the tracer automatically sends the traces to the appropriate DD_AGENT_HOST (for IP:Port) or DD_TRACE_AGENT_URL (for UDS) endpoint.