Continuous Tracing with GPU Monitoring

This product is not supported for your selected Datadog site. ().

Overview

Continuous tracing with GPU Monitoring enables lightweight GPU activity tracing for selected Kubernetes workloads and Slurm batch jobs. Troubleshooting large, distributed workloads can be cumbersome and time-consuming. With this tracing capability in GPU Monitoring, you can identify and investigate bottlenecks using detailed execution traces that tie CUDA and NCCL operations back to your model and PyTorch operations.

Flame graph view of a torch.step trace, showing CPU spans aligned with GPU stream activity, including NCCL allgather operations and CUDA kernel launches.

With continuous tracing, you can increase the throughput of your AI workloads and get more out of every GPU:

  • Speed up training runs: Find the communication, data loading, and kernel bottlenecks that slow down each training step. Use traces with the Training page to see how each training run used its GPUs.
  • Serve more requests per GPU: See where inference requests spend time on the GPU. Use traces with the Inference page to compare latency and throughput across your model deployments.
  • Improve GPU efficiency: Spot idle and underused GPU time so you can right-size workloads and reduce wasted GPU hours.

Setup

Continuous tracing requires GPU Monitoring. Before you configure tracing, enable GPU Monitoring on your Datadog Agent.

Select the tab that matches your environment:

Prerequisites

To begin continuously tracing your workloads, you must be running the Datadog Cluster Agent version 7.80 or later.

1. Enable GPU Monitoring

If you haven’t already, set up GPU Monitoring on your GPU hosts. GPU tracing does not work unless GPU Monitoring is enabled on the Datadog Agent that runs on the same hosts as your GPU workloads.

2. Configure GPU tracing

In both configurations, the ddTraceVersions.c setting installs the Datadog C SDK, the native SDK that GPU tracing uses to capture CUDA and NCCL activity.

Datadog Operator: Merge the following configuration into the existing DatadogAgent resource:

spec:
  features:
    apm:
      enabled: true
      instrumentation:
        enabled: true
        targets:
          - name: gpu-monitoring
            podSelector:
              matchLabels:
                admission.datadoghq.com/gpu.enabled: "true"
            ddTraceVersions:
              c: "0"
            ddTraceConfigs:
              - name: DD_INJECT_NATIVE
                value: "always"
              - name: DD_TRACE_HOOK_MODULES
                value: "gpu"

Apply the configuration and wait for the DatadogAgent rollout to complete.

Helm: Merge the following configuration into your existing datadog-values.yaml file:

datadog:
  apm:
    instrumentation:
      enabled: true
      targets:
        - name: gpu-monitoring
          podSelector:
            matchLabels:
              admission.datadoghq.com/gpu.enabled: "true"
          ddTraceVersions:
            c: "0"
          ddTraceConfigs:
            - name: DD_INJECT_NATIVE
              value: "always"
            - name: DD_TRACE_HOOK_MODULES
              value: "gpu"

Upgrade your Helm release and wait for the rollout to complete:

helm upgrade -f datadog-values.yaml <RELEASE_NAME> datadog/datadog

3. Label the GPU workload

Add the label to the controller’s pod template. The workload must be outside the Agent namespace. For Jobs, use spec.template.metadata.labels. For KubeRay, label the head and worker pod template:

spec:
  template:
    metadata:
      labels:
        admission.datadoghq.com/gpu.enabled: "true"

Apply the resource and wait for the rollout to complete.

4. Verify setup

# Confirm setup containers completed
kubectl get pod <NEW_GPU_POD> -n <GPU_WORKLOAD_NAMESPACE> \
  -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{" exit="}{.state.terminated.exitCode}{"\n"}{end}'

# Confirm the GPU tracing settings
kubectl exec <NEW_GPU_POD> -n <GPU_WORKLOAD_NAMESPACE> -- sh -c \
  'env | grep -E "^(DD_INJECT_NATIVE|DD_TRACE_HOOK_MODULES|DD_SERVICE|DD_ENV)="'

No setup containers? Confirm the label is on the pod template, the pod is new, and the workload is outside the Agent namespace. Then check the Cluster Agent logs.

5. Explore traces

Run the workload, then query APM Trace Explorer with:

pod_name:<NEW_GPU_POD> kube_namespace:<GPU_WORKLOAD_NAMESPACE>

1. Enable GPU Monitoring

If you haven’t already, set up GPU Monitoring on your Slurm compute nodes. GPU tracing does not work unless GPU Monitoring is enabled on the Datadog Agent that runs on the same nodes as your Slurm jobs.

2. Install the Agent with Single Step Instrumentation

On each Slurm compute node, install the Datadog Agent with Single Step Instrumentation enabled. The following command enables Single Step Instrumentation for the host and installs the Datadog C SDK (c:0), the native SDK that GPU tracing uses to capture CUDA and NCCL activity from your jobs:

DD_API_KEY=<DATADOG_API_KEY> \
DD_SITE="<DATADOG_SITE>" \
DD_APM_INSTRUMENTATION_ENABLED=host \
DD_APM_INSTRUMENTATION_LIBRARIES=c:0 \
bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

Replace <DATADOG_API_KEY> with your Datadog API key and <DATADOG_SITE> with your Datadog site.

3. Enable the Slurm integration

Set up the Slurm integration to collect metrics about your Slurm cluster, partitions, nodes, and jobs.

4. Configure your batch script

Add the following environment variables to your sbatch script:

export DD_SERVICE=${DD_SERVICE:-slurm-training}
export DD_ENV=${DD_ENV:-dev}
export DD_INJECT_NATIVE=always
export DD_TRACE_HOOK_MODULES=gpu

DD_SERVICE and DD_ENV default to slurm-training and dev. To use different values, set these variables before you submit the job, or change the defaults in the script.

For example:

#!/bin/bash
#SBATCH --job-name=training
#SBATCH --nodes=2
#SBATCH --gpus-per-node=8

export DD_SERVICE=${DD_SERVICE:-slurm-training}
export DD_ENV=${DD_ENV:-dev}
export DD_INJECT_NATIVE=always
export DD_TRACE_HOOK_MODULES=gpu

srun python train.py

5. Explore traces

Submit the job with sbatch, then query APM Trace Explorer with the service and environment you configured:

service:slurm-training env:dev

Connect training runs to GPU hardware

If your Kubernetes workloads use labels or annotations to identify a training run or a group of runs, you can add those identifiers as tags on GPU metrics and spans. These tags tie training run data directly to the GPU hardware it ran on.

The following examples use the company.name/run-id and company.name/group-id pod annotations. Replace them with the annotations your workloads use.

For metrics, use tag extraction to map the annotations to tags:

Merge the following configuration into the existing DatadogAgent resource:

spec:
  global:
    kubernetesResourcesAnnotationsAsTags:
      pods:
        company.name/run-id: training_run_id
        company.name/group-id: training_group_id

Merge the following configuration into your existing datadog-values.yaml file:

datadog:
  kubernetesResourcesAnnotationsAsTags:
    pods:
      company.name/run-id: training_run_id
      company.name/group-id: training_group_id

For traces, add DD_TRAINING_RUN_ID and DD_TRAINING_GROUP_ID to the ddTraceConfigs block in Step 2: Configure GPU tracing. Set each variable from the same annotations:

ddTraceConfigs:
  - name: DD_INJECT_NATIVE
    value: "always"
  - name: DD_TRACE_HOOK_MODULES
    value: "gpu"
  - name: DD_TRAINING_RUN_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.annotations['company.name/run-id']
  - name: DD_TRAINING_GROUP_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.annotations['company.name/group-id']

To use pod labels instead of annotations, set kubernetesResourcesLabelsAsTags for metrics, and set fieldPath to metadata.labels['<LABEL_KEY>'] for traces.

After you apply the configuration, GPU metrics are tagged with training_run_id and training_group_id, and spans are tagged with training.run_id and training.group_id. Use these tags to filter GPU metrics and traces for the same training run.

Further reading