Continuous Tracing with GPU Monitoring
This product is not supported for your selected
Datadog site. (
).
Overview
Continuous tracing with GPU Monitoring enables lightweight GPU activity tracing for selected Kubernetes workloads and Slurm batch jobs. Troubleshooting large, distributed workloads can be cumbersome and time-consuming. With this tracing capability in GPU Monitoring, you can identify and investigate bottlenecks using detailed execution traces that tie CUDA and NCCL operations back to your model and PyTorch operations.
With continuous tracing, you can increase the throughput of your AI workloads and get more out of every GPU:
- Speed up training runs: Find the communication, data loading, and kernel bottlenecks that slow down each training step. Use traces with the Training page to see how each training run used its GPUs.
- Serve more requests per GPU: See where inference requests spend time on the GPU. Use traces with the Inference page to compare latency and throughput across your model deployments.
- Improve GPU efficiency: Spot idle and underused GPU time so you can right-size workloads and reduce wasted GPU hours.
Setup
Select the tab that matches your environment:
Prerequisites
To begin continuously tracing your workloads, you must be running the Datadog Cluster Agent version 7.80 or later.
1. Enable GPU Monitoring
If you haven’t already, set up GPU Monitoring on your GPU hosts. GPU tracing does not work unless GPU Monitoring is enabled on the Datadog Agent that runs on the same hosts as your GPU workloads.
In both configurations, the ddTraceVersions.c setting installs the Datadog C SDK, the native SDK that GPU tracing uses to capture CUDA and NCCL activity.
Datadog Operator: Merge the following configuration into the existing DatadogAgent resource:
spec:
features:
apm:
enabled: true
instrumentation:
enabled: true
targets:
- name: gpu-monitoring
podSelector:
matchLabels:
admission.datadoghq.com/gpu.enabled: "true"
ddTraceVersions:
c: "0"
ddTraceConfigs:
- name: DD_INJECT_NATIVE
value: "always"
- name: DD_TRACE_HOOK_MODULES
value: "gpu"
Apply the configuration and wait for the DatadogAgent rollout to complete.
Helm: Merge the following configuration into your existing datadog-values.yaml file:
datadog:
apm:
instrumentation:
enabled: true
targets:
- name: gpu-monitoring
podSelector:
matchLabels:
admission.datadoghq.com/gpu.enabled: "true"
ddTraceVersions:
c: "0"
ddTraceConfigs:
- name: DD_INJECT_NATIVE
value: "always"
- name: DD_TRACE_HOOK_MODULES
value: "gpu"
Upgrade your Helm release and wait for the rollout to complete:
helm upgrade -f datadog-values.yaml <RELEASE_NAME> datadog/datadog
3. Label the GPU workload
Add the label to the controller’s pod template. The workload must be outside the Agent namespace. For Jobs, use spec.template.metadata.labels. For KubeRay, label the head and worker pod template:
spec:
template:
metadata:
labels:
admission.datadoghq.com/gpu.enabled: "true"
Apply the resource and wait for the rollout to complete.
4. Verify setup
# Confirm setup containers completed
kubectl get pod <NEW_GPU_POD> -n <GPU_WORKLOAD_NAMESPACE> \
-o jsonpath='{range .status.initContainerStatuses[*]}{.name}{" exit="}{.state.terminated.exitCode}{"\n"}{end}'
# Confirm the GPU tracing settings
kubectl exec <NEW_GPU_POD> -n <GPU_WORKLOAD_NAMESPACE> -- sh -c \
'env | grep -E "^(DD_INJECT_NATIVE|DD_TRACE_HOOK_MODULES|DD_SERVICE|DD_ENV)="'
No setup containers? Confirm the label is on the pod template, the pod is new, and the workload is outside the Agent namespace. Then check the Cluster Agent logs.
5. Explore traces
Run the workload, then query APM Trace Explorer with:
pod_name:<NEW_GPU_POD> kube_namespace:<GPU_WORKLOAD_NAMESPACE>
1. Enable GPU Monitoring
If you haven’t already, set up GPU Monitoring on your Slurm compute nodes. GPU tracing does not work unless GPU Monitoring is enabled on the Datadog Agent that runs on the same nodes as your Slurm jobs.
2. Install the Agent with Single Step Instrumentation
On each Slurm compute node, install the Datadog Agent with Single Step Instrumentation enabled. The following command enables Single Step Instrumentation for the host and installs the Datadog C SDK (c:0), the native SDK that GPU tracing uses to capture CUDA and NCCL activity from your jobs:
DD_API_KEY=<DATADOG_API_KEY> \
DD_SITE="<DATADOG_SITE>" \
DD_APM_INSTRUMENTATION_ENABLED=host \
DD_APM_INSTRUMENTATION_LIBRARIES=c:0 \
bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"
Replace <DATADOG_API_KEY> with your Datadog API key and <DATADOG_SITE> with your Datadog site.
3. Enable the Slurm integration
Set up the Slurm integration to collect metrics about your Slurm cluster, partitions, nodes, and jobs.
Add the following environment variables to your sbatch script:
export DD_SERVICE=${DD_SERVICE:-slurm-training}
export DD_ENV=${DD_ENV:-dev}
export DD_INJECT_NATIVE=always
export DD_TRACE_HOOK_MODULES=gpu
DD_SERVICE and DD_ENV default to slurm-training and dev. To use different values, set these variables before you submit the job, or change the defaults in the script.
For example:
#!/bin/bash
#SBATCH --job-name=training
#SBATCH --nodes=2
#SBATCH --gpus-per-node=8
export DD_SERVICE=${DD_SERVICE:-slurm-training}
export DD_ENV=${DD_ENV:-dev}
export DD_INJECT_NATIVE=always
export DD_TRACE_HOOK_MODULES=gpu
srun python train.py
5. Explore traces
Submit the job with sbatch, then query APM Trace Explorer with the service and environment you configured:
service:slurm-training env:dev
Connect training runs to GPU hardware
If your Kubernetes workloads use labels or annotations to identify a training run or a group of runs, you can add those identifiers as tags on GPU metrics and spans. These tags tie training run data directly to the GPU hardware it ran on.
The following examples use the company.name/run-id and company.name/group-id pod annotations. Replace them with the annotations your workloads use.
For metrics, use tag extraction to map the annotations to tags:
Merge the following configuration into the existing DatadogAgent resource:
spec:
global:
kubernetesResourcesAnnotationsAsTags:
pods:
company.name/run-id: training_run_id
company.name/group-id: training_group_id
Merge the following configuration into your existing datadog-values.yaml file:
datadog:
kubernetesResourcesAnnotationsAsTags:
pods:
company.name/run-id: training_run_id
company.name/group-id: training_group_id
For traces, add DD_TRAINING_RUN_ID and DD_TRAINING_GROUP_ID to the ddTraceConfigs block in Step 2: Configure GPU tracing. Set each variable from the same annotations:
ddTraceConfigs:
- name: DD_INJECT_NATIVE
value: "always"
- name: DD_TRACE_HOOK_MODULES
value: "gpu"
- name: DD_TRAINING_RUN_ID
valueFrom:
fieldRef:
fieldPath: metadata.annotations['company.name/run-id']
- name: DD_TRAINING_GROUP_ID
valueFrom:
fieldRef:
fieldPath: metadata.annotations['company.name/group-id']
To use pod labels instead of annotations, set kubernetesResourcesLabelsAsTags for metrics, and set fieldPath to metadata.labels['<LABEL_KEY>'] for traces.
After you apply the configuration, GPU metrics are tagged with training_run_id and training_group_id, and spans are tagged with training.run_id and training.group_id. Use these tags to filter GPU metrics and traces for the same training run.
Further reading
Additional helpful documentation, links, and articles: