--- isPrivate: true title: Continuous Tracing with GPU Monitoring description: Enable GPU activity tracing for selected Kubernetes workloads. breadcrumbs: Docs > GPU Monitoring > Continuous Tracing with GPU Monitoring --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Continuous Tracing with GPU Monitoring {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} {% callout %} # Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com {% callout %} ##### Join the Preview! Continuous tracing with GPU Monitoring is in Early Access Preview. {% /callout %} {% /callout %} ## Overview{% #overview %} Continuous tracing with GPU Monitoring enables lightweight GPU activity tracing for selected Kubernetes workloads. Troubleshooting large, distributed workloads can be cumbersome and time-consuming. With this tracing capability in GPU Monitoring, you can identify and investigate bottlenecks using detailed execution traces that tie CUDA and NCCL operations back to your model and PyTorch operations. {% image source="https://docs.dd-static.net/images/gpu_monitoring/gpu-tracing.6f88537d9fc3469f7ce211677fb451a0.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/gpu-tracing.6f88537d9fc3469f7ce211677fb451a0.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Flame graph view of a torch.step trace, showing CPU spans aligned with GPU stream activity, including NCCL allgather operations and CUDA kernel launches." /%} ## Setup{% #setup %} ### Prerequisites{% #prerequisites %} To begin continuously tracing your workloads, you must first meet the following criteria: - You are running the Datadog Cluster Agent version 7.80 or later with [GPU Monitoring enabled](https://docs.datadoghq.com/gpu_monitoring/setup.md). - Minimum required CUDA and CUPTI version: 13. ### 1. Configure GPU tracing {% tab title="Datadog Operator" %} Merge the following configuration into the existing `DatadogAgent` resource: ```yaml spec: features: apm: enabled: true instrumentation: enabled: true targets: - name: gpu-monitoring podSelector: matchLabels: admission.datadoghq.com/gpu.enabled: "true" ddTraceVersions: c: "0" ddTraceConfigs: - name: DD_INJECT_NATIVE value: "always" - name: DD_TRACE_HOOK_MODULES value: "gpu" ``` Apply the configuration and wait for the `DatadogAgent` rollout to complete. {% /tab %} {% tab title="Helm" %} Merge the following configuration into your existing `datadog-values.yaml` file: ```yaml datadog: apm: instrumentation: enabled: true targets: - name: gpu-monitoring podSelector: matchLabels: admission.datadoghq.com/gpu.enabled: "true" ddTraceVersions: c: "0" ddTraceConfigs: - name: DD_INJECT_NATIVE value: "always" - name: DD_TRACE_HOOK_MODULES value: "gpu" ``` Upgrade your Helm release and wait for the rollout to complete: ```shell helm upgrade -f datadog-values.yaml datadog/datadog ``` {% /tab %} ### 2. Label the GPU workload Add the label to the controller's pod template. The workload must be outside the Agent namespace. For Jobs, use `spec.template.metadata.labels`. For KubeRay, label the head and worker pod template: ```yaml spec: template: metadata: labels: admission.datadoghq.com/gpu.enabled: "true" ``` Apply the resource and wait for the rollout to complete. ### 3. Verify setup ```shell # Confirm setup containers completed kubectl get pod -n \ -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{" exit="}{.state.terminated.exitCode}{"\n"}{end}' # Confirm the GPU tracing settings kubectl exec -n -- sh -c \ 'env | grep -E "^(DD_INJECT_NATIVE|DD_TRACE_HOOK_MODULES|DD_SERVICE|DD_ENV)="' ``` **No setup containers?** Confirm the label is on the pod template, the pod is new, and the workload is outside the Agent namespace. Then check the Cluster Agent logs. ### 4. Explore traces Run the workload, then query [APM Trace Explorer](https://docs.datadoghq.com/tracing/trace_explorer.md) with: ``` pod_name: kube_namespace: ``` ## Connect training runs to GPU hardware{% #connect-training-runs-to-gpu-hardware %} If your Kubernetes workloads use labels or annotations to identify a training run or a group of runs, you can add those identifiers as tags on GPU metrics and spans. These tags tie training run data directly to the GPU hardware it ran on. The following examples use the `company.name/run-id` and `company.name/group-id` pod annotations. Replace them with the annotations your workloads use. For metrics, use [tag extraction](https://docs.datadoghq.com/containers/kubernetes/tag.md#tag-extraction) to map the annotations to tags: {% tab title="Datadog Operator" %} Merge the following configuration into the existing `DatadogAgent` resource: ```yaml spec: global: kubernetesResourcesAnnotationsAsTags: pods: company.name/run-id: training_run_id company.name/group-id: training_group_id ``` {% /tab %} {% tab title="Helm" %} Merge the following configuration into your existing `datadog-values.yaml` file: ```yaml datadog: kubernetesResourcesAnnotationsAsTags: pods: company.name/run-id: training_run_id company.name/group-id: training_group_id ``` {% /tab %} For traces, add `DD_TRAINING_RUN_ID` and `DD_TRAINING_GROUP_ID` to the `ddTraceConfigs` block in Step 1: Configure GPU tracing. Set each variable from the same annotations: ```yaml ddTraceConfigs: - name: DD_INJECT_NATIVE value: "always" - name: DD_TRACE_HOOK_MODULES value: "gpu" - name: DD_TRAINING_RUN_ID valueFrom: fieldRef: fieldPath: metadata.annotations['company.name/run-id'] - name: DD_TRAINING_GROUP_ID valueFrom: fieldRef: fieldPath: metadata.annotations['company.name/group-id'] ``` To use pod labels instead of annotations, set `kubernetesResourcesLabelsAsTags` for metrics, and set `fieldPath` to `metadata.labels['']` for traces. After you apply the configuration, GPU metrics are tagged with `training_run_id` and `training_group_id`, and spans are tagged with `training.run_id` and `training.group_id`. Use these tags to filter GPU metrics and traces for the same training run. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md) - [Learn more about what GPU Monitoring offers](https://docs.datadoghq.com/gpu_monitoring.md)