---
title: Continuous Tracing with GPU Monitoring
description: >-
  Enable GPU activity tracing for selected Kubernetes workloads and Slurm batch
  jobs.
breadcrumbs: Docs > GPU Monitoring > Continuous Tracing with GPU Monitoring
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Continuous Tracing with GPU Monitoring

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Continuous tracing with GPU Monitoring enables lightweight GPU activity tracing for selected Kubernetes workloads and Slurm batch jobs. Troubleshooting large, distributed workloads can be cumbersome and time-consuming. With this tracing capability in GPU Monitoring, you can identify and investigate bottlenecks using detailed execution traces that tie CUDA and NCCL operations back to your model and PyTorch operations.

{% image
   source="https://docs.dd-static.net/images/gpu_monitoring/gpu-tracing.6f88537d9fc3469f7ce211677fb451a0.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/gpu-tracing.6f88537d9fc3469f7ce211677fb451a0.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Flame graph view of a torch.step trace, showing CPU spans aligned with GPU stream activity, including NCCL allgather operations and CUDA kernel launches." /%}

With continuous tracing, you can increase the throughput of your AI workloads and get more out of every GPU:

- **Speed up training runs**: Find the communication, data loading, and kernel bottlenecks that slow down each training step. Use traces with the [Training page](https://docs.datadoghq.com/gpu_monitoring/training.md) to see how each training run used its GPUs.
- **Serve more requests per GPU**: See where inference requests spend time on the GPU. Use traces with the [Inference page](https://docs.datadoghq.com/gpu_monitoring/inference.md) to compare latency and throughput across your model deployments.
- **Improve GPU efficiency**: Spot idle and underused GPU time so you can right-size workloads and reduce wasted GPU hours.

## Setup{% #setup %}

{% alert level="info" %}
Continuous tracing requires GPU Monitoring. Before you configure tracing, [enable GPU Monitoring on your Datadog Agent](https://docs.datadoghq.com/gpu_monitoring/setup.md).
{% /alert %}

Select the tab that matches your environment:

{% tab title="Kubernetes" %}
### Prerequisites{% #prerequisites %}

To begin continuously tracing your workloads, you must be running the Datadog Cluster Agent version 7.80 or later.

### 1. Enable GPU Monitoring

If you haven't already, [set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md) on your GPU hosts. GPU tracing does not work unless GPU Monitoring is enabled on the Datadog Agent that runs on the same hosts as your GPU workloads.

### 2. Configure GPU tracing

In both configurations, the `ddTraceVersions.c` setting installs the Datadog C SDK, the native SDK that GPU tracing uses to capture CUDA and NCCL activity.

**Datadog Operator**: Merge the following configuration into the existing `DatadogAgent` resource:

```yaml
spec:
  features:
    apm:
      enabled: true
      instrumentation:
        enabled: true
        targets:
          - name: gpu-monitoring
            podSelector:
              matchLabels:
                admission.datadoghq.com/gpu.enabled: "true"
            ddTraceVersions:
              c: "0"
            ddTraceConfigs:
              - name: DD_INJECT_NATIVE
                value: "always"
              - name: DD_TRACE_HOOK_MODULES
                value: "gpu"
```

Apply the configuration and wait for the `DatadogAgent` rollout to complete.

**Helm**: Merge the following configuration into your existing `datadog-values.yaml` file:

```yaml
datadog:
  apm:
    instrumentation:
      enabled: true
      targets:
        - name: gpu-monitoring
          podSelector:
            matchLabels:
              admission.datadoghq.com/gpu.enabled: "true"
          ddTraceVersions:
            c: "0"
          ddTraceConfigs:
            - name: DD_INJECT_NATIVE
              value: "always"
            - name: DD_TRACE_HOOK_MODULES
              value: "gpu"
```

Upgrade your Helm release and wait for the rollout to complete:

```shell
helm upgrade -f datadog-values.yaml <RELEASE_NAME> datadog/datadog
```

### 3. Label the GPU workload

Add the label to the controller's pod template. The workload must be outside the Agent namespace. For Jobs, use `spec.template.metadata.labels`. For KubeRay, label the head and worker pod template:

```yaml
spec:
  template:
    metadata:
      labels:
        admission.datadoghq.com/gpu.enabled: "true"
```

Apply the resource and wait for the rollout to complete.

### 4. Verify setup

```shell
# Confirm setup containers completed
kubectl get pod <NEW_GPU_POD> -n <GPU_WORKLOAD_NAMESPACE> \
  -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{" exit="}{.state.terminated.exitCode}{"\n"}{end}'

# Confirm the GPU tracing settings
kubectl exec <NEW_GPU_POD> -n <GPU_WORKLOAD_NAMESPACE> -- sh -c \
  'env | grep -E "^(DD_INJECT_NATIVE|DD_TRACE_HOOK_MODULES|DD_SERVICE|DD_ENV)="'
```

**No setup containers?** Confirm the label is on the pod template, the pod is new, and the workload is outside the Agent namespace. Then check the Cluster Agent logs.

### 5. Explore traces

Run the workload, then query [APM Trace Explorer](https://docs.datadoghq.com/tracing/trace_explorer.md) with:

```
pod_name:<NEW_GPU_POD> kube_namespace:<GPU_WORKLOAD_NAMESPACE>
```

{% /tab %}

{% tab title="Slurm" %}
### 1. Enable GPU Monitoring

If you haven't already, [set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md) on your Slurm compute nodes. GPU tracing does not work unless GPU Monitoring is enabled on the Datadog Agent that runs on the same nodes as your Slurm jobs.

### 2. Install the Agent with Single Step Instrumentation

On each Slurm compute node, install the Datadog Agent with [Single Step Instrumentation](https://docs.datadoghq.com/tracing/trace_collection/single-step-apm.md) enabled. The following command enables Single Step Instrumentation for the host and installs the Datadog C SDK (`c:0`), the native SDK that GPU tracing uses to capture CUDA and NCCL activity from your jobs:

```shell
DD_API_KEY=<DATADOG_API_KEY> \
DD_SITE="<DATADOG_SITE>" \
DD_APM_INSTRUMENTATION_ENABLED=host \
DD_APM_INSTRUMENTATION_LIBRARIES=c:0 \
bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"
```

Replace `<DATADOG_API_KEY>` with your [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys) and `<DATADOG_SITE>` with your [Datadog site](https://docs.datadoghq.com/getting_started/site.md).

### 3. Enable the Slurm integration

Set up the [Slurm integration](https://docs.datadoghq.com/integrations/slurm.md) to collect metrics about your Slurm cluster, partitions, nodes, and jobs.

### 4. Configure your batch script

Add the following environment variables to your `sbatch` script:

```shell
export DD_SERVICE=${DD_SERVICE:-slurm-training}
export DD_ENV=${DD_ENV:-dev}
export DD_INJECT_NATIVE=always
export DD_TRACE_HOOK_MODULES=gpu
```

`DD_SERVICE` and `DD_ENV` default to `slurm-training` and `dev`. To use different values, set these variables before you submit the job, or change the defaults in the script.

For example:

```shell
#!/bin/bash
#SBATCH --job-name=training
#SBATCH --nodes=2
#SBATCH --gpus-per-node=8

export DD_SERVICE=${DD_SERVICE:-slurm-training}
export DD_ENV=${DD_ENV:-dev}
export DD_INJECT_NATIVE=always
export DD_TRACE_HOOK_MODULES=gpu

srun python train.py
```

### 5. Explore traces

Submit the job with `sbatch`, then query [APM Trace Explorer](https://docs.datadoghq.com/tracing/trace_explorer.md) with the service and environment you configured:

```
service:slurm-training env:dev
```

{% /tab %}

## Connect training runs to GPU hardware{% #connect-training-runs-to-gpu-hardware %}

If your Kubernetes workloads use labels or annotations to identify a training run or a group of runs, you can add those identifiers as tags on GPU metrics and spans. These tags tie training run data directly to the GPU hardware it ran on.

The following examples use the `company.name/run-id` and `company.name/group-id` pod annotations. Replace them with the annotations your workloads use.

For metrics, use [tag extraction](https://docs.datadoghq.com/containers/kubernetes/tag.md#tag-extraction) to map the annotations to tags:

{% tab title="Datadog Operator" %}
Merge the following configuration into the existing `DatadogAgent` resource:

```yaml
spec:
  global:
    kubernetesResourcesAnnotationsAsTags:
      pods:
        company.name/run-id: training_run_id
        company.name/group-id: training_group_id
```

{% /tab %}

{% tab title="Helm" %}
Merge the following configuration into your existing `datadog-values.yaml` file:

```yaml
datadog:
  kubernetesResourcesAnnotationsAsTags:
    pods:
      company.name/run-id: training_run_id
      company.name/group-id: training_group_id
```

{% /tab %}

For traces, add `DD_TRAINING_RUN_ID` and `DD_TRAINING_GROUP_ID` to the `ddTraceConfigs` block in Step 2: Configure GPU tracing. Set each variable from the same annotations:

```yaml
ddTraceConfigs:
  - name: DD_INJECT_NATIVE
    value: "always"
  - name: DD_TRACE_HOOK_MODULES
    value: "gpu"
  - name: DD_TRAINING_RUN_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.annotations['company.name/run-id']
  - name: DD_TRAINING_GROUP_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.annotations['company.name/group-id']
```

To use pod labels instead of annotations, set `kubernetesResourcesLabelsAsTags` for metrics, and set `fieldPath` to `metadata.labels['<LABEL_KEY>']` for traces.

After you apply the configuration, GPU metrics are tagged with `training_run_id` and `training_group_id`, and spans are tagged with `training.run_id` and `training.group_id`. Use these tags to filter GPU metrics and traces for the same training run.

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md)
- [Learn more about what GPU Monitoring offers](https://docs.datadoghq.com/gpu_monitoring.md)