--- title: GPU Monitoring Fleet Page description: >- An inventory of all your GPU-accelerated hosts that helps you diagnose performance issues. breadcrumbs: Docs > GPU Monitoring > GPU Monitoring Fleet Page --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # GPU Monitoring Fleet Page {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} [GPU Fleet Explorer](https://app.datadoghq.com/gpu-monitoring?mConfigure=false&mPage=fleet) provides a detailed breakdown across AI infrastructure (such as GPU devices, hosts, or Ray clusters) and AI workloads (such as Pods, Kube Containers, or Training Runs) for a specified time frame. This page helps you uncover provisioning inefficiencies and workload performance optimizations through resource telemetry, including GPU utilization, host-level metrics, and costs. It also surfaces Datadog's real-time detection of issues impacting your fleet and workloads, with guidance on how to remediate them. ## Detect issues with out-of-the-box monitors{% #detect-issues-with-out-of-the-box-monitors %} Datadog provides several out-of-the-box (OOTB) monitor templates that detect common GPU issues in real time, including: - Temperature spikes - Power cap throttling - Unmet GPU requests - XID errors - ECC errors - Bursty workloads - Idle devices You can customize any monitor's thresholds to fit your organization's needs. To access these templates, click the Monitors dropdown in the top-right corner of the page. {% image source="https://docs.dd-static.net/images/gpu_monitoring/fleet-ootb-monitors.75fb59b22341b106eec44fb5f0e3f77f.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/fleet-ootb-monitors.75fb59b22341b106eec44fb5f0e3f77f.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="Monitors dropdown in the top-right corner of the GPU Fleet page, showing OOTB monitor templates for Temperature, Power Cap Throttling, Unmet GPU Requests, Critical XID Errors, General XID Errors, ECC Errors, Bursty Workloads, and Idle Devices" /%} ## Break down your fleet by any tag{% #break-down-your-fleet-by-any-tag %} GPU Fleet Explorer gives you visibility from your AI workloads down to the underlying AI infrastructure. You can switch between workload entities like pods and training runs, and infrastructure entities such as devices, hosts, and clusters. {% image source="https://docs.dd-static.net/images/gpu_monitoring/gpu-fleet-sidenav.2c11a75c1ca5e4bd8fa356d9add6e794.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/gpu-fleet-sidenav.2c11a75c1ca5e4bd8fa356d9add6e794.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="Side navigation bar showing AI Infrastructure entities (Devices, Hosts, Kube Clusters, Ray Clusters) and AI Workloads entities (Pods, Kube Containers, Training Runs)" /%} **Note**: Kube Clusters, Pods, and Kube Containers options in the side navigation are only available if you use Kubernetes. Use the filter dropdowns at the top of the page to filter by a specific Provider, Device Type, Cluster, Region, Service, Data Center, Environment, or Team. You can also Search or Group by other tags using the search and group-by fields. For example, you can group by Service to view a row in the table for each unique service. Click the \> button next to any entry to see the devices for that service. {% image source="https://docs.dd-static.net/images/gpu_monitoring/host_row_expansion-2.55e8c0d64d463126d23d77279547f459.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/host_row_expansion-2.55e8c0d64d463126d23d77279547f459.png?auto=format&fit=max&w=850&dpr=2 2x" alt="GPU Fleet table showing services with their device types, with the row expand button highlighted" /%} {% image source="https://docs.dd-static.net/images/gpu_monitoring/filters_and_groupings-3.42d57d9a1318bcfc3ef873abe9f06d8b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/filters_and_groupings-3.42d57d9a1318bcfc3ef873abe9f06d8b.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Filter dropdowns and Group by selector at the top of the GPU Fleet page" /%} ## Use case-driven views and recommendations{% #use-case-driven-views-and-recommendations %} GPU Monitoring's Fleet Explorer page provides two dedicated use case-driven views: - **Provisioning**: Allocate capacity and manage quotas. - **Performance**: Optimize workload efficiency and throughput. ### Provisioning{% #provisioning %} The Provisioning tab detects any hardware health issues impacting the allocation of your devices to workloads and provides guidance on how to remediate them. {% image source="https://docs.dd-static.net/images/gpu_monitoring/provisioning-tab-3.5824222dc4159e65cf06d542a725ece9.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/provisioning-tab-3.5824222dc4159e65cf06d542a725ece9.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="The Provisioning use case-driven view" /%} For each detected issue, Datadog recommends step-by-step remediation actions to help you resolve it. {% image source="https://docs.dd-static.net/images/gpu_monitoring/critical-xid-recommendation.281468b0f8b20ab547eaed032a29019e.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/critical-xid-recommendation.281468b0f8b20ab547eaed032a29019e.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="Recommended remediation actions for a critical XID error" /%} #### Provisioning summary graph{% #provisioning-summary-graph %} The summary graph provides out-of-the-box (OOTB) visualizations for key telemetry tied to your selected use case-driven view. For the Provisioning use case, this breaks down your Provisioned, Allocated, and Active devices so you can reclaim wasted idle spend and rediscover available devices that can be allocated to workloads. {% image source="https://docs.dd-static.net/images/gpu_monitoring/summary-graph.51cd0c1b25cc451b77324406774e451a.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/summary-graph.51cd0c1b25cc451b77324406774e451a.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="Summary graph showing Provisioned Devices, Allocated Devices, and Active Devices breakdowns" /%} Expand this section below to see a table of the available options and what they represent. {% collapsible-section #provisioning-summary-graph-table %} ##### See full list of Provisioning summary graph options | Option | Definition | | ------------------- | ------------------------------------------------------------ | | Provisioned Devices | Breakdown of provisioned devices and active devices. | | Allocated Devices | Breakdown of available devices by allocated vs. unallocated. | | Active Devices | Breakdown of allocated devices by active vs. idle. | {% /collapsible-section %} ### Performance{% #performance %} The Performance tab detects any hardware health or workload issues that are stalling the workloads running on your GPU devices. This provides a single source of truth and guidance to platform engineers and AI/ML teams on how to remediate those issues. {% image source="https://docs.dd-static.net/images/gpu_monitoring/performance-tab-3.a0c0b27c7fbcb4b6f902d066cf48a977.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/performance-tab-3.a0c0b27c7fbcb4b6f902d066cf48a977.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="The Performance use case-driven view" /%} For each detected issue, Datadog recommends step-by-step remediation actions to help you resolve it. {% image source="https://docs.dd-static.net/images/gpu_monitoring/power-cap-recommendation.d366b0611ffbd3546ed608df0a74665d.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/power-cap-recommendation.d366b0611ffbd3546ed608df0a74665d.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="Recommended remediation actions for a GPU power cap throttling issue" /%} #### Performance summary graph{% #performance-summary-graph %} The summary graph provides out-of-the-box (OOTB) visualizations for key telemetry tied to your selected use case-driven view. For the Performance use case, this breaks down key resource utilization metrics like GPU Utilization or GPU Saturation alongside network fabric metrics, power, temperature, and more. {% image source="https://docs.dd-static.net/images/gpu_monitoring/summary-graph-performance.260dd42233947fb3e2dffa08e0d177ee.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/summary-graph-performance.260dd42233947fb3e2dffa08e0d177ee.jpg?auto=format&fit=max&w=850&dpr=2 2x" alt="Summary graph showing GPU Util, GPU Saturation, and GPU Memory breakdowns" /%} Expand the section below to see a table of the available options and what they represent. {% collapsible-section #performance-summary-graph-table %} ##### See full list of Performance summary graph options | Option | Definition | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CPU Util | Percentage of time the CPU spent running user space processes. | | Host Memory | Percentage of usable memory in use. | | GPU Util | Average percentage of time each streaming multiprocessor was active (lower values indicate idle time). | | GPU Saturation | Measures how fully the GPU's parallel execution capacity is being used during the time frame (average ratio of active warps to the maximum warps supported per streaming multiprocessor across all SMs). | | GPU Memory | Percentage of GPU memory used to total GPU memory limit. | | PCIe RX | Bytes received through PCI from the GPU device per second. | | PCIe TX | Bytes transmitted through PCI to the GPU device per second. | | NVLink RX | Total RX of all NVLINK links. | | NVLink TX | Total TX of all NVLINK links. | | Graphics Activity | Fraction of time the GPU was performing any compute work during the interval. A coarse signal of whether the GPU is busy or idle. | | Power | Power usage for the GPU device. On GA100 and previous architectures, this represents the instantaneous power at that moment. For newer architectures, it represents the average power draw (Watts) over one second. | | Temperature | Temperature of a GPU device. | | SM Clock | SM clock frequency in MHz. | | NVLink Active Links | Number of active NVLINK links for the device. | | ECC Errors | Total count of uncorrected ECC errors. | | XID Errors | Count of NVIDIA XID errors, indicating hardware or driver-level issues. | {% /collapsible-section %} ## Inventory of your GPU-powered infrastructure{% #inventory-of-your-gpu-powered-infrastructure %} This table breaks down your GPU-powered infrastructure by any tag of your choosing. If you haven't specified an additional tag in the Group by field, results are grouped by your selected view: Cluster, Host, or Device. You can click on the gear icon to customize which metrics are displayed within the table. Expand the section below to see a full list of the available metrics. {% collapsible-section #metric-full-list %} ### See the full list of available metrics | Metric | Definition | Metric Name | Provisioning Tab | Performance Tab | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ---------------- | --------------- | | Idle Cost | (Only nonzero for time frames longer than 2 days) The cost of GPU resources that are reserved and allocated, but not used. | `gpu_monitoring.estimated_idle_cost` | ✓ | ✓ | | Total Devices | GPU devices with Datadog's GPU Monitoring correctly configured and reporting metrics. | `kubernetes_state.node.gpu_capacity` | ✓ | — | | Kubernetes Available | Healthy GPU devices that are powered on and available for allocation, according to the Kubernetes orchestrator. | `kubernetes_state.node.gpu_allocatable` | ✓ | — | | Allocated Devices | (Only available if using Kubernetes) Count of devices that have been allocated to a workload. | `gpu.device.total` | ✓ | — | | Unallocated Devices | Count of devices not allocated and available for use during time frame. | ✓ | — | | Active Devices | Count of devices that are actively used for a workload or are busy. If using Kubernetes: count of allocated devices that are actively used for a workload. | `gpu.gr_engine_active` | ✓ | — | | Idle Devices | GPU devices allocated to workloads but not doing any work during the time frame. A device is considered idle if `gpu.gr_engine_active` equals 0. | `gpu.gr_engine_active` | ✓ | — | | CPU Utilization | Percentage of time the CPU spent running user space processes. | `system.cpu.user` | — | ✓ | | Host Memory | Percentage of usable memory in use. | `system.mem.pct_usable` | — | ✓ | | GPU Utilization | Average percentage of time each streaming multiprocessor was active (lower values indicate idle time). | `gpu.sm_active` | — | ✓ | | GPU Saturation | Measures how fully the GPU's parallel execution capacity is being used during the time frame (average ratio of active warps to the maximum warps supported per streaming multiprocessor across all SMs). | `gpu.sm_occupancy` | — | ✓ | | GPU Memory | Percentage of GPU memory used to total GPU memory limit. | `100 - (gpu.memory.free / gpu.memory.limit * 100)` | — | ✓ | | PCIe RX Throughput | Bytes received through PCI from the GPU device per second. | `gpu.pci.throughput.rx` | — | ✓ | | PCIe TX Throughput | Bytes transmitted through PCI to the GPU device per second. | `gpu.pci.throughput.tx` | — | ✓ | | NVLink RX | Total RX of all NVLINK links. | `gpu.nvlink.throughput.raw.rx` | — | ✓ | | NVLink TX | Total TX of all NVLINK links. | `gpu.nvlink.throughput.raw.tx` | — | ✓ | | Power | Power usage for the GPU device.**Note**: On GA100 and previous architectures, this represents the instantaneous power at that moment.For newer architectures, it represents the average power draw (Watts) over one second. | `gpu.power.usage` | — | ✓ | | Temperature | Temperature of a GPU device. | `gpu.temperature` | — | ✓ | {% /collapsible-section %} ## Details side panel{% #details-side-panel %} Clicking any row in the Fleet table opens a side panel with more details for the selected cluster, host, or device. ### Connected entities{% #connected-entities %} Datadog's GPU Monitoring doesn't need to rely on NVIDIA'S DCGM Exporter. It uses the Datadog Agent to observe GPUs directly, providing insight into GPU usage and costs for pods and processes. Under the Connected Entities section in any detail view, you can see SM activity, GPU core utilization (only if System Probe is enabled), and the memory usage of pods, processes, and Slurm jobs. This helps you identify which workloads to cut or optimize to decrease total spend. **Note**: The Pods tab is only available if you're using Kubernetes. {% tab title="Cluster side panel" %} Within this side panel, you have a cluster-specific funnel that identifies: - Number of Total, Allocated (Kubernetes users only) , Active, and Effective devices within that particular cluster - Estimated total and idle cost of that cluster - Connected entities of that cluster: pods, processes, and Slurm jobs - Four key metrics (customizable) for that cluster: Core Utilization (only if System probe is enabled), Memory Utilization, PCIe Throughput, and Graphics Activity - Table of hosts associated with that cluster {% image source="https://docs.dd-static.net/images/gpu_monitoring/cluster_sidepanel.880f2caeb503225de4a517d04227b39f.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/cluster_sidepanel.880f2caeb503225de4a517d04227b39f.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Cluster specific side panel that breaks down idle devices, costs and connected entities" /%} {% /tab %} {% tab title="Host side panel" %} Within this side panel, you have a host-specific view that identifies: - Host-related metadata such as provider, instance type, CPU utilization, system memory used, system memory total, system I/O util, SM activity, and temperature - (only available for Kubernetes users) The specific GPU devices allocated to that host sorted by Graphics Engine Activity - Connected entities of that host: pods, processes, and Slurm jobs {% image source="https://docs.dd-static.net/images/gpu_monitoring/host_sidepanel.d4d52b29cb0427c321e8b749604c2df0.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/host_sidepanel.d4d52b29cb0427c321e8b749604c2df0.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Host specific side panel that displays the GPU devices tied to that host and connected entities" /%} {% /tab %} {% tab title="Device side panel" %} Within this side panel, you have a device-specific view that identifies: - Recommendations (if any) for how to use this device more effectively - Device-related details: device type, SM activity, and temperature - Four key metrics tied to GPUs: SM Activity, Memory Utilization, Power, and Graphics Engine Activity - Connected entities of that cluster: pods and processes {% image source="https://docs.dd-static.net/images/gpu_monitoring/device_sidepanel.1fdda7ff5d6507f7232eceaca8d20be2.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/device_sidepanel.1fdda7ff5d6507f7232eceaca8d20be2.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Device specific side panel that displays recommendations for how to use the device more effectively and other key telemetry." /%} {% /tab %} ## Installation recommendations{% #installation-recommendations %} Datadog actively surveys your infrastructure and detects installation gaps that may diminish the value you get out of GPU Monitoring. In this modal, you can find installation update recommendations to get the optimal value of GPU Monitoring. For example, making sure your hosts have the [latest version](https://github.com/DataDog/datadog-agent/releases) of the Datadog Agent installed, installing the latest version of the NVIDIA driver, and checking for misconfigured hosts. {% alert level="danger" %} Do not use Datadog Agent v7.82.0, which can cause unexpected kernel panics. {% /alert %} To view advanced GPU Monitoring features such as attribution of GPU resources by related processes or SLURM jobs, you must enable [Live Processes](https://docs.datadoghq.com/infrastructure/process.md) and the [Slurm](https://docs.datadoghq.com/integrations/slurm.md) integration, respectively. {% image source="https://docs.dd-static.net/images/gpu_monitoring/installation.d39249d01b9f30bce8947c4d29a9736f.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/installation.d39249d01b9f30bce8947c4d29a9736f.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Modal containing installation guidance for smoother GPU Monitoring user experience." /%} ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Optimize and troubleshoot AI infrastructure with Datadog GPU Monitoring](https://www.datadoghq.com/blog/datadog-gpu-monitoring/)