---
title: GPU Monitoring
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: Docs > GPU Monitoring
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# GPU Monitoring

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Datadog's [GPU Monitoring](https://app.datadoghq.com/gpu-monitoring) helps you understand and optimize the performance, health, and cost of your GPU fleet and the AI workloads running on it, all in one place. With GPU Monitoring, you can:

- Get unified, fleet-wide GPU visibility across cloud, on-premises, and neocloud providers.
- Identify the key drivers of GPU inefficiencies and wasted spend by team, service, cluster, and more.
- Improve the throughput and performance of the AI training and inference workloads running on your GPUs.
- Pinpoint optimization opportunities, such as idle devices and zombie processes, to reclaim wasted spend.

GPU Monitoring supports fleets deployed across the major cloud providers (AWS, GCP, Azure, Oracle Cloud), on-premises, or through GPU-as-a-Service platforms such as CoreWeave and Lambda Labs. You don't need to manually set up individual vendor tools such as NVIDIA's DCGM.

You can access insights into your GPU fleet by deploying the Datadog Agent on your GPU-accelerated hosts. For setup instructions, see [Set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md).

## Key capabilities{% #key-capabilities %}

### Capacity plan with confidence{% #capacity-plan-with-confidence %}

{% callout %}
# Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com

{% callout %}
##### Join the Preview!

[Capacity Planning](https://docs.datadoghq.com/gpu_monitoring/capacity_planning.md) in GPU Monitoring is in Early Access Preview. Complete the form to request access.

[Request Access](https://www.datadoghq.com/product-preview/gpu-monitoring-capacity-planning/)
{% /callout %}

{% /callout %}

View your entire fleet's size and spend across hyperscalers, on-premises, and neocloud providers.

{% image
   source="https://docs.dd-static.net/images/gpu_monitoring/funnel-3.c8cda845fd9db2bcaf4c55b7b4269302.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/funnel-3.c8cda845fd9db2bcaf4c55b7b4269302.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Funnel visualization titled 'Your GPU fleet at a glance.' Displays total, active, and effective devices. Highlights underutilized GPU cores and idle devices." /%}

- Attribute GPU usage and quotas by team, service, or any tag to allocate capacity more fairly across teams and distinguish capacity shortages from inefficiencies.
- Identify resource contention and forecast demand for each GPU type to shorten procurement cycles and make spend more predictable.
- Diagnose why workloads are delayed or evicted.

{% image
   source="https://docs.dd-static.net/images/gpu_monitoring/device_allocation.2c00c30cea0b81ece4a5dcfae6529867.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/device_allocation.2c00c30cea0b81ece4a5dcfae6529867.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Charts to help visualize GPU allocation. A line graph titled 'Device Allocation Over Time', plotting counts of total/allocated/active devices, including a 4-week future forecast. A donut chart titled 'Cloud Provider Instance Breakdown', displaying prevalence of cloud provider instances across the fleet. A 'Device Type Breakdown' displaying allocated/total for various GPU devices." /%}

### Maximize AI workload throughput{% #maximize-ai-workload-throughput %}

{% callout %}
# Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com

{% callout %}
##### Join the Preview!

[Training](https://docs.datadoghq.com/gpu_monitoring/training.md) and [Inference](https://docs.datadoghq.com/gpu_monitoring/inference.md) in GPU Monitoring are in Early Access Preview. To request access, complete the form on each page.
{% /callout %}

{% /callout %}

Analyze trends in GPU resources and metrics (including GPU utilization, power, and memory) by host, node, or pod over time to understand how your devices affect model and application performance.

- Troubleshoot stalled workloads with shared context for both platform and ML teams instead of switching between siloed tools.
- Identify whether workload slowdowns are caused by unhealthy hardware, communication bottlenecks, memory bandwidth limitations, or scheduling issues.
- Detect bottlenecks in your training and inference workloads with lightweight, continuous tracing.
- Discover recommendations on the optimizations that produce the largest throughput improvements.

{% image
   source="https://docs.dd-static.net/images/gpu_monitoring/training-page.5defcbdb6aa492637958fa99a235a8eb.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/training-page.5defcbdb6aa492637958fa99a235a8eb.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="GPU Monitoring Training page showing run insights, a bar graph of runs over time, and a list of training runs." /%}

{% image
   source="https://docs.dd-static.net/images/gpu_monitoring/inference-page3.7bfb15dd305d3b7c21466cb3e22a0de7.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/inference-page3.7bfb15dd305d3b7c21466cb3e22a0de7.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="GPU Monitoring Inference page showing key metrics for a model deployment, such as time-to-first-token and output tokens." /%}

### Safeguard device reliability{% #safeguard-device-reliability %}

GPUs are an expensive and scarce resource with higher failure rates than standard servers. GPU Monitoring provides out-of-the-box monitors and proactive recommendations to help you detect and remediate hardware issues before they impact your mission-critical workloads.

- Catch thermal throttling early with out-of-the-box alerts before failures spread across the cluster.
- Proactively detect and remediate ECC and XID errors with prescriptive next steps that help teams respond quickly.
- Isolate unhealthy GPUs before failures cascade.

### Reduce wasted spend{% #reduce-wasted-spend %}

Identify total spend on GPU infrastructure and attribute those costs to specific workloads and instances. Directly correlate GPU usage to related pods or processes.

- Break down total and idle GPU spend by any tag and time frame to see where spend is concentrated.
- Identify the key drivers of wasted GPU spend and inefficiencies, such as teams with the highest GPU waste, to support internal chargebacks and data-driven quota management decisions.
- Give teams clear reporting on GPU usage and spend to make cost optimization part of their daily operations.

{% image
   source="https://docs.dd-static.net/images/gpu_monitoring/cost-and-utilization2.012a4fcc5a8041ffada6d08885aa9f2c.jpg?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/cost-and-utilization2.012a4fcc5a8041ffada6d08885aa9f2c.jpg?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Cost & Utilization page showing GPU hours and estimated cost by cluster, and top clusters by idle hours and idle cost." /%}

## Ready to start?{% #ready-to-start %}

See [Set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md) for instructions on how to set up Datadog's GPU Monitoring.

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md)
- [Optimize and troubleshoot AI infrastructure with Datadog GPU Monitoring](https://www.datadoghq.com/blog/datadog-gpu-monitoring/)
- [Monitor TAS and gang scheduling for AI training in Kubernetes](https://www.datadoghq.com/blog/monitor-tas-and-gang-scheduling-for-ai-training-in-kubernetes/)
- [How Kubernetes DRA can improve GPU utilization and cut costs](https://www.datadoghq.com/blog/how-kubernetes-dra-can-improve-gpu-utilization-and-cut-costs/)
- [GPU Monitoring Reference Architecture](https://www.datadoghq.com/architecture/gpu-monitoring/)