--- title: Plan GPU Capacity with GPU Monitoring description: >- Track GPU quota usage across teams and queues, find oversubscribed GPU types, and understand why workloads are pending or evicted. breadcrumbs: Docs > GPU Monitoring > Plan GPU Capacity with GPU Monitoring --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Plan GPU Capacity with GPU Monitoring {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} {% callout %} # Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com {% callout %} ##### Join the Preview! Capacity Planning in GPU Monitoring is in Early Access Preview. Complete the form to request access. [Request Access](https://www.datadoghq.com/product-preview/gpu-monitoring-capacity-planning/) {% /callout %} {% /callout %} ## Overview{% #overview %} When many teams share pools of expensive GPUs, it is hard to see which teams are underusing their devices. It is also hard to spot resource contention and to tell why jobs are stuck pending. The Capacity Planning page in GPU Monitoring gives platform teams and workload owners one shared view of GPU provisioning across queues. No manual filtering or querying is required. Use it to run workloads reliably and get more out of your GPU fleet. With the Capacity Planning page, you can answer questions such as: - **Which teams or projects are over- or underusing their quota?** Find unused quota and reallocate compute fairly across your organization. - **Which GPU types are oversubscribed?** Identify where demand exceeds quota so you can add capacity. - **What is the current state of a workload?** See whether each workload is admitted, pending, or evicted, and how many GPUs it uses. - **Why is a workload delayed or evicted?** Correlate pending and preempted workloads with quota and contention in their queue. {% image source="https://docs.dd-static.net/images/gpu_monitoring/capacity-planning-page.0f844214b152bc54c4e89b48f1c1dd35.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/capacity-planning-page.0f844214b152bc54c4e89b48f1c1dd35.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Capacity Planning page in GPU Monitoring showing GPU quota, idle allocated GPU, preemption, and fleet coverage graphs above a list of workloads." /%} ## Prerequisites{% #prerequisites %} To use the Capacity Planning page, you must meet the following criteria: - You are running Datadog Agent version 7.84.1 or later with [GPU Monitoring enabled](https://docs.datadoghq.com/gpu_monitoring/setup.md). To upgrade, see [Upgrade the Agent with Fleet Automation](https://docs.datadoghq.com/agent/guide/upgrade_agent_fleet_automation.md). - Your GPU workloads run on Kubernetes. - You have enabled the [Kueue integration](https://app.datadoghq.com/integrations?search=kueue) in Datadog. For details, see the [Kueue integration documentation](https://docs.datadoghq.com/integrations/kueue.md). ## Explore the Capacity Planning page{% #explore-the-capacity-planning-page %} To get started, navigate to the [Capacity Planning page](https://app.datadoghq.com/gpu-monitoring?mConfigure=false&mPage=capacity-planning) in GPU Monitoring. ### Filter by team and queue{% #filter-by-team-and-queue %} Use the filters at the top of the page to scope the view. You can filter by Team, Cluster Name, Cluster Queue, Resource Flavor, Provider, Datacenter, and Environment. Save a filtered view to return to it later. ### Quota and utilization{% #quota-and-utilization %} The graphs at the top of the page show how GPU quota is used across your fleet: | Graph | Description | | ------------------------------- | --------------------------------------------------------------------------------------------------- | | GPU Quota Unallocated - Count | Number of GPUs in quota that are not in use, grouped by ClusterQueue. | | GPU Quota Unallocated - Percent | Percentage of GPU quota that is not in use, grouped by ClusterQueue. | | Idle Allocated GPUs | Number of allocated GPUs that are not doing engine work, grouped by ClusterQueue. | | Preemption Rate | Workloads evicted by preemption per admitted workload, grouped by ClusterQueue. | | Fleet GPU Coverage | Allocated GPU quota split into active, idle, and unattributed devices across the fleet. | | Saturation - GPUs Over Quota | ClusterQueue and ResourceFlavor pairs ranked by the number of GPUs used beyond their nominal quota. | Queues with high unallocated quota or many idle allocated GPUs are candidates for reallocation. Queues that consistently run over quota, or have a high preemption rate, may need more capacity. ### Queues and workloads{% #queues-and-workloads %} The table at the bottom of the page has two tabs: - Queues: Lists each queue with its quota and usage. - Workloads: Lists each workload with its team, cluster, ClusterQueue, GPU usage by resource flavor, and latest status, such as Admitted. Use the Workloads tab to check the current state of a workload and to investigate workloads that are pending or were evicted. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md) - [Explore your GPU fleet with the Fleet page](https://docs.datadoghq.com/gpu_monitoring/fleet.md) - [Learn more about what GPU Monitoring offers](https://docs.datadoghq.com/gpu_monitoring.md)