--- title: Plan GPU Capacity with GPU Monitoring description: >- Track GPU quota usage across teams and queues, find oversubscribed GPU types, and understand why workloads are pending or evicted. breadcrumbs: Docs > GPU Monitoring > Plan GPU Capacity with GPU Monitoring --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Plan GPU Capacity with GPU Monitoring {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} {% callout %} # Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com {% callout %} ##### Join the Preview! Capacity Planning in GPU Monitoring is in Early Access Preview. Complete the form to request access. [Request Access](https://www.datadoghq.com/product-preview/gpu-monitoring-capacity-planning/) {% /callout %} {% /callout %} ## Overview{% #overview %} When many teams share pools of expensive GPUs, it is hard to see which teams are underusing their devices. It is also hard to spot resource contention and to tell why jobs are stuck pending. The Capacity Planning page in GPU Monitoring gives platform teams and workload owners one shared view of GPU provisioning across queues. No manual filtering or querying is required. Use it to run workloads reliably and get more out of your GPU fleet. With the Capacity Planning page, you can answer questions such as: - **Which teams or projects are over- or underusing their quota?** Find unused quota and reallocate compute fairly across your organization. - **Which GPU types are oversubscribed?** Identify where demand exceeds quota so you can add capacity. - **What is the current state of a workload?** See whether each workload is admitted, pending, or evicted, and how many GPUs it uses. - **Why is a workload delayed or evicted?** Correlate pending and preempted workloads with quota and contention in their queue. {% image source="https://docs.dd-static.net/images/gpu_monitoring/capacity-planning-page-3.2a7221b2601dc32327d0fad729c64a07.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/gpu_monitoring/capacity-planning-page-3.2a7221b2601dc32327d0fad729c64a07.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Capacity Planning page in GPU Monitoring showing GPU quota, idle allocated GPU, preemption, fleet coverage, and saturation graphs above a list of workloads." /%} ## Prerequisites{% #prerequisites %} To use the Capacity Planning page, you must meet the following criteria: - You have [GPU Monitoring enabled](https://docs.datadoghq.com/gpu_monitoring/setup.md). - You are running Datadog Agent and Cluster Agent version 7.84.1 or later. To upgrade, see [Upgrade the Agent with Fleet Automation](https://docs.datadoghq.com/agent/guide/upgrade_agent_fleet_automation.md). - Your GPU workloads run on Kubernetes and are managed by Kueue. ## Setup{% #setup %} The Capacity Planning page combines two sources of data: - **Kueue metrics**, such as GPU quota and usage for each ClusterQueue. The Kueue integration collects these metrics. - **GPU Monitoring metrics tagged with Kueue metadata**, such as the Kueue workload and resource flavor that each GPU is allocated to. The Cluster Agent collects this metadata. Complete the following steps to collect both. ### Collect Kueue metrics{% #collect-kueue-metrics %} 1. In the Kueue configuration, set `metrics.enableClusterQueueResources: true` and restart the Kueue controller manager. Kueue reports ClusterQueue quota and usage metrics, such as `kueue.cluster_queue.nominal_quota.gpu` and `kueue.cluster_queue.resource_usage.gpu`, only when this setting is enabled. The Capacity Planning graphs require these metrics. 1. Install the [Kueue integration](https://app.datadoghq.com/integrations?search=kueue) and configure the Kueue check as a cluster check. For instructions, see the [Kueue integration documentation](https://docs.datadoghq.com/integrations/kueue.md). 1. Confirm that the Kueue check runs successfully on its assigned Agent and reports quota and usage metrics before continuing. For steps to check discovery, dispatching, and Agent status, see [Troubleshooting Cluster and Endpoint Checks](https://docs.datadoghq.com/containers/troubleshooting/cluster-and-endpoint-checks.md). ### Collect Kueue metadata in the Cluster Agent{% #collect-kueue-metadata-in-the-cluster-agent %} The Cluster Agent reads Kueue ClusterQueue, LocalQueue, ResourceFlavor, and Workload resources. It tags GPU metrics with the Kueue workload, queue, and resource flavor that each GPU is allocated to. To collect this metadata, grant the Cluster Agent read access to Kueue resources, then enable Kueue metadata collection. #### Grant the Cluster Agent access to Kueue resources{% #grant-the-cluster-agent-access-to-kueue-resources %} 1. Find the name of the Cluster Agent service account. The Cluster Agent deployment name ends in `-cluster-agent`: ```shell kubectl get deployments -n -o custom-columns=NAME:.metadata.name,SERVICE_ACCOUNT:.spec.template.spec.serviceAccountName ``` 1. Create a file named `datadog-cluster-agent-kueue-rbac.yaml` with the following content. Replace `` with the service account name from the previous step, and `` with the namespace where the Datadog Agent is installed. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: datadog-cluster-agent-kueue rules: - apiGroups: - kueue.x-k8s.io resources: - clusterqueues - localqueues - resourceflavors - workloads verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: datadog-cluster-agent-kueue roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: datadog-cluster-agent-kueue subjects: - kind: ServiceAccount name: namespace: ``` 1. Apply the file: ```shell kubectl apply -f datadog-cluster-agent-kueue-rbac.yaml ``` #### Enable Kueue metadata collection{% #enable-kueue-metadata-collection %} These examples enable `cluster_agent.kueue.enabled` through the equivalent `DD_CLUSTER_AGENT_KUEUE_ENABLED` environment variable. Choose the tab that matches how you manage the Cluster Agent. Merge the example into your existing configuration, preserving its other settings. {% tab title="Datadog Operator" %} 1. In your `DatadogAgent` resource, set the `DD_CLUSTER_AGENT_KUEUE_ENABLED` environment variable on the Cluster Agent: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: override: clusterAgent: env: - name: DD_CLUSTER_AGENT_KUEUE_ENABLED value: "true" ``` 1. Apply your changes. The Datadog Operator redeploys the Cluster Agent with the new configuration. ```shell kubectl apply -n -f datadog-agent.yaml ``` {% /tab %} {% tab title="Helm" %} 1. In your `datadog-values.yaml` file, set the `DD_CLUSTER_AGENT_KUEUE_ENABLED` environment variable on the Cluster Agent: ```yaml clusterAgent: env: - name: DD_CLUSTER_AGENT_KUEUE_ENABLED value: "true" ``` 1. Upgrade your Helm release. Helm redeploys the Cluster Agent with the new configuration. ```shell helm upgrade -n -f datadog-values.yaml datadog/datadog ``` {% /tab %} {% tab title="Kubernetes manifests" %} 1. In your existing Cluster Agent Deployment manifest, add the following entry to the Cluster Agent container's `env` list under `spec.template.spec.containers`: ```yaml - name: DD_CLUSTER_AGENT_KUEUE_ENABLED value: "true" ``` 1. Apply the updated manifest. Replace `cluster-agent-deployment.yaml` with your manifest's filename. Kubernetes rolls out the updated Cluster Agent pods. ```shell kubectl apply -n -f cluster-agent-deployment.yaml ``` {% /tab %} ### Verify the setup{% #verify-the-setup %} After Kueue-managed workloads start running on GPUs: 1. In the [Metrics Explorer](https://docs.datadoghq.com/metrics/explorer.md), filter by `kube_cluster_name` and `kueue_cluster_queue` for the cluster and ClusterQueue you configured. Query `kueue.cluster_queue.nominal_quota.gpu` and `kueue.cluster_queue.resource_usage.gpu`. If either metric has no data, check the Kueue check configuration and the `metrics.enableClusterQueueResources` setting. 1. Using the same filters, query `gpu.gr_engine_active` grouped by `kueue_workload` and `kueue_resource_flavor`. Confirm that your running workload and its resource flavor appear. If either is missing, check that Kueue metadata collection is enabled. Also inspect the Cluster Agent logs for permission errors on `kueue.x-k8s.io` resources. 1. Open the [Capacity Planning page](https://app.datadoghq.com/gpu-monitoring?mConfigure=false&mPage=capacity-planning) and filter to the same cluster and ClusterQueue. Confirm that the queue and running workload appear. ## Explore the Capacity Planning page{% #explore-the-capacity-planning-page %} To get started, navigate to the [Capacity Planning page](https://app.datadoghq.com/gpu-monitoring?mConfigure=false&mPage=capacity-planning) in GPU Monitoring. ### Filter by team and queue{% #filter-by-team-and-queue %} Use the filters at the top of the page to scope the view. You can filter by Team, Cluster Name, Cluster Queue, Resource Flavor, Provider, Datacenter, and Environment. Save a filtered view to return to it later. ### Quota and utilization{% #quota-and-utilization %} The graphs at the top of the page show how GPU quota is used across your fleet: | Graph | Description | | ------------------------------- | --------------------------------------------------------------------------------------------------- | | GPU Quota Unallocated - Count | Number of GPUs in quota that are not in use, grouped by ClusterQueue. | | GPU Quota Unallocated - Percent | Percentage of GPU quota that is not in use, grouped by ClusterQueue. | | Idle Allocated GPUs | Number of allocated GPUs that are not doing engine work, grouped by ClusterQueue. | | Preemption Rate | Workloads evicted by preemption per admitted workload, grouped by ClusterQueue. | | Fleet GPU Coverage | Allocated GPU quota split into active, idle, and unattributed devices across the fleet. | | Saturation - GPUs Over Quota | ClusterQueue and ResourceFlavor pairs ranked by the number of GPUs used beyond their nominal quota. | Queues with high unallocated quota or many idle allocated GPUs are candidates for reallocation. Queues that consistently run over quota, or have a high preemption rate, may need more capacity. ### Queues and workloads{% #queues-and-workloads %} The table at the bottom of the page has two tabs: - Queues: Lists each queue with its quota and usage. - Workloads: Lists each workload with its team, cluster, ClusterQueue, GPU usage by resource flavor, and latest status, such as Admitted. Use the Workloads tab to check the current state of a workload and to investigate workloads that are pending or were evicted. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Set up GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring/setup.md) - [Analyze your GPU fleet with GPU Monitoring Explorer](https://docs.datadoghq.com/gpu_monitoring/explorer.md) - [Learn more about what GPU Monitoring offers](https://docs.datadoghq.com/gpu_monitoring.md)