---
title: Kueue
description: >-
  Monitor the health and performance of Kueue, the Kubernetes-native job
  queueing system.
breadcrumbs: Docs > Integrations > Kueue
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Kueue
Supported OS Integration version3.0.0
{% callout %}
# Important note for users on the following Datadog sites: us2.ddog-gov.com

{% alert level="info" %}
To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com).
{% /alert %}

{% /callout %}
  Kueue Overview (light mode)Kueue Overview (dark mode)
## Overview{% #overview %}

This check monitors Kueue through the Datadog Agent.

Kueue is a Kubernetes workload queueing system that allows you to manage and schedule workloads on your Kubernetes cluster. It provides a way to prioritize and manage workloads, and to ensure that workloads are scheduled in a fair and efficient manner. This integration collects metrics from the Kueue controller manager and Kueue API server to help you monitor the health and performance of your Kueue cluster.

## Setup{% #setup %}

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/containers/kubernetes/integrations.md).

### Installation{% #installation %}

The Kueue check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package. No additional installation is required on your server.

### Configuration{% #configuration %}

Configure this integration as an [endpoints check](https://docs.datadoghq.com/containers/cluster_agent/endpointschecks.md), so that each Kueue controller manager pod is scraped individually. Endpoints checks are dispatched by the Cluster Agent to the node Agent running on the node of each Kueue pod, so cluster checks must be enabled. A node Agent must run on the nodes that host Kueue; if Kueue runs on control plane nodes, add the matching tolerations to the node Agent.

Kueue serves its metrics over HTTPS (port `8443` by default) with a self-signed certificate, and authorizes callers with their service account token. The node Agent service account needs the following permissions:

- `get` on the `/metrics` non-resource URL. The default Datadog Helm chart and Datadog Operator node Agent roles already include it.
- `get` and `list` on `workloads` in the `kueue.x-k8s.io` API group, for Workload lifecycle events. Set `collect_workload_events: false` to disable event collection.

1. To collect optional ClusterQueue resource metrics, such as `kueue.cluster_queue.resource_usage.gpu`, configure Kueue with `metrics.enableClusterQueueResources: true` and restart the Kueue controller manager.

1. Provide an endpoints check configuration to the Cluster Agent, targeting the Kueue metrics Service:

   ```yaml
   clusterAgent:
     confd:
       kueue.yaml: |-
         advanced_ad_identifiers:
           - kube_endpoints:
               name: kueue-controller-manager-metrics-service
               namespace: kueue-system
         cluster_check: true
         init_config:
         instances:
           - openmetrics_endpoint: https://%%host%%:%%port%%/metrics
             tls_verify: false
             auth_token:
               reader:
                 type: file
                 path: /var/run/secrets/kubernetes.io/serviceaccount/token
               writer:
                 type: header
                 name: Authorization
                 value: "Bearer <TOKEN>"
                 placeholder: "<TOKEN>"
   ```

`%%host%%` resolves to the IP of each Kueue pod, and `%%port%%` to the last port of the Service endpoints. If your Kueue metrics Service exposes more than one port, set the port explicitly.

1. Alternatively, annotate the Kueue metrics Service with Autodiscovery endpoints check annotations:

   ```yaml
   ad.datadoghq.com/endpoints.checks: |
     {
       "kueue": {
         "instances": [
           {
             "openmetrics_endpoint": "https://%%host%%:%%port%%/metrics",
             "tls_verify": false,
             "auth_token": {
               "reader": {"type": "file", "path": "/var/run/secrets/kubernetes.io/serviceaccount/token"},
               "writer": {"type": "header", "name": "Authorization", "value": "Bearer <TOKEN>", "placeholder": "<TOKEN>"}
             }
           }
         ]
       }
     }
   ```

A cluster check that scrapes the Service URL, such as `https://kueue-controller-manager-metrics-service.kueue-system.svc:8443/metrics`, only works with a single Kueue replica. See High availability.

See the [sample kueue.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/kueue/datadog_checks/kueue/data/conf.yaml.example) for all available configuration options.

### High availability{% #high-availability %}

When Kueue runs with more than one replica, every replica exports the same state gauges, such as `kueue.pending_workloads` or `kueue.cluster_queue.resource_usage.*`, labeled with `replica_role:leader` or `replica_role:follower`. Most counters and histograms, and the `kueue.local_queue.resource_usage.*`, `kueue.local_queue.resource_reservation.*` and `kueue.local_queue.status` metrics, are only exported by the leader. Scraping through the Service reaches a random replica, so leader-only metrics would be missing from some collection runs. Use the endpoints check configuration above to scrape every replica.

With one check instance per replica:

- Samples labeled `replica_role:follower` are dropped, so state gauges are not reported twice. Samples without a `replica_role` label, such as Go runtime, process, and controller-runtime metrics, are reported for every replica. Set `collect_follower_metrics: true` to keep follower samples.
- Every instance polls the Workload resources, but only the instance that scraped the leader, or a replica reporting `replica_role:standalone` or no `replica_role` label, submits Workload events. The role is read from the collected samples, so do not exclude follower samples with `exclude_metrics_by_labels` or exclude every `replica_role`-labeled metric: the follower instances would then see no role and submit duplicate events.
- After a leader failover, the new leader reports its gauges immediately, and its counters start from zero. It keeps exporting the series it recorded as a follower, still labeled `replica_role:follower`; those are dropped, and the replica is treated as the leader. Transitions that its instance observes before the replica reports `replica_role:leader` are not submitted as events. Few transitions happen in this window, since Workload conditions are written by the leader.

### Cluster agent configuration and GPU monitoring integration{% #cluster-agent-configuration-and-gpu-monitoring-integration %}

Enabling [GPU monitoring](https://docs.datadoghq.com/gpu_monitoring.md) will enrich the Kueue integration with GPU-related data, and will also show GPU-related tags in the Kueue metrics. In order to enable this part of the integration, two settings need to be configured in the Datadog Agent configuration:

```yaml
gpu:
  enabled: true

cluster_agent:
  kueue:
    enabled: true
```

### Log collection{% #log-collection %}

The Kueue controller manager writes logs to its container output, which Kubernetes captures as container logs. Collecting logs is disabled by default in the Datadog Agent. To enable it, see [Kubernetes Log Collection](https://docs.datadoghq.com/containers/kubernetes/log.md). Logs are collected by the node Agent running on the node that hosts the Kueue controller manager.

After log collection has been enabled, set the Kueue log configuration as an Autodiscovery annotation on the controller manager's pod template. This allows it to persist despite pod restarts. Add it under `spec.template.metadata.annotations` of the `kueue-controller-manager` deployment, or set `controllerManager.manager.podAnnotations` if you install Kueue with the Helm chart:

```yaml
ad.datadoghq.com/manager.logs: |
  [
    {
      "source": "kueue",
      "service": "<SERVICE>"
    }
  ]
```

This annotation targets the container named `manager`, which is the container name used by both the Kueue release manifests and the Helm chart. Replace `manager` with the name (`.spec.containers[i].name`) of your Kueue container if you use a different name.

### Validation{% #validation %}

[Run the Cluster Agent's `clusterchecks` subcommand](https://docs.datadoghq.com/containers/troubleshooting/cluster-and-endpoint-checks.md#dispatching-logic-in-the-cluster-agent) and look for one `kueue` check per Kueue pod under the `Pod-backed Endpoints-Checks` section. Then run the `status` subcommand on the node Agents listed for those checks.

## Data Collected{% #data-collected %}

### Metrics{% #metrics %}

|  |
|  |
| **kueue.admission.attempts.count**(count)                                      | The total number of workload admission attempts. Each admission attempt might try to admit more than one workload. The 'result' label can be 'success' when at least one workload is admitted or 'inadmissible' when no workload is admitted.                                                                                 |
| **kueue.admission.wait_time.seconds.bucket**(count)                           | The time between workload creation or requeueing and admission, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                                   |
| **kueue.admission.wait_time.seconds.count**(count)                            | The time between workload creation or requeueing and admission, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                                   |
| **kueue.admission.wait_time.seconds.sum**(count)                              | The time between workload creation or requeueing and admission, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                                   |
| **kueue.admission_attempt.duration.seconds.bucket**(count)                    | The duration of an admission attempt. The 'result' label can be 'success' when at least one workload is admitted or 'inadmissible' when no workload is admitted.*Shown as second*                                                                                                                                             |
| **kueue.admission_attempt.duration.seconds.count**(count)                     | The duration of an admission attempt. The 'result' label can be 'success' when at least one workload is admitted or 'inadmissible' when no workload is admitted.*Shown as second*                                                                                                                                             |
| **kueue.admission_attempt.duration.seconds.sum**(count)                       | The duration of an admission attempt. The 'result' label can be 'success' when at least one workload is admitted or 'inadmissible' when no workload is admitted.*Shown as second*                                                                                                                                             |
| **kueue.admission_checks.wait_time.seconds.bucket**(count)                   | The time between quota reservation and admission for a workload, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                                  |
| **kueue.admission_checks.wait_time.seconds.count**(count)                    | The time between quota reservation and admission for a workload, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                                  |
| **kueue.admission_checks.wait_time.seconds.sum**(count)                      | The time between quota reservation and admission for a workload, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                                  |
| **kueue.admission_cycle.preemption_skips**(gauge)                            | The number of Workloads in the ClusterQueue that received preemption candidates but were skipped because other ClusterQueues needed the same resources in the same cycle.                                                                                                                                                     |
| **kueue.admitted.active_workloads**(gauge)                                    | The number of admitted workloads that are active, per 'kueue_cluster_queue'.                                                                                                                                                                                                                                                  |
| **kueue.admitted.workloads.count**(count)                                      | The total number of admitted workloads, per 'kueue_cluster_queue'.                                                                                                                                                                                                                                                            |
| **kueue.admitted_until_ready.wait_time.seconds.bucket**(count)              | The time between workload admission and the workload becoming ready, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                              |
| **kueue.admitted_until_ready.wait_time.seconds.count**(count)               | The time between workload admission and the workload becoming ready, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                              |
| **kueue.admitted_until_ready.wait_time.seconds.sum**(count)                 | The time between workload admission and the workload becoming ready, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                              |
| **kueue.build_info**(gauge)                                                   | Kueue build information. The value is 1 and is labeled by Git version, Git commit, build date, Go version, compiler, and platform.                                                                                                                                                                                            |
| **kueue.certwatcher.read_certificate.count**(count)                           | Kueue certwatcher read certificate metric.                                                                                                                                                                                                                                                                                    |
| **kueue.certwatcher.read_certificate_errors.count**(count)                   | Kueue certwatcher read certificate errors metric.                                                                                                                                                                                                                                                                             |
| **kueue.cluster_queue.borrowing_limit.cpu**(gauge)                           | Reports the ClusterQueue's resource borrowing limit across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.cluster_queue.borrowing_limit.gpu**(gauge)                           | Reports the ClusterQueue's resource borrowing limit across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.cluster_queue.borrowing_limit.memory**(gauge)                        | Reports the ClusterQueue's resource borrowing limit across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.cluster_queue.borrowing_limit.other**(gauge)                         | Reports the ClusterQueue's resource borrowing limit across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.cluster_queue.info**(gauge)                                           | Reports ClusterQueue hierarchy information. The metric has value 1 and can be joined using labels.                                                                                                                                                                                                                            |
| **kueue.cluster_queue.lending_limit.cpu**(gauge)                             | Reports the ClusterQueue's resource lending limit across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.lending_limit.gpu**(gauge)                             | Reports the ClusterQueue's resource lending limit across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.lending_limit.memory**(gauge)                          | Reports the ClusterQueue's resource lending limit across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.lending_limit.other**(gauge)                           | Reports the ClusterQueue's resource lending limit across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.nominal_quota.cpu**(gauge)                             | Reports the ClusterQueue's nominal resource quota across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.nominal_quota.gpu**(gauge)                             | Reports the ClusterQueue's nominal resource quota across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.nominal_quota.memory**(gauge)                          | Reports the ClusterQueue's nominal resource quota across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.nominal_quota.other**(gauge)                           | Reports the ClusterQueue's nominal resource quota across all flavors.                                                                                                                                                                                                                                                         |
| **kueue.cluster_queue.resource_pending.cpu**(gauge)                          | Reports the ClusterQueue's total pending resource requests. Unlike resource_reservation, pending workloads have not yet been assigned to flavors.                                                                                                                                                                             |
| **kueue.cluster_queue.resource_pending.gpu**(gauge)                          | Reports the ClusterQueue's total pending resource requests. Unlike resource_reservation, pending workloads have not yet been assigned to flavors.                                                                                                                                                                             |
| **kueue.cluster_queue.resource_pending.memory**(gauge)                       | Reports the ClusterQueue's total pending resource requests. Unlike resource_reservation, pending workloads have not yet been assigned to flavors.                                                                                                                                                                             |
| **kueue.cluster_queue.resource_pending.other**(gauge)                        | Reports the ClusterQueue's total pending resource requests. Unlike resource_reservation, pending workloads have not yet been assigned to flavors.                                                                                                                                                                             |
| **kueue.cluster_queue.resource_reservation.cpu**(gauge)                      | Reports the ClusterQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                     |
| **kueue.cluster_queue.resource_reservation.gpu**(gauge)                      | Reports the ClusterQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                     |
| **kueue.cluster_queue.resource_reservation.memory**(gauge)                   | Reports the ClusterQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                     |
| **kueue.cluster_queue.resource_reservation.other**(gauge)                    | Reports the ClusterQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                     |
| **kueue.cluster_queue.resource_usage.cpu**(gauge)                            | Reports the ClusterQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                           |
| **kueue.cluster_queue.resource_usage.gpu**(gauge)                            | Reports the ClusterQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                           |
| **kueue.cluster_queue.resource_usage.memory**(gauge)                         | Reports the ClusterQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                           |
| **kueue.cluster_queue.resource_usage.other**(gauge)                          | Reports the ClusterQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                           |
| **kueue.cluster_queue.status**(gauge)                                         | Reports the 'status' of each 'kueue_cluster_queue'. Possible values are 'pending', 'active', and 'terminating'. For a ClusterQueue, the metric reports a value of 1 for exactly one status.                                                                                                                                   |
| **kueue.cluster_queue.weighted_share**(gauge)                                | Reports the maximum ratio of usage above nominal quota to lendable resources in the cohort across all resources provided by the ClusterQueue, divided by weight. A value of 0 means that the ClusterQueue's usage is below the nominal quota. If the ClusterQueue has a weight of 0 and is borrowing, the metric returns NaN. |
| **kueue.cohort.info**(gauge)                                                   | Reports Cohort hierarchy information. The metric has value 1 and can be joined using labels.                                                                                                                                                                                                                                  |
| **kueue.cohort.weighted_share**(gauge)                                        | Reports the maximum ratio of usage above nominal quota to lendable resources in the Cohort across all resources provided by the Cohort, divided by weight. A value of 0 means that the Cohort's usage is below the nominal quota. If the Cohort has a weight of 0 and is borrowing, the metric returns NaN.                   |
| **kueue.cohort_subtree.admitted.active_workloads**(gauge)                    | The number of admitted Workloads that are active, per cohort subtree.                                                                                                                                                                                                                                                         |
| **kueue.cohort_subtree.admitted.workloads.count**(count)                      | The total number of admitted workloads, per cohort subtree.                                                                                                                                                                                                                                                                   |
| **kueue.cohort_subtree.quota.cpu**(gauge)                                     | Reports the cohort's nominal quota aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                         |
| **kueue.cohort_subtree.quota.gpu**(gauge)                                     | Reports the cohort's nominal quota aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                         |
| **kueue.cohort_subtree.quota.memory**(gauge)                                  | Reports the cohort's nominal quota aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                         |
| **kueue.cohort_subtree.quota.other**(gauge)                                   | Reports the cohort's nominal quota aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                         |
| **kueue.cohort_subtree.resource_reservations.cpu**(gauge)                    | Reports the cohort's resource reservations aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                 |
| **kueue.cohort_subtree.resource_reservations.gpu**(gauge)                    | Reports the cohort's resource reservations aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                 |
| **kueue.cohort_subtree.resource_reservations.memory**(gauge)                 | Reports the cohort's resource reservations aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                 |
| **kueue.cohort_subtree.resource_reservations.other**(gauge)                  | Reports the cohort's resource reservations aggregated within the cohort subtree. Values are reported per resource and flavor.                                                                                                                                                                                                 |
| **kueue.controller.runtime.active_workers**(gauge)                            | Kueue controller runtime active workers metric.                                                                                                                                                                                                                                                                               |
| **kueue.controller.runtime.conversion_webhook_panics.count**(count)          | Kueue controller runtime conversion webhook panics metric.                                                                                                                                                                                                                                                                    |
| **kueue.controller.runtime.max_concurrent_reconciles**(gauge)                | Kueue controller runtime max concurrent reconciles metric.                                                                                                                                                                                                                                                                    |
| **kueue.controller.runtime.reconcile.count**(count)                            | Kueue controller runtime reconcile metric.                                                                                                                                                                                                                                                                                    |
| **kueue.controller.runtime.reconcile_errors.count**(count)                    | Kueue controller runtime reconcile errors metric.                                                                                                                                                                                                                                                                             |
| **kueue.controller.runtime.reconcile_panics.count**(count)                    | Kueue controller runtime reconcile panics metric.                                                                                                                                                                                                                                                                             |
| **kueue.controller.runtime.reconcile_time.seconds.bucket**(count)             | Kueue controller runtime reconcile time seconds metric.*Shown as second*                                                                                                                                                                                                                                                      |
| **kueue.controller.runtime.reconcile_time.seconds.count**(count)              | Kueue controller runtime reconcile time seconds metric.*Shown as second*                                                                                                                                                                                                                                                      |
| **kueue.controller.runtime.reconcile_time.seconds.sum**(count)                | Kueue controller runtime reconcile time seconds metric.*Shown as second*                                                                                                                                                                                                                                                      |
| **kueue.controller.runtime.reconcile_timeouts.count**(count)                  | Kueue controller runtime reconcile timeouts metric.                                                                                                                                                                                                                                                                           |
| **kueue.controller.runtime.terminal_reconcile_errors.count**(count)          | Kueue controller runtime terminal reconcile errors metric.                                                                                                                                                                                                                                                                    |
| **kueue.controller.runtime.webhook_latency.seconds.bucket**(count)            | Kueue controller runtime webhook latency seconds metric.*Shown as second*                                                                                                                                                                                                                                                     |
| **kueue.controller.runtime.webhook_latency.seconds.count**(count)             | Kueue controller runtime webhook latency seconds metric.*Shown as second*                                                                                                                                                                                                                                                     |
| **kueue.controller.runtime.webhook_latency.seconds.sum**(count)               | Kueue controller runtime webhook latency seconds metric.*Shown as second*                                                                                                                                                                                                                                                     |
| **kueue.controller.runtime.webhook_panics.count**(count)                      | Kueue controller runtime webhook panics metric.                                                                                                                                                                                                                                                                               |
| **kueue.controller.runtime.webhook_requests.count**(count)                    | Kueue controller runtime webhook requests metric.                                                                                                                                                                                                                                                                             |
| **kueue.controller.runtime.webhook_requests_in_flight**(gauge)              | Kueue controller runtime webhook requests in flight metric.                                                                                                                                                                                                                                                                   |
| **kueue.evicted_workloads.count**(count)                                      | The number of evicted workloads per cluster queue, tagged by reason, underlying cause, priority class, and replica role.                                                                                                                                                                                                      |
| **kueue.evicted_workloads_once.count**(count)                                | The number of unique workload evictions per cluster queue, tagged by eviction reason, underlying cause, priority class, and replica role.                                                                                                                                                                                     |
| **kueue.finished_workloads**(gauge)                                           | The number of finished workloads, per 'kueue_cluster_queue'.                                                                                                                                                                                                                                                                  |
| **kueue.finished_workloads.count**(count)                                     | The total number of finished workloads, per 'kueue_cluster_queue'.                                                                                                                                                                                                                                                            |
| **kueue.go.gc.duration.seconds.count**(count)                                  | The summary count of garbage collection cycles.*Shown as second*                                                                                                                                                                                                                                                              |
| **kueue.go.gc.duration.seconds.quantile**(gauge)                               | A summary of the pause duration of garbage collection cycles.*Shown as second*                                                                                                                                                                                                                                                |
| **kueue.go.gc.duration.seconds.sum**(count)                                    | The sum of the pause duration of garbage collection cycles.*Shown as second*                                                                                                                                                                                                                                                  |
| **kueue.go.gc.gogc.percent**(gauge)                                            | Kueue go gc gogc percent metric.*Shown as percent*                                                                                                                                                                                                                                                                            |
| **kueue.go.gc.gomemlimit.bytes**(gauge)                                        | Kueue go gc gomemlimit bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                           |
| **kueue.go.goroutines**(gauge)                                                 | Number of goroutines that currently exist.                                                                                                                                                                                                                                                                                    |
| **kueue.go.info**(gauge)                                                       | Kueue go info metric.                                                                                                                                                                                                                                                                                                         |
| **kueue.go.memstats.alloc_bytes**(gauge)                                      | Kueue go memstats alloc bytes metric.                                                                                                                                                                                                                                                                                         |
| **kueue.go.memstats.alloc_bytes.count**(count)                                | Kueue go memstats alloc bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                          |
| **kueue.go.memstats.buck_hash.sys_bytes**(gauge)                             | Kueue go memstats buck hash sys bytes metric.                                                                                                                                                                                                                                                                                 |
| **kueue.go.memstats.frees.count**(count)                                       | Kueue go memstats frees metric.                                                                                                                                                                                                                                                                                               |
| **kueue.go.memstats.gc.sys_bytes**(gauge)                                     | Kueue go memstats gc sys bytes metric.                                                                                                                                                                                                                                                                                        |
| **kueue.go.memstats.heap_alloc.bytes**(gauge)                                 | Kueue go memstats heap alloc bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                     |
| **kueue.go.memstats.heap_idle.bytes**(gauge)                                  | Kueue go memstats heap idle bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                      |
| **kueue.go.memstats.heap_inuse.bytes**(gauge)                                 | Kueue go memstats heap inuse bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                     |
| **kueue.go.memstats.heap_objects**(gauge)                                     | Kueue go memstats heap objects metric.                                                                                                                                                                                                                                                                                        |
| **kueue.go.memstats.heap_released.bytes**(gauge)                              | Kueue go memstats heap released bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                  |
| **kueue.go.memstats.heap_sys.bytes**(gauge)                                   | Kueue go memstats heap sys bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                       |
| **kueue.go.memstats.mallocs.count**(count)                                     | Kueue go memstats mallocs metric.                                                                                                                                                                                                                                                                                             |
| **kueue.go.memstats.mcache_inuse.bytes**(gauge)                               | Kueue go memstats mcache inuse bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                   |
| **kueue.go.memstats.mcache_sys.bytes**(gauge)                                 | Kueue go memstats mcache sys bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                     |
| **kueue.go.memstats.mspan_inuse.bytes**(gauge)                                | Kueue go memstats mspan inuse bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                    |
| **kueue.go.memstats.mspan_sys.bytes**(gauge)                                  | Kueue go memstats mspan sys bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                      |
| **kueue.go.memstats.next_gc.bytes**(gauge)                                    | Kueue go memstats next gc bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                        |
| **kueue.go.memstats.other_sys.bytes**(gauge)                                  | Kueue go memstats other sys bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                      |
| **kueue.go.memstats.stack_inuse.bytes**(gauge)                                | Kueue go memstats stack inuse bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                    |
| **kueue.go.memstats.stack_sys.bytes**(gauge)                                  | Kueue go memstats stack sys bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                      |
| **kueue.go.memstats.sys.bytes**(gauge)                                         | Kueue go memstats sys bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                            |
| **kueue.go.memstats.time_since_last_gc.seconds**(gauge)                     | Kueue go memstats time since last gc seconds metric.*Shown as second*                                                                                                                                                                                                                                                         |
| **kueue.go.threads**(gauge)                                                    | Kueue go threads metric.                                                                                                                                                                                                                                                                                                      |
| **kueue.leader_election.master_status**(gauge)                               | Kueue leader election master status metric.                                                                                                                                                                                                                                                                                   |
| **kueue.local_queue.admission.wait_time.seconds.bucket**(count)              | The time between workload creation or requeueing and admission, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                     |
| **kueue.local_queue.admission.wait_time.seconds.count**(count)               | The time between workload creation or requeueing and admission, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                     |
| **kueue.local_queue.admission.wait_time.seconds.sum**(count)                 | The time between workload creation or requeueing and admission, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                     |
| **kueue.local_queue.admission_checks.wait_time.seconds.bucket**(count)      | The time between quota reservation and admission for a workload, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                    |
| **kueue.local_queue.admission_checks.wait_time.seconds.count**(count)       | The time between quota reservation and admission for a workload, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                    |
| **kueue.local_queue.admission_checks.wait_time.seconds.sum**(count)         | The time between quota reservation and admission for a workload, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                    |
| **kueue.local_queue.admitted.active_workloads**(gauge)                       | The number of admitted Workloads that are active, per 'kueue_local_queue'.                                                                                                                                                                                                                                                    |
| **kueue.local_queue.admitted.workloads.count**(count)                         | The total number of admitted workloads, per 'kueue_local_queue'.                                                                                                                                                                                                                                                              |
| **kueue.local_queue.admitted_until_ready.wait_time.seconds.bucket**(count) | The time between workload admission and the workload becoming ready, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                |
| **kueue.local_queue.admitted_until_ready.wait_time.seconds.count**(count)  | The time between workload admission and the workload becoming ready, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                |
| **kueue.local_queue.admitted_until_ready.wait_time.seconds.sum**(count)    | The time between workload admission and the workload becoming ready, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                                |
| **kueue.local_queue.evicted_workloads.count**(count)                         | The number of evicted workloads per local queue, tagged by namespace, reason, underlying cause, priority class, and replica role.                                                                                                                                                                                             |
| **kueue.local_queue.finished_workloads**(gauge)                              | The number of finished workloads, per 'kueue_local_queue'.                                                                                                                                                                                                                                                                    |
| **kueue.local_queue.finished_workloads.count**(count)                        | The total number of finished workloads, per 'kueue_local_queue'.                                                                                                                                                                                                                                                              |
| **kueue.local_queue.pending_workloads**(gauge)                               | The number of pending workloads, per 'kueue_local_queue' and 'status'. The 'status' value can be 'active' for workloads in the admission queue or 'inadmissible' for workloads with a failed admission attempt that are not retried until cluster conditions that could make them admissible change.                          |
| **kueue.local_queue.quota_reserved.wait_time.seconds.bucket**(count)        | The time from when a workload is created or requeued until it receives a quota reservation, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                         |
| **kueue.local_queue.quota_reserved.wait_time.seconds.count**(count)         | The time from when a workload is created or requeued until it receives a quota reservation, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                         |
| **kueue.local_queue.quota_reserved.wait_time.seconds.sum**(count)           | The time from when a workload is created or requeued until it receives a quota reservation, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                         |
| **kueue.local_queue.quota_reserved.workloads.count**(count)                  | The total number of quota-reserved workloads, per 'kueue_local_queue'.                                                                                                                                                                                                                                                        |
| **kueue.local_queue.ready_wait_time.seconds.bucket**(count)                 | The time between workload creation or requeueing and the workload becoming ready, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                   |
| **kueue.local_queue.ready_wait_time.seconds.count**(count)                  | The time between workload creation or requeueing and the workload becoming ready, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                   |
| **kueue.local_queue.ready_wait_time.seconds.sum**(count)                    | The time between workload creation or requeueing and the workload becoming ready, per 'kueue_local_queue'.*Shown as second*                                                                                                                                                                                                   |
| **kueue.local_queue.reserving.active_workloads**(gauge)                      | The number of Workloads that are reserving quota, per 'kueue_local_queue'.                                                                                                                                                                                                                                                    |
| **kueue.local_queue.resource_reservation.cpu**(gauge)                        | Reports the LocalQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.local_queue.resource_reservation.gpu**(gauge)                        | Reports the LocalQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.local_queue.resource_reservation.memory**(gauge)                     | Reports the LocalQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.local_queue.resource_reservation.other**(gauge)                      | Reports the LocalQueue's total resource reservation across all flavors.                                                                                                                                                                                                                                                       |
| **kueue.local_queue.resource_usage.cpu**(gauge)                              | Reports the LocalQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                             |
| **kueue.local_queue.resource_usage.gpu**(gauge)                              | Reports the LocalQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                             |
| **kueue.local_queue.resource_usage.memory**(gauge)                           | Reports the LocalQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                             |
| **kueue.local_queue.resource_usage.other**(gauge)                            | Reports the LocalQueue's total resource usage across all flavors.                                                                                                                                                                                                                                                             |
| **kueue.local_queue.status**(gauge)                                           | Reports the 'active' status of each 'kueue_local_queue'. Possible values are 'True', 'False', and 'Unknown'. For a LocalQueue, only one status has a value of 1.                                                                                                                                                              |
| **kueue.pending_workloads**(gauge)                                            | The number of pending workloads, per 'kueue_cluster_queue' and 'status'. The 'status' value can be 'active' for workloads in the admission queue or 'inadmissible' for workloads with a failed admission attempt that are not retried until cluster conditions that could make them admissible change.                        |
| **kueue.pods_ready_to_evicted_time.seconds.bucket**(count)                 | Bucket counts for the time between the workload's pods becoming ready and workload eviction, tagged by cluster queue, reason, underlying cause, and replica role.*Shown as second*                                                                                                                                            |
| **kueue.pods_ready_to_evicted_time.seconds.count**(count)                  | Count of observations for the time between the workload's pods becoming ready and workload eviction, tagged by cluster queue, reason, underlying cause, and replica role.*Shown as second*                                                                                                                                    |
| **kueue.pods_ready_to_evicted_time.seconds.sum**(count)                    | Total time between the workload's pods becoming ready and workload eviction, tagged by cluster queue, reason, underlying cause, and replica role.*Shown as second*                                                                                                                                                            |
| **kueue.preempted_workloads.count**(count)                                    | The number of preempted workloads per preempting cluster queue, tagged by preemption reason and replica role.                                                                                                                                                                                                                 |
| **kueue.process.cpu.seconds.count**(count)                                     | Kueue process cpu seconds metric.*Shown as second*                                                                                                                                                                                                                                                                            |
| **kueue.process.max_fds**(gauge)                                              | Kueue process max fds metric.                                                                                                                                                                                                                                                                                                 |
| **kueue.process.network_receive.bytes.count**(count)                          | Kueue process network receive bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                    |
| **kueue.process.network_transmit.bytes.count**(count)                         | Kueue process network transmit bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                   |
| **kueue.process.open_fds**(gauge)                                             | Kueue process open fds metric.                                                                                                                                                                                                                                                                                                |
| **kueue.process.resident_memory.bytes**(gauge)                                | Kueue process resident memory bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                    |
| **kueue.process.uptime.seconds**(gauge)                                        | Kueue process uptime seconds metric.*Shown as second*                                                                                                                                                                                                                                                                         |
| **kueue.process.virtual_memory.bytes**(gauge)                                 | Kueue process virtual memory bytes metric.*Shown as byte*                                                                                                                                                                                                                                                                     |
| **kueue.process.virtual_memory.max_bytes**(gauge)                            | Kueue process virtual memory max bytes metric.                                                                                                                                                                                                                                                                                |
| **kueue.quota_reserved.wait_time.seconds.bucket**(count)                     | The time from when a workload is created or requeued until it receives a quota reservation, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                       |
| **kueue.quota_reserved.wait_time.seconds.count**(count)                      | The time from when a workload is created or requeued until it receives a quota reservation, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                       |
| **kueue.quota_reserved.wait_time.seconds.sum**(count)                        | The time from when a workload is created or requeued until it receives a quota reservation, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                       |
| **kueue.quota_reserved.workloads.count**(count)                               | The total number of quota-reserved workloads, per 'kueue_cluster_queue'.                                                                                                                                                                                                                                                      |
| **kueue.ready_wait_time.seconds.bucket**(count)                              | The time between workload creation or requeueing and the workload becoming ready, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                 |
| **kueue.ready_wait_time.seconds.count**(count)                               | The time between workload creation or requeueing and the workload becoming ready, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                 |
| **kueue.ready_wait_time.seconds.sum**(count)                                 | The time between workload creation or requeueing and the workload becoming ready, per 'kueue_cluster_queue'.*Shown as second*                                                                                                                                                                                                 |
| **kueue.replaced_workload_slices.count**(count)                              | The number of replaced workload slices, per 'kueue_cluster_queue'.                                                                                                                                                                                                                                                            |
| **kueue.reserving.active_workloads**(gauge)                                   | The number of Workloads that are reserving quota, per 'kueue_cluster_queue'.                                                                                                                                                                                                                                                  |
| **kueue.workload.creation_latency.seconds.bucket**(count)                     | The time between job creation and workload creation, per 'job_kind'. Entries are only recorded for objects with generation 1.*Shown as second*                                                                                                                                                                                |
| **kueue.workload.creation_latency.seconds.count**(count)                      | The time between job creation and workload creation, per 'job_kind'. Entries are only recorded for objects with generation 1.*Shown as second*                                                                                                                                                                                |
| **kueue.workload.creation_latency.seconds.sum**(count)                        | The time between job creation and workload creation, per 'job_kind'. Entries are only recorded for objects with generation 1.*Shown as second*                                                                                                                                                                                |
| **kueue.workload.eviction_latency.seconds.bucket**(count)                     | Bucket counts for the time from workload eviction until the workload returns to pending and releases quota, tagged by cluster queue, reason, and replica role.*Shown as second*                                                                                                                                               |
| **kueue.workload.eviction_latency.seconds.count**(count)                      | Count of observations for the time from workload eviction until the workload returns to pending and releases quota, tagged by cluster queue, reason, and replica role.*Shown as second*                                                                                                                                       |
| **kueue.workload.eviction_latency.seconds.sum**(count)                        | Total time from workload eviction until the workload returns to pending and releases quota, tagged by cluster queue, reason, and replica role.*Shown as second*                                                                                                                                                               |
| **kueue.workqueue.adds.count**(count)                                          | Kueue workqueue adds metric.                                                                                                                                                                                                                                                                                                  |
| **kueue.workqueue.depth**(gauge)                                               | Kueue workqueue depth metric.                                                                                                                                                                                                                                                                                                 |
| **kueue.workqueue.longest_running_processor.seconds**(gauge)                 | Kueue workqueue longest running processor seconds metric.*Shown as second*                                                                                                                                                                                                                                                    |
| **kueue.workqueue.queue_duration.seconds.bucket**(count)                      | Kueue workqueue queue duration seconds metric.*Shown as second*                                                                                                                                                                                                                                                               |
| **kueue.workqueue.queue_duration.seconds.count**(count)                       | Kueue workqueue queue duration seconds metric.*Shown as second*                                                                                                                                                                                                                                                               |
| **kueue.workqueue.queue_duration.seconds.sum**(count)                         | Kueue workqueue queue duration seconds metric.*Shown as second*                                                                                                                                                                                                                                                               |
| **kueue.workqueue.retries.count**(count)                                       | Kueue workqueue retries metric.                                                                                                                                                                                                                                                                                               |
| **kueue.workqueue.unfinished_work.seconds**(gauge)                            | Kueue workqueue unfinished work seconds metric.*Shown as second*                                                                                                                                                                                                                                                              |
| **kueue.workqueue.work_duration.seconds.bucket**(count)                       | Kueue workqueue work duration seconds metric.*Shown as second*                                                                                                                                                                                                                                                                |
| **kueue.workqueue.work_duration.seconds.count**(count)                        | Kueue workqueue work duration seconds metric.*Shown as second*                                                                                                                                                                                                                                                                |
| **kueue.workqueue.work_duration.seconds.sum**(count)                          | Kueue workqueue work duration seconds metric.*Shown as second*                                                                                                                                                                                                                                                                |

### Events{% #events %}

By default, the Kueue integration polls the Kueue Workload custom resources and sends Datadog events for lifecycle transitions:

- `kueue.workload.created`: a Workload appears after the check has initialized its state.
- `kueue.workload.pending`: the Workload cannot reserve quota and its `QuotaReserved` condition becomes `False` with reason `Pending` or `Inadmissible`.
- `kueue.workload.quota_reserved`: the `QuotaReserved` condition becomes `True`.
- `kueue.workload.admitted`: the `Admitted` condition becomes `True`.
- `kueue.workload.running`: the `PodsReady` condition becomes `True`.
- `kueue.workload.evicted`: the `Evicted` condition becomes `True`.
- `kueue.workload.finished`: the `Finished` condition becomes `True`.

Events are tagged with the Workload namespace, name, UID, LocalQueue, transition, priority, and ClusterQueue when available. Instance tags, including the tags you set in `tags` and the tags Autodiscovery adds from the controller pod, apply to metrics only, not to events. Eviction events also include the eviction reason, and preemption events include the Kueue preemption reason when available. Pending events include the reason reported by Kueue.

Events also carry the scheduler tags that the Agent adds to the pods of every batch scheduler, so that the same queries work for Kueue and other schedulers: `job_scheduler:kueue`, `job_scheduler_workload` (the Workload name), and `job_scheduler_queue` (the ClusterQueue, when the Workload has been admitted).

Each event also carries `kueue_workload_event_index`, a per-Workload integer that increases for every event the check emits for that Workload. Kueue often stamps several transitions with the same second-resolution timestamp; sort by `timestamp`, then by `kueue_workload_event_index` as an integer, to order those events. The index is kept in the Agent persistent cache so a process restart continues the sequence for Workloads still present in the API. The cache is cleared when the check configuration changes, and a new pod starts the sequence at `0` because the run directory is not preserved across container recreation.

The first collection run seeds the Workload state and does not emit events for already existing transitions.

### Service Checks{% #service-checks %}

**kueue.openmetrics.health**

Returns `CRITICAL` if the Agent is unable to connect to the Kueue OpenMetrics endpoint, otherwise returns `OK`.

*Statuses: ok, critical*

## Uninstallation{% #uninstallation %}

To disable the integration, remove the `kueue.yaml` entry from the Cluster Agent `confd` configuration, or remove the Autodiscovery annotations from the Kueue metrics Service.

## Support{% #support %}

Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).
