---
title: KAI Scheduler
description: Monitor KAI Scheduler quotas, fair share, scheduling latency, and preemption
breadcrumbs: Docs > Integrations > KAI Scheduler
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# KAI Scheduler
Supported OS Integration version1.0.0
{% callout %}
# Important note for users on the following Datadog sites: us2.ddog-gov.com

{% alert level="info" %}
To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

This check monitors [KAI Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) through the Datadog Agent.

KAI Scheduler is a Kubernetes scheduler for AI and GPU workloads. It schedules pods into hierarchical queues with quotas, fair share, priorities, and preemption. This integration collects metrics from the KAI scheduler and queue-controller to help you monitor queue quotas and allocation, fair share, scheduling latency, and preemption, and sends events for PodGroup life cycle transitions.

## Setup{% #setup %}

### Installation{% #installation %}

The KAI Scheduler check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package. No additional installation is required on your server.

### Configuration{% #configuration %}

This integration is part of GPU Monitoring. Enable GPU Monitoring in the configuration of the Agent that runs the check, which is the Cluster Check Runner or node Agent that the cluster check is dispatched to:

```yaml
gpu:
  enabled: true
```

The equivalent environment variable is `DD_GPU_ENABLED=true`. When GPU Monitoring is disabled, the Agent skips every configured KAI Scheduler instance.

KAI Scheduler exposes its metrics from two deployments in the `kai-scheduler` namespace. Each one serves Prometheus metrics over plain HTTP on port `8080`:

- The scheduler, through the `kai-scheduler-<shard>` service (`kai-scheduler-default` for the default shard), serves scheduling cycle, action, preemption, and queue fair share metrics.
- The queue-controller, through the `queue-controller` service, serves queue quota and allocation metrics.

A single instance scrapes any combination of these endpoints. Each metric is tagged with `kai_component` (`scheduler` or `queue_controller`) so that the generic metrics of each component can be told apart.

KAI Scheduler is a cluster-level service. Configure this integration as a Cluster Agent [cluster check](https://docs.datadoghq.com/containers/cluster_agent/clusterchecks.md?tab=helm#configuration-from-configuration-files) so only one Agent instance scrapes the endpoints. For file- or ConfigMap-based configuration, set `cluster_check: true` in the instance:

```yaml
clusterAgent:
  confd:
    kai_scheduler.yaml: |-
      cluster_check: true
      init_config:
      instances:
      - scheduler_endpoint: http://kai-scheduler-default.kai-scheduler.svc:8080/metrics
        queue_controller_endpoint: http://queue-controller.kai-scheduler.svc:8080/metrics
```

If you run several scheduling shards, add one `scheduler_endpoint` instance per shard service. If you enable `collect_podgroup_events` on more than one instance, each instance polls every PodGroup in the cluster and reports the same events. Set `collect_podgroup_events: false` on all instances except one.

#### Binder metrics (optional){% #binder-metrics-optional %}

The binder, through the `binder` service, binds the pods that the scheduler places. It exposes no KAI-specific metrics, only generic controller-runtime, Go, and process metrics. Its `bindrequest` controller reconcile errors, reconcile latency, and work queue depth show pods that the scheduler placed but that failed to bind, for example because of a GPU sharing reservation or volume error. To collect them, add `binder_endpoint: http://binder.kai-scheduler.svc:8080/metrics` to the instance. These metrics are tagged with `kai_component:binder`.

See the [sample kai_scheduler.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/kai_scheduler/datadog_checks/kai_scheduler/data/conf.yaml.example) for all available configuration options.

#### Metrics namespace{% #metrics-namespace %}

The check expects KAI's default metrics namespace, `kai`. Do not change it with the `--metrics-namespace` flag or the `queueController.metricsNamespace` setting. The scheduler also exposes an unprefixed copy of each of its metrics that is never updated, so the check only collects the `kai_`-prefixed names.

### Validation{% #validation %}

[Run the Cluster Agent's `clusterchecks` subcommand](https://docs.datadoghq.com/containers/troubleshooting/cluster-and-endpoint-checks.md#dispatching-logic-in-the-cluster-agent) and look for `kai_scheduler` under the Checks section.

## Data Collected{% #data-collected %}

### Metrics{% #metrics %}

|  |
|  |
| **kai.action.latency.milliseconds**(gauge)                          | Duration of the last run of each scheduling action.*Shown as millisecond*                                                                           |
| **kai.action.podgroups.attempted.count**(count)                     | Number of pod groups each scheduling action attempted to schedule.                                                                                  |
| **kai.action.podgroups.scheduled.count**(count)                     | Number of pod groups each scheduling action scheduled.                                                                                              |
| **kai.action.scenarios.filtered.count**(count)                      | Number of scenarios each scheduling action filtered out before simulation.                                                                          |
| **kai.action.scenarios.simulated.count**(count)                     | Number of scenarios each scheduling action simulated.                                                                                               |
| **kai.certwatcher.read_certificate.count**(count)                  | Number of TLS certificate reads by the controller-runtime certificate watcher.                                                                      |
| **kai.certwatcher.read_certificate_errors.count**(count)          | Number of failed TLS certificate reads by the controller-runtime certificate watcher.                                                               |
| **kai.controller.runtime.active_workers**(gauge)                   | Number of reconcile workers currently in use, per controller.                                                                                       |
| **kai.controller.runtime.conversion_webhook_panics.count**(count) | Number of panics in the conversion webhook.                                                                                                         |
| **kai.controller.runtime.max_concurrent_reconciles**(gauge)       | Maximum number of concurrent reconciles allowed, per controller.                                                                                    |
| **kai.controller.runtime.reconcile.count**(count)                   | Number of reconciliations, per controller and result.                                                                                               |
| **kai.controller.runtime.reconcile_errors.count**(count)           | Number of reconciliations that returned an error, per controller.                                                                                   |
| **kai.controller.runtime.reconcile_panics.count**(count)           | Number of reconciliations that panicked, per controller.                                                                                            |
| **kai.controller.runtime.reconcile_time.seconds**(gauge)           | Distribution of reconciliation durations, per controller.*Shown as second*                                                                          |
| **kai.controller.runtime.reconcile_time.seconds.count**(count)     | Number of timed reconciliations, per controller.                                                                                                    |
| **kai.controller.runtime.reconcile_time.seconds.sum**(count)       | Total time spent reconciling, per controller.*Shown as second*                                                                                      |
| **kai.controller.runtime.reconcile_timeouts.count**(count)         | Number of reconciliations that timed out, per controller.                                                                                           |
| **kai.controller.runtime.terminal_reconcile_errors.count**(count) | Number of reconciliations that returned a terminal error, which is not retried, per controller.                                                     |
| **kai.controller.runtime.webhook_latency.seconds**(gauge)          | Distribution of admission webhook request latencies, per webhook path.*Shown as second*                                                             |
| **kai.controller.runtime.webhook_latency.seconds.count**(count)    | Number of timed admission webhook requests, per webhook path.                                                                                       |
| **kai.controller.runtime.webhook_latency.seconds.sum**(count)      | Total time spent processing admission webhook requests, per webhook path.*Shown as second*                                                          |
| **kai.controller.runtime.webhook_panics.count**(count)             | Number of panics in admission webhooks.                                                                                                             |
| **kai.controller.runtime.webhook_requests.count**(count)           | Number of admission webhook requests, per HTTP status code.                                                                                         |
| **kai.controller.runtime.webhook_requests_in_flight**(gauge)     | Number of admission webhook requests currently being served.                                                                                        |
| **kai.evicted_pods.count**(count)                                  | Number of pods evicted, by namespace, node pool, and action.                                                                                        |
| **kai.go.gc.duration.seconds.count**(count)                         | The summary count of garbage collection cycles.                                                                                                     |
| **kai.go.gc.duration.seconds.quantile**(gauge)                      | A summary of the pause duration of garbage collection cycles.*Shown as second*                                                                      |
| **kai.go.gc.duration.seconds.sum**(count)                           | The sum of the pause duration of garbage collection cycles.*Shown as second*                                                                        |
| **kai.go.gc.gogc.percent**(gauge)                                   | Heap growth percentage that triggers garbage collection, set by `GOGC`. 100 by default.*Shown as percent*                                           |
| **kai.go.gc.gomemlimit.bytes**(gauge)                               | Soft memory limit of the Go runtime, set by `GOMEMLIMIT`. The maximum int64 value when unset.*Shown as byte*                                        |
| **kai.go.goroutines**(gauge)                                        | Number of goroutines that currently exist.                                                                                                          |
| **kai.go.info**(gauge)                                              | Always 1. The `go_version` tag holds the Go version the component was built with.                                                                   |
| **kai.go.memstats.alloc_bytes**(gauge)                             | Heap memory allocated to objects and still in use.*Shown as byte*                                                                                   |
| **kai.go.memstats.alloc_bytes.count**(count)                       | Heap memory allocated since the process started, including memory already freed.*Shown as byte*                                                     |
| **kai.go.memstats.buck_hash.sys_bytes**(gauge)                    | Memory used by the profiling bucket hash table.*Shown as byte*                                                                                      |
| **kai.go.memstats.frees.count**(count)                              | Number of heap objects freed.                                                                                                                       |
| **kai.go.memstats.gc.sys_bytes**(gauge)                            | Memory used for garbage collection metadata.*Shown as byte*                                                                                         |
| **kai.go.memstats.heap_alloc.bytes**(gauge)                        | Heap memory allocated to objects and still in use. Same value as `kai.go.memstats.alloc_bytes`.*Shown as byte*                                      |
| **kai.go.memstats.heap_idle.bytes**(gauge)                         | Heap memory not in use, either free or returned to the operating system.*Shown as byte*                                                             |
| **kai.go.memstats.heap_inuse.bytes**(gauge)                        | Heap memory in use, including space reserved for objects but not yet used.*Shown as byte*                                                           |
| **kai.go.memstats.heap_objects**(gauge)                            | Number of allocated heap objects, live or not yet swept.                                                                                            |
| **kai.go.memstats.heap_released.bytes**(gauge)                     | Heap memory returned to the operating system.*Shown as byte*                                                                                        |
| **kai.go.memstats.heap_sys.bytes**(gauge)                          | Heap memory obtained from the operating system.*Shown as byte*                                                                                      |
| **kai.go.memstats.mallocs.count**(count)                            | Number of heap objects allocated.                                                                                                                   |
| **kai.go.memstats.mcache_inuse.bytes**(gauge)                      | Memory in use by mcache structures, the per-processor allocation caches.*Shown as byte*                                                             |
| **kai.go.memstats.mcache_sys.bytes**(gauge)                        | Memory obtained from the operating system for mcache structures.*Shown as byte*                                                                     |
| **kai.go.memstats.mspan_inuse.bytes**(gauge)                       | Memory in use by mspan structures, which track heap spans.*Shown as byte*                                                                           |
| **kai.go.memstats.mspan_sys.bytes**(gauge)                         | Memory obtained from the operating system for mspan structures.*Shown as byte*                                                                      |
| **kai.go.memstats.next_gc.bytes**(gauge)                           | Target heap size of the next garbage collection cycle.*Shown as byte*                                                                               |
| **kai.go.memstats.other_sys.bytes**(gauge)                         | Memory used for other runtime allocations, such as trace buffers and profiling.*Shown as byte*                                                      |
| **kai.go.memstats.stack_inuse.bytes**(gauge)                       | Memory used by goroutine stacks.*Shown as byte*                                                                                                     |
| **kai.go.memstats.stack_sys.bytes**(gauge)                         | Memory obtained from the operating system for stacks, including OS thread stacks.*Shown as byte*                                                    |
| **kai.go.memstats.sys.bytes**(gauge)                                | Total memory obtained from the operating system by the Go runtime.*Shown as byte*                                                                   |
| **kai.go.memstats.time_since_last_gc.seconds**(gauge)            | Time since the last garbage collection.*Shown as second*                                                                                            |
| **kai.go.threads**(gauge)                                           | Number of operating system threads created by the Go runtime.                                                                                       |
| **kai.leader_election.master_status**(gauge)                      | Whether this replica holds the leader election lease: 1 for the leader, 0 for a standby.                                                            |
| **kai.plugin.latency.milliseconds**(gauge)                          | Duration of the last OnSessionOpen or OnSessionClose call of each scheduler plugin.*Shown as millisecond*                                           |
| **kai.preemption.attempts.count**(count)                            | Number of preemption attempts.*Shown as attempt*                                                                                                    |
| **kai.process.cpu.seconds.count**(count)                            | User and system CPU time used by the component process.*Shown as second*                                                                            |
| **kai.process.max_fds**(gauge)                                     | Maximum number of file descriptors the component process can open.                                                                                  |
| **kai.process.network_receive.bytes.count**(count)                 | Bytes received over the network by the component process.*Shown as byte*                                                                            |
| **kai.process.network_transmit.bytes.count**(count)                | Bytes sent over the network by the component process.*Shown as byte*                                                                                |
| **kai.process.open_fds**(gauge)                                    | Number of file descriptors open by the component process.                                                                                           |
| **kai.process.resident_memory.bytes**(gauge)                       | Resident memory size of the component process.*Shown as byte*                                                                                       |
| **kai.process.uptime.seconds**(gauge)                               | Time since the component process started.*Shown as second*                                                                                          |
| **kai.process.virtual_memory.bytes**(gauge)                        | Virtual memory size of the component process.*Shown as byte*                                                                                        |
| **kai.process.virtual_memory.max_bytes**(gauge)                   | Maximum virtual memory available to the component process.*Shown as byte*                                                                           |
| **kai.queue.allocated.cpu**(gauge)                                  | Resources currently allocated to the queue. Resource: CPU.*Shown as core*                                                                           |
| **kai.queue.allocated.gpu**(gauge)                                  | Resources currently allocated to the queue. Resource: GPU.*Shown as device*                                                                         |
| **kai.queue.allocated.memory**(gauge)                               | Resources currently allocated to the queue. Resource: memory.*Shown as byte*                                                                        |
| **kai.queue.fair_share.cpu**(gauge)                                | Fair share of the queue computed by the scheduler in its last cycle. Resource: CPU.*Shown as core*                                                  |
| **kai.queue.fair_share.gpu**(gauge)                                | Fair share of the queue computed by the scheduler in its last cycle. Resource: GPU.*Shown as device*                                                |
| **kai.queue.fair_share.memory**(gauge)                             | Fair share of the queue computed by the scheduler in its last cycle. Resource: memory.*Shown as byte*                                               |
| **kai.queue.info**(gauge)                                           | Always 1 for each queue known to the queue-controller.                                                                                              |
| **kai.queue.quota.cpu**(gauge)                                      | Quota of the queue, from `spec.resources`. -1 means unlimited. Resource: CPU.*Shown as core*                                                        |
| **kai.queue.quota.gpu**(gauge)                                      | Quota of the queue, from `spec.resources`. -1 means unlimited. Resource: GPU.*Shown as device*                                                      |
| **kai.queue.quota.memory**(gauge)                                   | Quota of the queue, from `spec.resources`. -1 means unlimited. Resource: memory.*Shown as byte*                                                     |
| **kai.queue.usage.cpu**(gauge)                                      | Historical usage of the queue reported by the UsageDB. Units depend on the UsageDB configuration, and the value is 0 without one. Resource: CPU.    |
| **kai.queue.usage.gpu**(gauge)                                      | Historical usage of the queue reported by the UsageDB. Units depend on the UsageDB configuration, and the value is 0 without one. Resource: GPU.    |
| **kai.queue.usage.memory**(gauge)                                   | Historical usage of the queue reported by the UsageDB. Units depend on the UsageDB configuration, and the value is 0 without one. Resource: memory. |
| **kai.scenario_search.budget.action.seconds**(gauge)               | Configured scenario search budget per action. 0 means unlimited.*Shown as second*                                                                   |
| **kai.scenario_search.budget.generator.seconds**(gauge)            | Configured scenario search budget per generator. 0 means unlimited.*Shown as second*                                                                |
| **kai.scenario_search.budget.job.seconds**(gauge)                  | Configured scenario search budget per job. 0 means unlimited.*Shown as second*                                                                      |
| **kai.scenario_search.budget_exhausted.count**(count)             | Number of times an action exhausted its scenario search budget.*Shown as event*                                                                     |
| **kai.scenario_search.duration.seconds**(gauge)                    | Duration of bounded scenario search attempts per generator, in seconds.*Shown as second*                                                            |
| **kai.scenario_search.duration.seconds.count**(count)              | Duration of bounded scenario search attempts per generator, in seconds. Number of observations.                                                     |
| **kai.scenario_search.duration.seconds.sum**(count)                | Duration of bounded scenario search attempts per generator, in seconds. Sum of observations.*Shown as second*                                       |
| **kai.scenario_search.jobs.count**(count)                          | Number of jobs considered by bounded scenario search, by action, result and whether the budget was reduced.*Shown as job*                           |
| **kai.scenario_search.scenarios.count**(count)                     | Number of bounded search scenarios, by action, generator and state.                                                                                 |
| **kai.scheduling_cycle.latency.milliseconds**(gauge)               | Duration of the last full scheduling cycle, including all actions and binding.*Shown as millisecond*                                                |
| **kai.session.close.latency.milliseconds**(gauge)                   | Duration of the last session close, including plugin cleanup.*Shown as millisecond*                                                                 |
| **kai.session.open.latency.milliseconds**(gauge)                    | Duration of the last session open, including cluster snapshot and plugin initialization.*Shown as millisecond*                                      |
| **kai.task.bind_latency.milliseconds**(gauge)                      | Time the binder takes to bind a pod, including bind request creation, in milliseconds.*Shown as millisecond*                                        |
| **kai.task.bind_latency.milliseconds.count**(count)                | Time the binder takes to bind a pod, including bind request creation, in milliseconds. Number of observations.                                      |
| **kai.task.bind_latency.milliseconds.sum**(count)                  | Time the binder takes to bind a pod, including bind request creation, in milliseconds. Sum of observations.*Shown as millisecond*                   |
| **kai.task.scheduling_latency.milliseconds**(gauge)                | Time from pod creation until the scheduler attempts to bind it, in milliseconds.*Shown as millisecond*                                              |
| **kai.task.scheduling_latency.milliseconds.count**(count)          | Time from pod creation until the scheduler attempts to bind it, in milliseconds. Number of observations.                                            |
| **kai.task.scheduling_latency.milliseconds.sum**(count)            | Time from pod creation until the scheduler attempts to bind it, in milliseconds. Sum of observations.*Shown as millisecond*                         |
| **kai.usage_query.latency.milliseconds**(gauge)                    | Latency of UsageDB queries, in milliseconds. Only reported when a UsageDB is configured.*Shown as millisecond*                                      |
| **kai.usage_query.latency.milliseconds.count**(count)              | Latency of UsageDB queries, in milliseconds. Only reported when a UsageDB is configured. Number of observations.                                    |
| **kai.usage_query.latency.milliseconds.sum**(count)                | Latency of UsageDB queries, in milliseconds. Only reported when a UsageDB is configured. Sum of observations.*Shown as millisecond*                 |
| **kai.workqueue.adds.count**(count)                                 | Number of items added to the work queue, per queue.                                                                                                 |
| **kai.workqueue.depth**(gauge)                                      | Number of items waiting in the work queue, per queue.                                                                                               |
| **kai.workqueue.longest_running_processor.seconds**(gauge)        | How long the longest-running item in the work queue has been processed, per queue.*Shown as second*                                                 |
| **kai.workqueue.queue_duration.seconds**(gauge)                    | Distribution of the time items wait in the work queue before being processed, per queue.*Shown as second*                                           |
| **kai.workqueue.queue_duration.seconds.count**(count)              | Number of items taken from the work queue, per queue.                                                                                               |
| **kai.workqueue.queue_duration.seconds.sum**(count)                | Total time items waited in the work queue before being processed, per queue.*Shown as second*                                                       |
| **kai.workqueue.retries.count**(count)                              | Number of items requeued for a retry, per queue.                                                                                                    |
| **kai.workqueue.unfinished_work.seconds**(gauge)                   | Time spent on items still being processed, per queue. Large values indicate stuck workers.*Shown as second*                                         |
| **kai.workqueue.work_duration.seconds**(gauge)                     | Distribution of the time taken to process a work queue item, per queue.*Shown as second*                                                            |
| **kai.workqueue.work_duration.seconds.count**(count)               | Number of work queue items processed, per queue.                                                                                                    |
| **kai.workqueue.work_duration.seconds.sum**(count)                 | Total time spent processing work queue items, per queue.*Shown as second*                                                                           |

Queue metrics from the scheduler and the queue-controller share the `kai_queue` tag, which holds the queue's `metadata.name`, so that quota, allocation, fair share, and usage can be compared per queue. Queues with a `spec.displayName` are also tagged with `kai_queue_display_name`. Tags without a value, such as labels that KAI leaves empty, are not submitted. Memory metrics are reported in bytes, CPU metrics in cores, and GPU metrics in devices.

KAI counts evicted pods per pod group and never deletes those series. To keep the number of contexts bounded, `kai.evicted_pods.count` reports the pods evicted since the previous check run, summed by `kube_namespace`, `kai_nodepool`, and `action`. Evictions that happened before the first successful check run are not reported. The check keeps the previous value of up to 10,000 pod group series. Beyond that, it reports the increase of the summed counters instead, which undercounts evictions in a group whose pod group series KAI removed during that check run. To see which workloads were evicted and by which preemptor, use the `kai.podgroup.evicted` events.

The `kai.scheduling_cycle.latency.milliseconds`, `kai.session.*.latency.milliseconds`, `kai.action.latency.milliseconds`, and `kai.plugin.latency.milliseconds` gauges hold the duration of the most recent scheduling cycle only. The scheduler runs a cycle every second by default, so each check run samples one cycle and short spikes between check runs are not visible. A scheduler that is consistently slow is still visible in every sample.

Histograms, such as `kai.task.scheduling_latency.milliseconds` and `kai.task.bind_latency.milliseconds`, are submitted as [distributions](https://docs.datadoghq.com/metrics/distributions.md) so that percentiles can be queried, along with their `.count` and `.sum` counters. Set `histogram_buckets_as_distributions: false` to submit the buckets as `.bucket` counts instead.

### Events{% #events %}

By default, the KAI Scheduler integration polls the KAI PodGroup custom resources and sends Datadog events for life cycle transitions:

- **Created** (`kai.podgroup.created`): A PodGroup appears after the check has initialized its state.
- **Pending** (`kai.podgroup.pending`): The scheduler cannot schedule the PodGroup on a node pool and adds an `UnschedulableOnNodePool` scheduling condition for it.
- **Admitted** (`kai.podgroup.admitted`): The scheduler allocates pods to a PodGroup that has none allocated, such as a new PodGroup, an evicted one, or a Job that starts its next pod, and updates its `kai.scheduler/last-start-timestamp` annotation.
- **Evicted** (`kai.podgroup.evicted`): The scheduler evicts some or all of the PodGroup's pods to preempt, reclaim, or consolidate, or because they are background pods whose capacity it gives to other pods, and updates its `kai.scheduler/last-eviction-timestamp` annotation. KAI sets this annotation starting with version `0.17.0`. Stale gang evictions do not update it, so they are only counted in `kai.evicted_pods.count`.

KAI does not record quota reservation, pod readiness, or completion on the PodGroup, so there are no equivalents of the Kueue `quota_reserved`, `running`, and `finished` Workload events.

Events are tagged with `kube_namespace`, `kai_pod_group` (the PodGroup name), `kai_pod_group_uid`, `kai_queue`, `kai_nodepool`, `kai_transition`, and `kai_pod_group_priority_class` when the PodGroup sets a priority class. Events also carry the scheduler tags shared by all batch schedulers, so that the same queries work for KAI and Kueue: `job_scheduler:kai`, `job_scheduler_queue` (the queue name), and `job_scheduler_workload` (the PodGroup name). The `kai_queue` and `job_scheduler_queue` tags match the tags the Agent adds to the metrics of the PodGroup's pods, and so do `kai_pod_group` and `job_scheduler_workload` when the Agent collects tags with `orchestrator` cardinality. Pending events take the node pool from the scheduling condition, and other events from the `kai.scheduler/node-pool` PodGroup label, so other events are tagged `kai_nodepool:default` when KAI is configured with another `global.nodePoolLabelKey`. Instance tags, including the tags you set in `tags` and the tags Autodiscovery adds from the KAI pods, apply to metrics only, not to events.

Pending events include the scheduling error and a `kai_pending_reason` tag for each reason KAI reports, such as `OverLimit`. Admitted events include the queued wait time when the check can determine it, counted from the PodGroup creation or from the latest eviction. After an eviction of only some of the pods, it can also include time the PodGroup spent running. Eviction events include the message and the eviction action, and are tagged with `kai_eviction_action` (such as `preempt` or `reclaim`) and `kai_evicted_by` (the preemptor PodGroup), when available from the `Evict` Kubernetes event that KAI records on the PodGroup.

Each event also includes `kai_pod_group_event_index`, an integer that goes up by one for each event the check emits for a PodGroup. KAI timestamps transitions to the second, so several events can have the same timestamp. To put events in order, sort by timestamp first, then by `kai_pod_group_event_index` as an integer.

The Agent stores the index in its persistent cache. After a process restart, the sequence picks up where it left off for PodGroups that still exist in the API. The sequence restarts at 0 in two cases:

- The check configuration changes, which clears the cache.
- A new pod starts. The run directory doesn't survive container recreation.

The first collection run records the current state of each PodGroup. It doesn't emit events for transitions that happened before that run.

Collecting events requires `list` role-based access control (RBAC) permission on `podgroups.scheduling.run.ai`, and `list` on `events` for the eviction details. Every instance with `collect_podgroup_events` enabled polls all the PodGroups of the cluster, so when you configure several instances, such as one per scheduling shard, set `collect_podgroup_events: false` on all but one of them. Set it to `false` on every instance to disable events.

### Service Checks{% #service-checks %}

The KAI Scheduler integration does not include any service checks of its own. The `kai.openmetrics.health` service check, inherited from the OpenMetrics base check, reports whether each endpoint can be scraped.

## Troubleshooting{% #troubleshooting %}

Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).
