--- title: Kubernetes API server metrics description: Collect metrics from the Kubernetes APIserver breadcrumbs: Docs > Integrations > Kubernetes API server metrics --- # Kubernetes API server metrics Supported OS Integration version7.5.0 ## Overview{% #overview %} This check monitors [Kube_apiserver_metrics](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver). **Minimum Agent version:** 6.11.3 ## Setup{% #setup %} ### Installation{% #installation %} The Kube_apiserver_metrics check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package, so you do not need to install anything else on your server. ### Configuration{% #configuration %} If your Kubernetes clusters have master nodes and is running a pod and container for the `kube-apiserver` image, the Datadog Agent [automatically discovers](https://docs.datadoghq.com/agent/kubernetes/integrations.md) this pod and configures the integration relative to its `kube_apiserver_metrics.d/auto_conf.yaml` file. However, if you are using a managed Kubernetes distribution like GKE, EKS, or AKS you may not have a running `kube-apiserver` pod present for the Agent to discover. In this case, you can setup the integration against the `kubernetes` Service in the `default` namespace. - The main use case to run the `kube_apiserver_metrics` check is as a [Cluster Level Check](https://docs.datadoghq.com/agent/cluster_agent/clusterchecks.md). - You can do this with annotations on your service, or by using a local file through the Datadog Operator, Helm Chart or manually. - To collect metrics, set the following parameters and values in an [Autodiscovery](https://docs.datadoghq.com/agent/kubernetes/integrations.md) template. | Parameter | Value | | -------------------- | ----------------------------------------------------------- | | `` | `["kube_apiserver_metrics"]` | | `` | `[{}]` | | `` | `[{"prometheus_url": "https://%%host%%:%%port%%/metrics"}]` | You can review all available configuration options in the [kube_apiserver_metrics.yaml](https://github.com/DataDog/integrations-core/blob/master/kube_apiserver_metrics/datadog_checks/kube_apiserver_metrics/data/conf.yaml.example). #### Annotate service{% #annotate-service %} You can annotate the kubernetes service in your `default` namespace with the following: {% tab title="Annotations v2 (for Datadog Agent v7.36)" %} ```yaml ad.datadoghq.com/endpoints.checks: | { "kube_apiserver_metrics": { "instances": [ { "prometheus_url": "https://%%host%%:%%port%%/metrics" } ] } } ``` {% /tab %} {% tab title="Annotations v1 (for Datadog Agent \\< v7.36)" %} ```yaml annotations: ad.datadoghq.com/endpoints.check_names: '["kube_apiserver_metrics"]' ad.datadoghq.com/endpoints.init_configs: '[{}]' ad.datadoghq.com/endpoints.instances: '[{ "prometheus_url": "https://%%host%%:%%port%%/metrics"}]' ``` {% /tab %} Then the Datadog Cluster Agent schedules the check(s) for each endpoint onto Datadog Agent(s). #### Local file{% #local-file %} You can also run the check by configuring the endpoints directly in the `kube_apiserver_metrics.yaml` file, in the `conf.d/` folder at the root of your [Agent's configuration directory](https://docs.datadoghq.com/agent/guide/agent-configuration-files.md#agent-configuration-directory) to dispatch as a [Cluster Check](https://docs.datadoghq.com//containers/cluster_agent/clusterchecks.md?tab=datadogoperator#setting-up-check-configurations). **Note**: You must add `cluster_check: true` to your configuration file if using a local file or ConfigMap to configure Cluster Checks. Provide a [configuration](https://docs.datadoghq.com/containers/cluster_agent/clusterchecks.md?tab=helm#configuration-from-configuration-files) to your Cluster Agent to setup a Cluster Check: {% tab title="Helm" %} ```yaml clusterAgent: confd: kube_apiserver_metrics.yaml: |- advanced_ad_identifiers: - kube_endpoints: name: "kubernetes" namespace: "default" cluster_check: true init_config: instances: - prometheus_url: "https://%%host%%:%%port%%/metrics" ``` {% /tab %} {% tab title="Operator" %} ```yaml spec: #(...) override: clusterAgent: extraConfd: configDataMap: kube_apiserver_metrics.yaml: |- advanced_ad_identifiers: - kube_endpoints: name: "kubernetes" namespace: "default" cluster_check: true init_config: instances: - prometheus_url: "https://%%host%%:%%port%%/metrics" ``` {% /tab %} These configurations trigger the Agent to make a request to the `kubernetes` service in the `default` namespace at its defined Endpoint IP Addresses and defined port. ### Validation{% #validation %} [Run the Agent's status subcommand](https://docs.datadoghq.com/agent/faq/agent-commands.md#agent-status-and-information) and look for `kube_apiserver_metrics` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **kube\_apiserver.APIServiceRegistrationController\_depth**(gauge) | The current depth of workqueue: APIServiceRegistrationController | | **kube\_apiserver.admission\_controller\_admission\_duration\_seconds.count**(count) | The admission controller latency histogram in seconds identified by name and broken out for each operation and API resource and type (validate or admit) count | | **kube\_apiserver.admission\_controller\_admission\_duration\_seconds.sum**(gauge) | The admission controller latency histogram in seconds identified by name and broken out for each operation and API resource and type (validate or admit)*Shown as second* | | **kube\_apiserver.admission\_step\_admission\_latencies\_seconds.count**(count) | The admission sub-step latency histogram broken out for each operation and API resource and step type (validate or admit) count | | **kube\_apiserver.admission\_step\_admission\_latencies\_seconds.sum**(gauge) | The admission sub-step latency broken out for each operation and API resource and step type (validate or admit)*Shown as second* | | **kube\_apiserver.admission\_step\_admission\_latencies\_seconds\_summary.count**(count) | The admission sub-step latency summary broken out for each operation and API resource and step type (validate or admit) count | | **kube\_apiserver.admission\_step\_admission\_latencies\_seconds\_summary.quantile**(gauge) | The admission sub-step latency summary broken out for each operation and API resource and step type (validate or admit) quantile*Shown as second* | | **kube\_apiserver.admission\_step\_admission\_latencies\_seconds\_summary.sum**(gauge) | The admission sub-step latency summary broken out for each operation and API resource and step type (validate or admit)*Shown as second* | | **kube\_apiserver.admission\_webhook\_admission\_latencies\_seconds.count**(count) | The admission webhook latency identified by name and broken out for each operation and API resource and type (validate or admit) count | | **kube\_apiserver.admission\_webhook\_admission\_latencies\_seconds.sum**(gauge) | The admission webhook latency identified by name and broken out for each operation and API resource and type (validate or admit)*Shown as second* | | **kube\_apiserver.aggregator\_unavailable\_apiservice**(gauge) | Gauge of APIServices which are marked as unavailable broken down by APIService name (alpha; Kubernetes 1.14+) | | **kube\_apiserver.apiserver\_admission\_webhook\_fail\_open\_count**(gauge) | Admission webhook fail open count, identified by name and broken out for each admission type (validating or mutating). | | **kube\_apiserver.apiserver\_admission\_webhook\_fail\_open\_count.count**(count) | Admission webhook fail open count, identified by name and broken out for each admission type (validating or mutating). | | **kube\_apiserver.apiserver\_admission\_webhook\_request\_total**(gauge) | Admission webhook request total, identified by name and broken out for each admission type (alpha; Kubernetes 1.23+) | | **kube\_apiserver.apiserver\_admission\_webhook\_request\_total.count**(count) | Admission webhook request total, identified by name and broken out for each admission type (alpha; Kubernetes 1.23+) | | **kube\_apiserver.apiserver\_dropped\_requests\_total**(gauge) | The accumulated number of requests dropped with 'Try again later' response*Shown as request* | | **kube\_apiserver.apiserver\_dropped\_requests\_total.count**(count) | The monotonic count of requests dropped with 'Try again later' response*Shown as request* | | **kube\_apiserver.apiserver\_request\_count**(gauge) | The accumulated number of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (deprecated in Kubernetes 1.15)*Shown as request* | | **kube\_apiserver.apiserver\_request\_count.count**(count) | The monotonic count of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (deprecated in Kubernetes 1.15)*Shown as request* | | **kube\_apiserver.apiserver\_request\_terminations\_total.count**(count) | The number of requests the apiserver terminated in self-defense (Kubernetes 1.17+)*Shown as request* | | **kube\_apiserver.apiserver\_request\_total**(gauge) | The accumulated number of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (Kubernetes 1.15+; replaces apiserver_request_count)*Shown as request* | | **kube\_apiserver.apiserver\_request\_total.count**(count) | The monotonic count of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (Kubernetes 1.15+; replaces apiserver_request_count.count)*Shown as request* | | **kube\_apiserver.audit\_event**(gauge) | The accumulated number audit events generated and sent to the audit backend*Shown as event* | | **kube\_apiserver.audit\_event.count**(count) | The monotonic count of audit events generated and sent to the audit backend*Shown as event* | | **kube\_apiserver.authenticated\_user\_requests**(gauge) | The accumulated number of authenticated requests broken out by username*Shown as request* | | **kube\_apiserver.authenticated\_user\_requests.count**(count) | The monotonic count of authenticated requests broken out by username*Shown as request* | | **kube\_apiserver.authentication\_attempts.count**(count) | The counter of authenticated attempts (Kubernetes 1.16+)*Shown as request* | | **kube\_apiserver.authentication\_duration\_seconds.count**(count) | The authentication duration histogram broken out by result (Kubernetes 1.17+) | | **kube\_apiserver.authentication\_duration\_seconds.sum**(gauge) | The authentication duration histogram broken out by result (Kubernetes 1.17+)*Shown as second* | | **kube\_apiserver.current\_inflight\_requests**(gauge) | The maximal number of currently used inflight request limit of this apiserver per request kind in last second. | | **kube\_apiserver.envelope\_encryption\_dek\_cache\_fill\_percent**(gauge) | Percent of the cache slots currently occupied by cached DEKs. | | **kube\_apiserver.etcd.db.total\_size**(gauge) | The total size of the etcd database file physically allocated in bytes (alpha; Kubernetes 1.19+)*Shown as byte* | | **kube\_apiserver.etcd\_object\_counts**(gauge) | The number of stored objects at the time of last check split by kind (alpha; deprecated in Kubernetes 1.22)*Shown as object* | | **kube\_apiserver.etcd\_request\_duration\_seconds.count**(count) | Etcd request latencies count for each operation and object type (alpha) | | **kube\_apiserver.etcd\_request\_duration\_seconds.sum**(gauge) | Etcd request latencies for each operation and object type (alpha)*Shown as second* | | **kube\_apiserver.etcd\_request\_errors\_total**(count) | Etcd failed request counts for each operation and object type*Shown as request* | | **kube\_apiserver.etcd\_requests\_total**(count) | Etcd request counts for each operation and object type*Shown as request* | | **kube\_apiserver.flowcontrol\_current\_executing\_requests**(gauge) | Number of requests in initial (for a WATCH) or any (for a non-WATCH) execution stage in the API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_current\_executing\_seats**(gauge) | Number of seats (concurrency units) currently occupied by executing requests in the API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_current\_inqueue\_requests**(count) | Number of requests currently pending in queues of the API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_dispatched\_requests\_total**(count) | Number of requests executed by API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_nominal\_limit\_seats**(gauge) | Nominal limit on the number of execution seats available to requests in the API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_rejected\_requests\_total.count**(count) | Number of requests rejected by API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_request\_concurrency\_limit**(gauge) | Shared concurrency limit in the API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_request\_wait\_duration\_seconds.count**(count) | The request wait duration histogram count in the API Priority and Fairness subsystem | | **kube\_apiserver.flowcontrol\_request\_wait\_duration\_seconds.sum**(gauge) | The request wait duration histogram sum in the API Priority and Fairness subsystem*Shown as second* | | **kube\_apiserver.go\_goroutines**(gauge) | The number of goroutines that currently exist | | **kube\_apiserver.go\_threads**(gauge) | The number of OS threads created*Shown as thread* | | **kube\_apiserver.grpc\_client\_handled\_total**(count) | The total number of RPCs completed by the client regardless of success or failure*Shown as request* | | **kube\_apiserver.grpc\_client\_msg\_received\_total**(count) | The total number of gRPC stream messages received by the client*Shown as message* | | **kube\_apiserver.grpc\_client\_msg\_sent\_total**(count) | The total number of gRPC stream messages sent by the client*Shown as message* | | **kube\_apiserver.grpc\_client\_started\_total**(count) | The total number of RPCs started on the client*Shown as request* | | **kube\_apiserver.http\_requests\_total**(gauge) | The accumulated number of HTTP requests made*Shown as request* | | **kube\_apiserver.http\_requests\_total.count**(count) | The monotonic count of the number of HTTP requests made*Shown as request* | | **kube\_apiserver.kubernetes\_feature\_enabled**(gauge) | Whether a Kubernetes feature gate is enabled or not, identified by name and stage (alpha; Kubernetes 1.26+) | | **kube\_apiserver.longrunning\_gauge**(gauge) | The gauge of all active long-running apiserver requests broken out by verb, group, version, resource, scope, and component. Not all requests are tracked this way.*Shown as request* | | **kube\_apiserver.process\_cpu\_total**(count) | Total user and system CPU time spent in seconds.*Shown as second* | | **kube\_apiserver.process\_resident\_memory\_bytes**(gauge) | The resident memory size in bytes*Shown as byte* | | **kube\_apiserver.process\_virtual\_memory\_bytes**(gauge) | The virtual memory size in bytes*Shown as byte* | | **kube\_apiserver.registered\_watchers**(gauge) | The number of currently registered watchers for a given resource*Shown as object* | | **kube\_apiserver.request\_duration\_seconds.count**(count) | The response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope, and component count | | **kube\_apiserver.request\_duration\_seconds.sum**(gauge) | The response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope, and component*Shown as second* | | **kube\_apiserver.request\_latencies.count**(count) | The response latency distribution in microseconds for each verb, resource, and subresource count | | **kube\_apiserver.request\_latencies.sum**(gauge) | The response latency distribution in microseconds for each verb, resource and subresource*Shown as microsecond* | | **kube\_apiserver.requested\_deprecated\_apis**(gauge) | Gauge of deprecated APIs that have been requested, broken out by API group, version, resource, subresource, and removed_release*Shown as request* | | **kube\_apiserver.rest\_client\_request\_latency\_seconds.count**(count) | The request latency in seconds broken down by verb and URL count | | **kube\_apiserver.rest\_client\_request\_latency\_seconds.sum**(gauge) | The request latency in seconds broken down by verb and URL*Shown as second* | | **kube\_apiserver.rest\_client\_requests\_total**(gauge) | The accumulated number of HTTP requests partitioned by status code method and host*Shown as request* | | **kube\_apiserver.rest\_client\_requests\_total.count**(count) | The monotonic count of HTTP requests partitioned by status code method and host*Shown as request* | | **kube\_apiserver.slis.kubernetes\_healthcheck**(gauge) | Result of a single kubernetes apiserver healthcheck (alpha; requires k8s v1.26+) | | **kube\_apiserver.slis.kubernetes\_healthcheck\_total**(count) | The monotonic count of all kubernetes apiserver healthchecks (alpha; requires k8s v1.26+) | | **kube\_apiserver.storage\_list\_evaluated\_objects\_total**(gauge) | The number of objects tested in the course of serving a LIST request from storage (alpha; Kubernetes 1.23+)*Shown as object* | | **kube\_apiserver.storage\_list\_fetched\_objects\_total**(gauge) | The number of objects read from storage in the course of serving a LIST request (alpha; Kubernetes 1.23+)*Shown as object* | | **kube\_apiserver.storage\_list\_returned\_objects\_total**(gauge) | The number of objects returned for a LIST request from storage (alpha; Kubernetes 1.23+)*Shown as object* | | **kube\_apiserver.storage\_list\_total**(gauge) | The number of LIST requests served from storage (alpha; Kubernetes 1.23+)*Shown as object* | | **kube\_apiserver.storage\_objects**(gauge) | The number of stored objects at the time of last check split by kind (Kubernetes 1.21+; replaces etcd_object_counts)*Shown as object* | | **kube\_apiserver.watch\_events\_sizes.count**(count) | The watch event size distribution (Kubernetes 1.16+) | | **kube\_apiserver.watch\_events\_sizes.sum**(gauge) | The watch event size distribution (Kubernetes 1.16+)*Shown as byte* | ### Service Checks{% #service-checks %} Kube_apiserver_metrics does not include any service checks. ### Events{% #events %} Kube_apiserver_metrics does not include any events. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).