Network Performance Monitoring is now generally available! Network Monitoring is now available!

Kubernetes API server metrics

Agent Check Agent Check

Supported OS: Linux

Overview

This check monitors Kube_apiserver_metrics.

Setup

Installation

The Kube_apiserver_metrics check is included in the Datadog Agent package, so you do not need to install anything else on your server.

Configuration

The main use case to run the kube_apiserver_metrics check is as a Cluster Level Check. Refer to the dedicated documentation for Cluster Level Checks. You can annotate the service of your apiserver with the following:

Annotations:       ad.datadoghq.com/endpoint.check_names: ["kube_apiserver_metrics"]
                   ad.datadoghq.com/endpoint.init_configs: [{}]
                   ad.datadoghq.com/endpoint.instances:
                     [
                       {
                         "prometheus_url": "%%host%%:%%port%%/metrics",
                         "bearer_token_auth": "true"
                       }
                     ]
                   ad.datadoghq.com/service.check_names: [""]
                   ad.datadoghq.com/service.init_configs: [{}]
                   ad.datadoghq.com/service.instances: [{}]

Then the Datadog Cluster Agent schedules the check(s) for each endpoint onto Datadog Agent(s).

Disclaimer: Your apiserver(s) need to run as pods, other methods (e.g. systemd unit) are not supported at the moment.

You can also run the check by configuring the endpoints directly in the kube_apiserver_metrics.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory. See the sample kube_apiserver_metrics.d/conf.yaml for all available configuration options.

By default the Agent running the check tries to get the service account bearer token to authenticate against the APIServer. If you are not using RBACs, set bearer_token_auth to false.

Finally, if you run the Datadog Agent on the master nodes, you can rely on Autodiscovery to schedule the check. It is automatic if you are running the official image k8s.gcr.io/kube-apiserver.

Validation

Run the Agent’s status subcommand and look for kube_apiserver_metrics under the Checks section.

Data Collected

Metrics

kube_apiserver.longrunning_gauge
(gauge)
Gauge of all active long-running apiserver requests broken out by verb API resource and scope. Not all requests are tracked this way.
Shown as request
kube_apiserver.current_inflight_requests
(gauge)
Maximal number of currently used inflight request limit of this apiserver per request kind in last second.
kube_apiserver.audit_event
(gauge)
Accumulated number audit events generated and sent to the audit backend
Shown as event
kube_apiserver.go_threads
(gauge)
Number of OS threads created
Shown as thread
kube_apiserver.go_goroutines
(gauge)
Number of goroutines that currently exist
kube_apiserver.APIServiceRegistrationController_depth
(gauge)
Current depth of workqueue: APIServiceRegistrationController
kube_apiserver.etcd_object_counts
(gauge)
Number of stored objects at the time of last check split by kind
Shown as object
kube_apiserver.rest_client_requests_total
(gauge)
Accumulated number of HTTP requests partitioned by status code method and host
Shown as request
kube_apiserver.apiserver_request_count
(gauge)
Accumulated number of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (deprecated since kubernetes 1.15)
Shown as request
kube_apiserver.apiserver_dropped_requests_total
(gauge)
Accumulated number of requests dropped with 'Try again later' response
Shown as request
kube_apiserver.http_requests_total
(gauge)
Accumulated number of HTTP requests made
Shown as request
kube_apiserver.authenticated_user_requests
(gauge)
Accumulated number of authenticated requests broken out by username
Shown as request
kube_apiserver.audit_event.count
(count)
Monotonic count of audit events generated and sent to the audit backend
Shown as event
kube_apiserver.rest_client_requests_total.count
(count)
Monotonic count of HTTP requests partitioned by status code method and host
Shown as request
kube_apiserver.apiserver_request_count.count
(count)
Monotonic count of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (deprecated since kubernetes 1.15)
Shown as request
kube_apiserver.apiserver_dropped_requests_total.count
(count)
Monotonic count of requests dropped with 'Try again later' response
Shown as request
kube_apiserver.http_requests_total.count
(count)
Monotonic count of the number of HTTP requests made
Shown as request
kube_apiserver.authenticated_user_requests.count
(count)
Monotonic count of authenticated requests broken out by username
Shown as request
kube_apiserver.apiserver_request_total
(gauge)
Accumulated number of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (Only present from kubernetes 1.15 to replace apiserver_request_count)
Shown as request
kube_apiserver.apiserver_request_total.count
(count)
Monotonic count of apiserver requests broken out for each verb API resource client and HTTP response contentType and code (Only present from kubernetes 1.15 to replace apiserver_request_count.count)
Shown as request

Service Checks

Kube_apiserver_metrics does not include any service checks.

Events

Kube_apiserver_metrics does not include any events.

Troubleshooting

Need help? Contact Datadog support.


Mistake in the docs? Feel free to contribute!