--- title: Kubernetes Prometheus and OpenMetrics metrics collection description: >- Collect Prometheus and OpenMetrics from Kubernetes workloads using the Datadog Agent with Autodiscovery breadcrumbs: >- Docs > Containers > Kubernetes > Kubernetes Prometheus and OpenMetrics metrics collection --- # Kubernetes Prometheus and OpenMetrics metrics collection ## Overview{% #overview %} Collect your exposed Prometheus and OpenMetrics metrics from your application running inside Kubernetes by using the Datadog Agent and the [OpenMetrics](https://docs.datadoghq.com/integrations/openmetrics/) or [Prometheus](https://docs.datadoghq.com/integrations/prometheus/) integrations. By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. Starting with version 6.5.0, the Agent includes [OpenMetrics](https://github.com/DataDog/integrations-core/tree/master/openmetrics) and [Prometheus](https://github.com/DataDog/integrations-core/tree/master/prometheus) checks capable of scraping Prometheus endpoints. For more advanced usage of the `OpenMetricsCheck` interface, including writing a custom check, see the [Developer Tools](https://docs.datadoghq.com/extend/custom_checks/prometheus/) section. This page explains the basic usage of these checks, which enable you to scrape custom metrics from Prometheus endpoints. For an explanation of how Prometheus and OpenMetrics metrics map to Datadog metrics, see the [Mapping Prometheus Metrics to Datadog Metrics](https://docs.datadoghq.com/integrations/guide/prometheus-metrics) guide. **Note**: Datadog recommends using the OpenMetrics check since it is more efficient and fully supports the Prometheus text format. Use the Prometheus check only when the metrics endpoint does not support a text format. ## Setup{% #setup %} ### Installation{% #installation %} [Deploy the Datadog Agent in your Kubernetes cluster](https://docs.datadoghq.com/agent/kubernetes/#installation). OpenMetrics and Prometheus checks are included in the [Datadog Agent](https://docs.datadoghq.com/getting_started/tagging/) package, so you don't need to install anything else on your containers or hosts. ### Configuration{% #configuration %} Configure your OpenMetrics or Prometheus check using Autodiscovery, by applying the following `annotations` to your **pod** exposing the OpenMetrics/Prometheus metrics: {% tab title="Kubernetes (AD v2)" %} **Note:** AD Annotations v2 was introduced in Datadog Agent version 7.36 to simplify integration configuration. For previous versions of the Datadog Agent, use AD Annotations v1. ```yaml # (...) metadata: #(...) annotations: ad.datadoghq.com/.checks: | { "openmetrics": { "init_config": {}, "instances": [ { "openmetrics_endpoint": "http://%%host%%:%%port%%/ ", "namespace": "", "metrics": [{"":""}] } ] } } spec: containers: - name: '' ``` {% /tab %} {% tab title="Kubernetes (AD v1)" %} ```yaml # (...) metadata: #(...) annotations: ad.datadoghq.com/.check_names: | ["openmetrics"] ad.datadoghq.com/.init_configs: | [{}] ad.datadoghq.com/.instances: | [ { "openmetrics_endpoint": "http://%%host%%:%%port%%/ ", "namespace": "", "metrics": [{"":""}] } ] spec: containers: - name: '' ``` {% /tab %} With the following configuration placeholder values: | Placeholder | Description | | ---------------------------------------- | -------------------------------------------------------------------------------- | | `` | Matches the name of the container that exposes the metrics. | | `` | URL path for the metrics served by the container, in Prometheus format. | | `` | Set namespace to be prefixed to every metric when viewed in Datadog. | | `` | Prometheus metrics key to be fetched from the Prometheus endpoint. | | `` | Transforms the `` metric key to `` in Datadog. | The `metrics` configuration is a list of metrics to retrieve as custom metrics. Include each metric to fetch and the desired metric name in Datadog as key value pairs, for example, `{"":""}`. To prevent excess custom metrics charges, Datadog recommends limiting the scope to only include metrics that you need. You can alternatively provide a list of metric names strings, interpreted as regular expressions, to bring the desired metrics with their current names. If you want **all** metrics, then use `".*"` rather than `"*"`. **Note:** Regular expressions can potentially send a lot of custom metrics. For a full list of available parameters for instances, including `namespace` and `metrics`, see the [sample configuration openmetrics.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example). **Note**: The check limits itself to 2000 metric by default. Specify the optional `max_returned_metrics` parameter to modify this limit. ## Getting started{% #getting-started %} ### Simple metric collection (OpenMetrics Check){% #simple-metric-collection-openmetrics-check %} 1. [Launch the Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest?platform=kubernetes). 1. Use the [Prometheus `prometheus.yaml`](https://docs.datadoghq.com/resources/yaml/prometheus.yaml) to launch an example Prometheus Deployment with the Autodiscovery configuration on the pod: {% tab title="Kubernetes (AD v2)" %} **Note:** AD Annotations v2 was introduced in Datadog Agent version 7.36 to simplify integration configuration. For previous versions of the Datadog Agent, use AD Annotations v1. ```yaml # (...) spec: template: metadata: annotations: ad.datadoghq.com/prometheus-example.checks: | { "openmetrics": { "instances": [ { "openmetrics_endpoint": "http://%%host%%:%%port%%/metrics", "namespace": "documentation_example_kubernetes", "metrics": [ {"promhttp_metric_handler_requests": "handler.requests"}, {"promhttp_metric_handler_requests_in_flight": "handler.requests.in_flight"}, "go_memory.*" ] } ] } } spec: containers: - name: prometheus-example # (...) ``` {% /tab %} {% tab title="Kubernetes (AD v1)" %} ```yaml # (...) spec: template: metadata: annotations: ad.datadoghq.com/prometheus-example.check_names: | ["openmetrics"] ad.datadoghq.com/prometheus-example.init_configs: | [{}] ad.datadoghq.com/prometheus-example.instances: | [ { "openmetrics_endpoint": "http://%%host%%:%%port%%/metrics", "namespace": "documentation_example_kubernetes", "metrics": [ {"promhttp_metric_handler_requests": "handler.requests"}, {"promhttp_metric_handler_requests_in_flight": "handler.requests.in_flight"}, "go_memory.*" ] } ] spec: containers: - name: prometheus-example # (...) ``` {% /tab %} Command to create the Prometheus Deployment: ```shell kubectl create -f prometheus.yaml ``` 1. Go into your [Fleet Automation](https://app.datadoghq.com/fleet?query=integration:openmetrics) page and filter for the `openmetrics` integration to view detailed information about the status of your checks. 1. Go into your [Metric summary](https://app.datadoghq.com/metric/summary) page to see the metrics collected from this example pod. This configuration will collect the metric `promhttp_metric_handler_requests`, `promhttp_metric_handler_requests_in_flight`, and all exposed metrics starting with `go_memory`. {% image source="https://datadog-docs.imgix.net/images/integrations/guide/prometheus_kubernetes/openmetrics_v2_collected_metric_kubernetes.d8faea61a524dec02d405afdb7e95fb3.png?auto=format" alt="Prometheus metric collected kubernetes" /%} ## Metric collection with Prometheus annotations (Prometheus Check){% #metric-collection-with-prometheus-annotations-prometheus-check %} With Prometheus Autodiscovery, the Datadog Agent is able to detect native Prometheus annotations (for example: `prometheus.io/scrape`, `prometheus.io/path`, `prometheus.io/port`) and schedule OpenMetrics checks automatically to collect Prometheus metrics in Kubernetes. **Note**: Datadog recommends using the OpenMetrics check since it is more efficient and fully supports the Prometheus text format. Use the Prometheus check only when the metrics endpoint does not support a text format. ### Requirements{% #requirements %} - Datadog Agent v7.27+ or v6.27+ (for Pod checks) - Datadog Cluster Agent v1.11+ (for service and endpoint checks) ### Configuration{% #configuration-1 %} It's recommended to first check which pods and services have the `prometheus.io/scrape=true` annotation before enabling this feature. This can be done with the following commands: ```shell kubectl get pods -o=jsonpath='{.items[?(@.metadata.annotations.prometheus\.io/scrape=="true")].metadata.name}' --all-namespaces kubectl get services -o=jsonpath='{.items[?(@.metadata.annotations.prometheus\.io/scrape=="true")].metadata.name}' --all-namespaces ``` Once the Prometheus scrape feature is enabled, the Datadog Agent collects custom metrics from these resources. If you do not want to collect the custom metrics from these resources, you can remove this annotation or update the Autodiscovery rules as described in the advanced configuration section. **Note**: Enabling this feature without advanced configuration can cause a significant increase in custom metrics, which can lead to billing implications. See the advanced configuration section to learn how to only collect metrics from a subset of containers/pods/services. #### Basic configuration{% #basic-configuration %} {% tab title="Datadog Operator" %} Update your Datadog Operator configuration to contain the following: In the `datadog-agent.yaml` file: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: global: credentials: apiKey: features: prometheusScrape: enabled: true enableServiceEndpoints: true ``` After making your changes, apply the new configuration by using the following command: ```shell kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml ``` {% /tab %} {% tab title="Helm" %} Update your Helm configuration to contain the following: In the `datadog-values.yaml` file: ```yaml datadog: # (...) prometheusScrape: enabled: true serviceEndpoints: true # (...) ``` After making your changes, upgrade your Datadog Helm chart using the following command: ```shell helm upgrade -f datadog-values.yaml datadog/datadog ``` {% /tab %} {% tab title="Manual (DaemonSet)" %} In your DaemonSet manifest for the Agent `daemonset.yaml`, add the following environment variables for the Agent container: ```yaml - name: DD_PROMETHEUS_SCRAPE_ENABLED value: "true" - name: DD_PROMETHEUS_SCRAPE_VERSION value: "2" ``` If the Cluster Agent is enabled, inside its manifest `cluster-agent-deployment.yaml`, add the following environment variables for the Cluster Agent container: ```yaml - name: DD_PROMETHEUS_SCRAPE_ENABLED value: "true" - name: DD_PROMETHEUS_SCRAPE_SERVICE_ENDPOINTS value: "true" ``` {% /tab %} This instructs the Datadog Agent to detect the pods that have native Prometheus annotations and generate corresponding OpenMetrics checks. It also instructs the Datadog Cluster Agent (if enabled) to detect the services that have native Prometheus annotations and generate corresponding OpenMetrics checks. - `prometheus.io/scrape=true`: Required. - `prometheus.io/path`: Optional, defaults to `/metrics`. - `prometheus.io/port`: Optional, default is `%%port%%`, a [template variable](https://docs.datadoghq.com/agent/faq/template_variables/) that is replaced by the container/service port. This configuration generates a check that collects all metrics exposed using the default configuration of the [OpenMetrics integration](https://docs.datadoghq.com/integrations/openmetrics/). #### Advanced configuration{% #advanced-configuration %} You can further configure metric collection (beyond native Prometheus annotations) with the `additionalConfigs` field. ##### Additional OpenMetrics check configurations{% #additional-openmetrics-check-configurations %} Use `additionalConfigs.configurations` to define additional OpenMetrics check configurations. See the [list of supported OpenMetrics parameters](https://github.com/DataDog/datadog-agent/blob/main/comp/core/autodiscovery/common/types/prometheus.go#L57-L123) that you can pass in `additionalConfigs`. ##### Custom Autodiscovery rules{% #custom-autodiscovery-rules %} Use `additionalConfigs.autodiscovery` to define custom Autodiscovery rules. These rules can be based on container names, Kubernetes annotations, or both. {% dl %} {% dt %} `additionalConfigs.autodiscovery.kubernetes_container_names` {% /dt %} {% dd %} A list of container names to target, in regular expression format. {% /dd %} {% dt %} `additionalConfigs.autodiscovery.kubernetes_annotations` {% /dt %} {% dd %} Two maps (`include` and `exclude`) of annotations to define discovery rules. Default: ```yaml include: prometheus.io/scrape: "true" exclude: prometheus.io/scrape: "false" ``` {% /dd %} {% /dl %} If both `kubernetes_container_names` and `kubernetes_annotations` are defined, **AND** logic is used (both rules must match). ##### Examples{% #examples %} The following configuration targets a container named `my-app` running in a pod with the annotation `app=my-app`. The OpenMetrics check configuration is customized to enable the `send_distribution_buckets` option and define a custom timeout of 5 seconds. {% tab title="Datadog Operator" %} Update your Datadog Operator configuration to contain the following: In the `datadog-agent.yaml` file: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: global: credentials: apiKey: features: prometheusScrape: enabled: true enableServiceEndpoints: true additionalConfigs: |- - autodiscovery: kubernetes_container_names: - my-app kubernetes_annotations: include: app: my-app configurations: - timeout: 5 send_distribution_buckets: true ``` {% /tab %} {% tab title="Helm" %} In the `datadog-values.yaml` file: ```yaml datadog: # (...) prometheusScrape: enabled: true serviceEndpoints: true additionalConfigs: - autodiscovery: kubernetes_container_names: - my-app kubernetes_annotations: include: app: my-app configurations: - timeout: 5 send_distribution_buckets: true ``` {% /tab %} {% tab title="Manual (DaemonSet)" %} For DaemonSet, advanced configuration is defined in the `DD_PROMETHEUS_SCRAPE_CHECKS` environment variable, not an `additionalConfigs` field. ```yaml - name: DD_PROMETHEUS_SCRAPE_ENABLED value: "true" - name: DD_PROMETHEUS_SCRAPE_CHECKS value: "[{\"autodiscovery\":{\"kubernetes_annotations\":{\"include\":{\"app\":\"my-app\"}},\"kubernetes_container_names\":[\"my-app\"]},\"configurations\":[{\"send_distribution_buckets\":true,\"timeout\":5}]}]" - name: DD_PROMETHEUS_SCRAPE_VERSION value: "2" ``` {% /tab %} ## From custom to official integration{% #from-custom-to-official-integration %} By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don't hesitate to [contribute](https://docs.datadoghq.com/extend/custom_checks/prometheus/)! Official integrations have their own dedicated directories. There's a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the [kube-proxy](https://github.com/DataDog/integrations-core/tree/master/kube_proxy) integration. ## Further Reading{% #further-reading %} - [Monitor your Kubernetes operators to keep applications running smoothly](https://www.datadoghq.com/blog/kubernetes-operator-performance) - [Collect your application logs](https://docs.datadoghq.com/agent/kubernetes/log/) - [Collect your application traces](https://docs.datadoghq.com/agent/kubernetes/apm/) - [Automatically collect your applications' metrics and logs](https://docs.datadoghq.com/agent/kubernetes/integrations/) - [Limit data collection to a subset of containers only](https://docs.datadoghq.com/agent/guide/autodiscovery-management/) - [Assign tags to all data emitted by a container](https://docs.datadoghq.com/agent/kubernetes/tag/) - [Mapping Prometheus Metrics to Datadog Metrics](https://docs.datadoghq.com/integrations/guide/prometheus-metrics/)