Kubernetes Prometheus and OpenMetrics metrics collection

Collect your exposed Prometheus and OpenMetrics metrics from your application running inside Kubernetes by using the Datadog Agent, and the Datadog-OpenMetrics or Datadog-Prometheus integrations.

Overview

Starting with version 6.5.0, the Agent includes OpenMetrics and Prometheus checks capable of scraping Prometheus endpoints. Datadog recommends using the OpenMetrics check since it is more efficient and fully supports Prometheus text format. For more advanced usage of the OpenMetricsCheck interface, including writing a custom check, see the Developer Tools section. Use the Prometheus check only when the metrics endpoint does not support a text format.

This page explains the basic usage of these checks, which enable you to scrape custom metrics from Prometheus endpoints.

For an explanation of how Prometheus and OpenMetrics metrics map to Datadog metrics, see the Mapping Prometheus Metrics to Datadog Metrics guide.

Setup

Installation

Deploy the Datadog Agent in your Kubernetes cluster. OpenMetrics and Prometheus checks are included in the Datadog Agent package, so you don’t need to install anything else on your containers or hosts.

Configuration

Configure your OpenMetrics or Prometheus check using Autodiscovery, by applying the following annotations to your OpenMetrics/Prometheus pod:

# (...)
metadata:
    #(...)
    annotations:
        ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: |
            ["openmetrics"]
        ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: |
            [{}]
        ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: |
            [
              {
                "prometheus_url": "http://%%host%%:%%port%%/<PROMETHEUS_ENDPOINT> ",
                "namespace": "<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>",
                "metrics": [{"<METRIC_TO_FETCH>":"<NEW_METRIC_NAME>"}]
              }
            ]
spec:
    containers:
        - name: '<CONTAINER_IDENTIFIER>'

With the following configuration placeholder values:

Placeholder	Description
`<PROMETHEUS_PORT>`	Port to connect to in order to access the Prometheus endpoint.
`<PROMETHEUS_ENDPOINT>`	URL for the metrics served by the container, in Prometheus format.
`<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>`	Set namespace to be prefixed to every metric when viewed in Datadog.
`<METRIC_TO_FETCH>`	Prometheus metrics key to be fetched from the Prometheus endpoint.
`<NEW_METRIC_NAME>`	Optional parameter which, if set, transforms the `<METRIC_TO_FETCH>` metric key to `<NEW_METRIC_NAME>` in Datadog. If you choose not to use this option, pass a list of strings rather than `key:value` pairs.

For a full list of available parameters for instances, including namespace and metrics, see the table in the Prometheus host collection guide.

Getting Started

Simple metric collection

Launch the Datadog Agent.

Use this Prometheus DaemonSet prometheus.yaml to launch a Prometheus pod with already the Autodiscovery configuration in it:

Autodiscovery configuration:

 # (...)
spec:
  replicas: 2
  selector:
    matchLabels:
      app: prometheus
      purpose: example
  template:
    metadata:
      labels:
        app: prometheus
        purpose: example
      annotations:
          ad.datadoghq.com/prometheus-example.check_names: |
            ["openmetrics"]
          ad.datadoghq.com/prometheus-example.init_configs: |
            [{}]
          ad.datadoghq.com/prometheus-example.instances: |
            [
              {
                "prometheus_url": "http://%%host%%:%%port%%/metrics",
                "namespace": "documentation_example_kubernetes",
                "metrics": [ {"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]
              }
            ]
  # (...)

Command to create the Prometheus pod:

kubectl create -f prometheus.yaml

Go into your Metric summary page to see the collected metrics: prometheus_target_interval_length_seconds*

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Datadog Docs