Kubernetes Prometheus and OpenMetrics metrics collection

Collect your exposed Prometheus and OpenMetrics metrics from your application running inside Kubernetes by using the Datadog Agent, and the Datadog-OpenMetrics or Datadog-Prometheus integrations.

Overview

Starting with version 6.5.0, the Agent includes OpenMetrics and Prometheus checks capable of scraping Prometheus endpoints. Datadog recommends using the OpenMetrics check since it is more efficient and fully supports Prometheus text format. For more advanced usage of the OpenMetricsCheck interface, including writing a custom check, see the Developer Tools section. Use the Prometheus check only when the metrics endpoint does not support a text format.

This page explains the basic usage of these checks, which enable you to scrape custom metrics from Prometheus endpoints.

Setup

Installation

Deploy the Datadog Agent in your Kubernetes cluster. OpenMetrics and Prometheus checks are included in the Datadog Agent package, so you don’t need to install anything else on your containers or hosts.

Configuration

Configure your OpenMetrics or Prometheus check using Autodiscovery, by applying the following annotations to your OpenMetrics/Prometheus pod:

# (...)
metadata:
    #(...)
    annotations:
        ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: |
            ["openmetrics"]
        ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: |
            [{}]
        ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: |
            [
              {
                "prometheus_url": "http://%%host%%:%%port%%/<PROMETHEUS_ENDPOINT> ",
                "namespace": "<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>",
                "metrics": [{"<METRIC_TO_FETCH>":"<NEW_METRIC_NAME>"}]
              }
            ]
spec:
    containers:
        - name: '<CONTAINER_IDENTIFIER>'

With the following configuration placeholder values:

Note: See the sample openmetrics.d/conf.yaml for all available configuration options.

Getting Started

Simple metric collection

1. Launch the Datadog Agent.

2. Use this Prometheus DaemonSet prometheus.yaml to launch a Prometheus pod with already the Autodiscovery configuration in it:

   Autodiscovery configuration:

    # (...)
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: prometheus
         purpose: example
     template:
       metadata:
         labels:
           app: prometheus
           purpose: example
         annotations:
             ad.datadoghq.com/prometheus-example.check_names: |
               ["openmetrics"]
             ad.datadoghq.com/prometheus-example.init_configs: |
               [{}]
             ad.datadoghq.com/prometheus-example.instances: |
               [
                 {
                   "prometheus_url": "http://%%host%%:%%port%%/metrics",
                   "namespace": "documentation_example_kubernetes",
                   "metrics": [ {"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]
                 }
               ]
     # (...)

   Command to create the Prometheus pod:

   kubectl create -f prometheus.yaml

3. Go into your Metric summary page to see the collected metrics: prometheus_target_interval_length_seconds*

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Further Reading

Additional helpful documentation, links, and articles:

Collect your logsDOCUMENTATION Collect your processesDOCUMENTATION Collect your tracesDOCUMENTATION Write your own custom Prometheus CheckDOCUMENTATION