Docker Prometheus and OpenMetrics metrics collection
Security Monitoring is now available Security Monitoring is now available

Docker Prometheus and OpenMetrics metrics collection

Collect your exposed Prometheus and OpenMetrics metrics from your application running inside your containers by using the Datadog Agent, and the Datadog-OpenMetrics or Datadog-Prometheus integrations.

Overview

Starting with version 6.5.0, the Agent includes OpenMetrics and Prometheus checks capable of scraping Prometheus endpoints. Datadog recommends using the OpenMetrics check since it is more efficient and fully supports Prometheus text format. For more advanced usage of the OpenMetricsCheck interface, including writing a custom check, see the Developer Tools section. Use the Prometheus check only when the metrics endpoint does not support a text format.

This page explains the basic usage of these checks, enabling you to import all your Prometheus exposed metrics within Datadog.

Setup

Installation

Launch the Docker Agent next to your other containers by replacing <DATADOG_API_KEY> with the API key for your organization in the command below:

DOCKER_CONTENT_TRUST=1 \
docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -e DD_API_KEY="<DATADOG_API_KEY>" \
              datadog/agent:latest
DOCKER_CONTENT_TRUST=1 \
docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro \
                              -v /proc/:/host/proc/:ro \
                              -v /cgroup/:/host/sys/fs/cgroup:ro \
                              -e DD_API_KEY="<DATADOG_API_KEY>" \
                              datadog/agent:latest
DOCKER_CONTENT_TRUST=1 \
docker run -d -e DD_API_KEY="<DATADOG_API_KEY>" \
              datadog/agent:latest

Note: add DD_SITE=datadoghq.eu if you are using Datadog EU site.

Configuration

The Agent detects if it’s running on Docker and automatically searches all container labels for Datadog-OpenMetrics labels. Autodiscovery expects labels to look like these examples, depending on the file type:

LABEL "com.datadoghq.ad.check_names"='["openmetrics"]'
LABEL "com.datadoghq.ad.init_configs"='[{}]'
LABEL "com.datadoghq.ad.instances"='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<NAMESPACE>\",\"metrics\":[{\"<METRIC_TO_FETCH>\": \"<NEW_METRIC_NAME>\"}]}"]'
labels:
    com.datadoghq.ad.check_names: '["openmetrics"]'
    com.datadoghq.ad.init_configs: '[{}]'
    com.datadoghq.ad.instances:  >
    [
      "{\
        "prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<NAMESPACE>\",
        \"metrics\":[{\"<METRIC_TO_FETCH>\": \"<NEW_METRIC_NAME>\"}]
      }"
    ]
-l com.datadoghq.ad.check_names='["openmetrics"]' -l com.datadoghq.ad.init_configs='[{}]' -l com.datadoghq.ad.instances='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<NAMESPACE>\",\"metrics\":[{\"<METRIC_TO_FETCH>\": \"<NEW_METRIC_NAME>\"}]}"]'

With the following configuration placeholder values:

PlaceholderDescription
<PROMETHEUS_PORT>Port to connect to in order to access the Prometheus endpoint.
<PROMETHEUS_ENDPOINT>URL for the metrics served by the container, in Prometheus format.
<NAMESPACE>Set namespace to be prefixed to every metric when viewed in Datadog.
<METRIC_TO_FETCH>Prometheus metrics key to be fetched from the Prometheus endpoint.
<NEW_METRIC_NAME>Optional parameter which, if set, transforms the <METRIC_TO_FETCH> metric key to <NEW_METRIC_NAME> in Datadog. If you choose not to use this option, pass a list of strings rather than key:value pairs.

Note: See the sample openmetrics.d/conf.yaml for all available configuration options.

Getting started

Simple metric collection

To get started with collecting metrics exposed by Prometheus running within a container, follow these steps:

  1. Launch the Datadog Agent:

    DOCKER_CONTENT_TRUST=1 \
    docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
                  -v /proc/:/host/proc/:ro \
                  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
                  -e DD_API_KEY="<DATADOG_API_KEY>" \
                  datadog/agent:latest
    DOCKER_CONTENT_TRUST=1 \
    docker run -d -e DD_API_KEY="<DATADOG_API_KEY>" \
                  datadog/agent:latest \
                  -v \\.\pipe\docker_engine:\\.\pipe\docker_engine

  2. To launch Prometheus in a container, run: docker run -p 9090:9090 prom/prometheus. In order to notify the Agent to query this container with the OpenMetrics check, add the following configuration:

    -l com.datadoghq.ad.check_names='["openmetrics"]' -l com.datadoghq.ad.init_configs='[{}]' -l com.datadoghq.ad.instances='[  {"prometheus_url":"http://%%host%%:%%port%%/metrics","namespace":"documentation_example_docker","metrics":[ {"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]}]'

    To launch the Prometheus container with the right annotations for Autodiscovery to work, run:

    docker run -p 9090:9090 -l com.datadoghq.ad.check_names='["openmetrics"]' -l com.datadoghq.ad.init_configs='[{}]' -l com. datadoghq.ad.instances='[{"prometheus_url":"http://%%host%%:%%port%%/metrics","namespace":"documentation_example_docker",  "metrics":[{"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]}]' prom/prometheus
  3. Go into your Metric summary page to see the collected metrics: prometheus_target_interval_length_seconds*

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Further Reading