Docker Prometheus and OpenMetrics metrics collection

Collect your exposed Prometheus and OpenMetrics metrics from your application running inside your containers by using the Datadog Agent, and the Datadog-OpenMetrics or Datadog-Prometheus integrations.

Overview

Starting with version 6.5.0, the Agent includes OpenMetrics and Prometheus checks capable of scraping Prometheus endpoints. Datadog recommends using the OpenMetrics check since it is more efficient and fully supports Prometheus text format. For more advanced usage of the OpenMetricsCheck interface, including writing a custom check, see the Developer Tools section. Use the Prometheus check only when the metrics endpoint does not support a text format.

This page explains the basic usage of these checks, enabling you to import all your Prometheus exposed metrics within Datadog.

Setup

Installation

Launch the Docker Agent next to your other containers by replacing <DATADOG_API_KEY> with the API key for your organization in the command below:

DOCKER_CONTENT_TRUST=1 \
docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -e DD_API_KEY="<DATADOG_API_KEY>" \
              datadog/agent:latest

DOCKER_CONTENT_TRUST=1 \
docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro \
                              -v /proc/:/host/proc/:ro \
                              -v /cgroup/:/host/sys/fs/cgroup:ro \
                              -e DD_API_KEY="<DATADOG_API_KEY>" \
                              datadog/agent:latest

DOCKER_CONTENT_TRUST=1 \
docker run -d -e DD_API_KEY="<DATADOG_API_KEY>" \
              datadog/agent:latest

Note: add DD_SITE=datadoghq.eu if you are using Datadog EU site.

Configuration

The Agent detects if it’s running on Docker and automatically searches all container labels for Datadog-OpenMetrics labels. Autodiscovery expects labels to look like these examples, depending on the file type:

LABEL "com.datadoghq.ad.check_names"='["openmetrics"]'
LABEL "com.datadoghq.ad.init_configs"='[{}]'
LABEL "com.datadoghq.ad.instances"='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<NAMESPACE>\",\"metrics\":[{\"<METRIC_TO_FETCH>\": \"<NEW_METRIC_NAME>\"}]}"]'

labels:
    com.datadoghq.ad.check_names: '["openmetrics"]'
    com.datadoghq.ad.init_configs: '[{}]'
    com.datadoghq.ad.instances:  >
    [
      "{\
        "prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<NAMESPACE>\",
        \"metrics\":[{\"<METRIC_TO_FETCH>\": \"<NEW_METRIC_NAME>\"}]
      }"
    ]

-l com.datadoghq.ad.check_names='["openmetrics"]' -l com.datadoghq.ad.init_configs='[{}]' -l com.datadoghq.ad.instances='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<NAMESPACE>\",\"metrics\":[{\"<METRIC_TO_FETCH>\": \"<NEW_METRIC_NAME>\"}]}"]'

With the following configuration placeholder values:

Note: See the sample openmetrics.d/conf.yaml for all available configuration options.

Getting started

Simple metric collection

To get started with collecting metrics exposed by Prometheus running within a container, follow these steps:

1. Launch the Datadog Agent:

   • Dropdown

   DOCKER_CONTENT_TRUST=1 \
   docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
                 -v /proc/:/host/proc/:ro \
                 -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
                 -e DD_API_KEY="<DATADOG_API_KEY>" \
                 datadog/agent:latest

   DOCKER_CONTENT_TRUST=1 \
   docker run -d -e DD_API_KEY="<DATADOG_API_KEY>" \
                 datadog/agent:latest \
                 -v \\.\pipe\docker_engine:\\.\pipe\docker_engine

2. To launch Prometheus in a container, run: docker run -p 9090:9090 prom/prometheus. In order to notify the Agent to query this container with the OpenMetrics check, add the following configuration:

   -l com.datadoghq.ad.check_names='["openmetrics"]' -l com.datadoghq.ad.init_configs='[{}]' -l com.datadoghq.ad.instances='[  {"prometheus_url":"http://%%host%%:%%port%%/metrics","namespace":"documentation_example_docker","metrics":[ {"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]}]'

   To launch the Prometheus container with the right annotations for Autodiscovery to work, run:

   docker run -p 9090:9090 -l com.datadoghq.ad.check_names='["openmetrics"]' -l com.datadoghq.ad.init_configs='[{}]' -l com. datadoghq.ad.instances='[{"prometheus_url":"http://%%host%%:%%port%%/metrics","namespace":"documentation_example_docker",  "metrics":[{"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]}]' prom/prometheus

3. Go into your Metric summary page to see the collected metrics: prometheus_target_interval_length_seconds*

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Further Reading

お役に立つドキュメント、リンクや記事:

Collect your application logsDOCUMENTATION Collect your application tracesDOCUMENTATION Collect automatically your applications metrics and logsDOCUMENTATION Limit data collection to a subset of containers onlyDOCUMENTATION Assign tags to all data emitted by a containerDOCUMENTATION