Nouvelles annonces sur les technologies sans serveur et réseau ainsi que sur le RUM (Real-User Monitoring) dévoilées à la conférence Dash ! Nouvelles annonces dévoilées à la conférence Dash !

Prometheus and OpenMetrics metrics collection

Cette page n'est pas encore disponible en français, sa traduction est en cours.
Si vous avez des questions ou des retours sur notre projet de traduction actuel, n'hésitez pas à nous contacter.

Collect your exposed Prometheus and OpenMetrics metrics from your application running inside containers or directly on your host by using the Datadog Agent, and the Datadog-OpenMetrics or Datadog-Prometheus integrations.

Overview

Starting with version 6.5.0, the Agent includes OpenMetrics and Prometheus checks capable of scraping Prometheus endpoints with a few lines of configuration. This page explains the basic usage of these checks, which enable you to scrape custom metrics from Prometheus endpoints.

Datadog recommends using the OpenMetrics check since it is more efficient and fully supports Prometheus text format. Use the Prometheus check only when the metrics endpoint does not support a text format.

For more advanced usage of the OpenMetricsCheck interface, including writing a custom check, see the Developer Tools section.

Setup

Installation

Install the Datadog Agent for your corresponding operating system. OpenMetrics and Prometheus checks are included in the Datadog Agent package, so you don’t need to install anything else on your containers or hosts.

Configuration

If you are running the Agent as a binary on a host, configure your OpenMetrics or Prometheus check as you would any other Agent integration. If you are running the Agent as a DaemonSet in Kubernetes, configure your OpenMetrics or Prometheus check using Autodiscovery.

First, edit the openmetrics.d/conf.yaml or prometheus.d/conf.yaml file in the conf.d/ folder at the root of your Agent’s configuration directory to configure your Datadog-OpenMetrics or Datadog-Prometheus integrations. Then restart the Agent to start collecting your metrics. See the sample openmetrics.d/conf.yaml or sample prometheus.d/conf.yaml for all available configuration options. This is the minimum required configuration needed to enable the integration:

init_config:

instances:
  - prometheus_url: 'localhost:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT>'
    namespace: '<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>'
    metrics:
      - '<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>'

The Agent detects if it’s running on Docker and automatically searches all containers labels for Datadog-OpenMetrics labels. Autodiscovery expects labels to look like these examples, depending on the file type:

Dockerfile

LABEL "com.datadoghq.ad.check_names"='["openmetrics"]'
LABEL "com.datadoghq.ad.init_configs"='["{}"]'
LABEL "com.datadoghq.ad.instances"='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

docker-compose.yaml

labels:
  com.datadoghq.ad.check_names: '["openmetrics"]'
  com.datadoghq.ad.init_configs: '["{}"]'
  com.datadoghq.ad.instances: '["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

docker run command

-l com.datadoghq.ad.check_names='["openmetrics"]' -l com.datadoghq.ad.init_configs='["{}"]' -l com.datadoghq.ad.instances='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

Like all Datadog Agent integrations, the Datadog-OpenMetrics integration can be configured using Autodiscovery. This allows you to send a given container/pod exposed OpenMetrics or Prometheus metrics to Datadog by applying the following annotations to it:

# (...)
metadata:
#(...)
  annotations:
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: |
        ["openmetrics"]
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: |
        [{}]
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: |
        [
          {
            "prometheus_url": "http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> ",
            "namespace": "<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>",
            "metrics": [{"<METRIC_TO_FETCH>":"<NEW_METRIC_NAME>"}]
          }
        ]
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'

With the following configuration placeholder values:

  • <PROMETHEUS_PORT>: Port to connect to in order to access the Prometheus endpoint.
  • <PROMETHEUS_ENDPOINT>: URL for the metrics served by the container, in Prometheus format.
  • <METRICS_NAMESPACE_PREFIX_FOR_DATADOG>: Set namespace to be prefixed to every metric when viewed in Datadog.
  • <METRIC_TO_FETCH>: Prometheus metrics key to be fetched from the Prometheus endpoint.
  • <NEW_METRIC_NAME>: Optional parameter which, if set, transforms the <METRIC_TO_FETCH> metric key to <NEW_METRIC_NAME> in Datadog.

Find below the full list of parameters that can be used for your instances:

NameTypeNecessityDefault value Description
prometheus_urlstringrequirednoneThe URL where your application metrics are exposed by Prometheus/OpenMetrics.
namespacestringrequirednoneThe namespace to be appended before all metrics namespaces. Your metrics are collected in the form namespace.metric_name.
metricslist of key:value elementsrequirednoneList of <METRIC_TO_FETCH>: <NEW_METRIC_NAME> pairs for metrics to be fetched from the Prometheus endpoint.
<NEW_METRIC_NAME> is optional. It transforms the name in Datadog if set.
This list should contain at least one metric.
prometheus_metrics_prefixstringoptionalnonePrefix for exposed Prometheus/OpenMetrics metrics.
health_service_checkbooleanoptionaltrueSend a service check reporting on the health of the Prometheus endpoint.
The check is named <NAMESPACE>.prometheus.health.
label_to_hostnamestringoptionalnoneOverride the hostname with the value of one label.
label_joinsobjectoptionalnoneThe label join allows you to target a metric and retrieve its label via a 1:1 mapping.
labels_mapperlist of key:value elementoptionalnoneThe label mapper allows you to rename some labels.
Format: <LABEL_TO_RENAME>: <NEW_LABEL_NAME>.
type_overrideslist of key:value elementoptionalnoneType override allows you to override a type in the Prometheus payload
or type an untyped metric (they’re ignored by default).
Supported <METRIC_TYPE>s are gauge, counter, histogram, and summary.
tagslist of key:value elementoptionalnoneList of tags to attach to every metric, event, and service check emitted by this integration.

Learn more about tagging.
send_distribution_bucketsbooleanoptionalfalseSet send_distribution_buckets to true to send and convert OpenMetrics histograms to Distribution metrics.

send_histograms_buckets must be set to true (default value).
send_histogram_bucketsbooleanoptionaltrueSet send_histograms_buckets to true to send the histograms bucket.
send_monotonic_counterbooleanoptionaltrueTo send counters as monotonic counter

see the relevant issue in GitHub.
exclude_labelslist of stringoptionalnoneList of labels to be excluded.
ssl_certstringoptionalnoneIf your Prometheus endpoint is secured, here are the settings to configure it:
Can either be: only the path to the certificate and thus you should specify the private key
, or it can be the path to a file containing both the certificate and the private key.
ssl_private_keystringoptionalnoneNeeded if the certificate does not include the private key.
WARNING: The private key to your local certificate must be unencrypted.
ssl_ca_certstringoptionalnoneThe path to the trusted CA used for generating custom certificates. Set this to false to disable SSL certificate
verification.
prometheus_timeoutintegeroptional10Set a timeout in seconds for the Prometheus/OpenMetrics query.
max_returned_metricsintegeroptional2000The check limits itself to 2000 metrics by default. Increase this limit if needed.

Note: All parameters but send_distribution_buckets are supported by both OpenMetrics check and Prometheus check.

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Further Reading