Join us at the Dash conference! July 16-17, NYC

Prometheus metrics collection

Collect your exposed Prometheus metrics from your application running inside containers or directly on your host using the Datadog Agent and the Datadog-Prometheus integration.

Overview

Starting with version 6.1.0, the Agent includes a new Prometheus check capable of scraping Prometheus endpoints with only a few lines of configuration. This page explains the basic usage of the generic Prometheus check, the fastest and simplest way to scrape custom metrics from Prometheus endpoints. For more advanced usage of the PrometheusCheck interface, including writing a custom check, see the Developer Tools section.

Setup

Installation

Install the Datadog Agent for your corresponding Operating system. The Prometheus/Openmetrics check is included in the Datadog Agent package, so you don’t need to install anything else on your containers or hosts.

Configuration

If running the Agent as a binary on a host, configure your Prometheus/Openmetrics check as any other Agent integration. If running the Agent as a DaemonSet in Kubernetes, configure your Prometheus/Openmetrics check using auto-discovery.

First, edit the prometheus.d/conf.yaml file in the conf.d/ folder at the root of your Agent’s configuration directory to configure your Datadog-Prometheus integration. Then restart the Agent to start collecting your metrics. Find below the minimum required configuration needed to enable the integration and see the sample prometheus.d/conf.yaml for all available configuration options.

init_config:

instances:
  - prometheus_url: 'localhost:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT>'
    namespace: '<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>'
    metrics:
      - '<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>'

The Agent detects if it’s running on Docker and automatically searches all containers labels for the Datadog-Prometheus labels. Autodiscovery expects labels to look like these examples, depending on the file type:

Dockerfile

LABEL "com.datadoghq.ad.check_names"='["prometheus"]'
LABEL "com.datadoghq.ad.init_configs"='["{}"]'
LABEL "com.datadoghq.ad.instances"='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

docker-compose.yaml

labels:
  com.datadoghq.ad.check_names: '["prometheus"]'
  com.datadoghq.ad.init_configs: '["{}"]'
  com.datadoghq.ad.instances: '["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

docker run command

-l com.datadoghq.ad.check_names='["prometheus"]' -l com.datadoghq.ad.init_configs='["{}"]' -l com.datadoghq.ad.instances='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

As all Datadog Agent integrations, the Datadog-Prometheus Integration can be configured using Autodiscovery. This allows you to send a given container/pod exposed Prometheus metrics to Datadog by applying the following annotations to it:

# (...)
metadata:
#(...)
  annotations:
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: |
        ["prometheus"]
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: |
        [{}]
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: |
        [
          {
            "prometheus_url": "http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> ",
            "namespace": "<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>",
            "metrics": ["<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>"]
          }
        ]
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'

With the following configuration placeholder values:

  • <PROMETHEUS_PORT>: Port to connect to in order to access the Prometheus endpoint.
  • <PROMETHEUS_ENDPOINT>: URL for the metrics as served by the container in Prometheus format
  • <METRICS_NAMESPACE_PREFIX_FOR_DATADOG>: Set namespace to be prefixed to every metric when viewed in DataDog UI
  • <PROMETHEUS_METRIC_TO_FETCH>: Prometheus metrics key to be fetched from the prometheus endpoint.
  • <DATADOG_NEW_METRIC_NAME>: Optional parameter which if set, transforms in Datadog the <PROMETHEUS_METRIC_TO_FETCH> metric key to <DATADOG_NEW_METRIC_NAME>.

Find below the full list of parameter that can be used for your instances:

Name Type Necessity Default value  Description
prometheus_url string required none The URL where your application metrics are exposed by Prometheus.
namespace string required none The namespace to be appended before all metrics namespace. Your metrics are collected in the form namespace.metric_name.
metrics list of key:value elements required none List of <METRIC_TO_FETCH>: <NEW_METRIC_NAME> for metrics to be fetched from the prometheus endpoint.
is optional. It transforms the name in Datadog if set.
This list should contain at least one metric
prometheus_metrics_prefix string optional none Prefix for exposed Prometheus metrics.
health_service_check boolean optional true Send a service check reporting about the health of the prometheus endpoint
It will be named .prometheus.health
label_to_hostname string optional none Override the hostname with the value of one label.
label_joins object optional none The label join allows to target a metric and retrieve it’s label via a 1:1 mapping
labels_mapper list of key:value element optional none The label mapper allows you to rename some labels
Format is :
type_overrides list of key:value element optional none Type override allows you to override a type in the prometheus payload
or type an untyped metrics (they’re ignored by default)
Supported are gauge, counter, histogram, summary
tags list of key:value element optional none List of tags to attach to every metric, event and service check emitted by this integration.

Learn more about tagging: https://docs.datadoghq.com/tagging/
send_histogram_buckets boolean optional true Set send_histograms_buckets to true to send the histograms bucket.
send_monotonic_counter boolean optional true To send counters as monotonic counter

see: https://github.com/DataDog/integrations-core/issues/1303
exclude_labels list of string optional none List of label to be excluded.
ssl_cert string optional none If your prometheus endpoint is secured, here are the settings to configure it
Can either be only the path to the certificate and thus you should specify the private key
or it can be the path to a file containing both the certificate & the private key
ssl_private_key string optional none Needed if the certificate does not include the private key
WARNING: The private key to your local certificate must be unencrypted.
ssl_ca_cert string optional none The path to the trusted CA used for generating custom certificates. Set this to false to disable SSL certificate
verification.
prometheus_timeout integer optional 10 Set a timeout in second for the prometheus query.
max_returned_metrics integer optional 2000 The check limits itself to 2000 metrics by default, increase this limit if needed.

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Further Reading