New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Prometheus metrics collection

Collect your exposed Prometheus metrics from your application running inside containers or directly on your host using the Datadog Agent and the Datadog-Prometheus integration.

Overview

Starting with version 6.1.0, the Agent includes a Prometheus check capable of scraping Prometheus endpoints with a few lines of configuration. This page explains the basic usage of the generic Prometheus check, which enables you to scrape custom metrics from Prometheus endpoints. For more advanced usage of the PrometheusCheck interface, including writing a custom check, see the Developer Tools section.

Setup

Installation

Install the Datadog Agent for your corresponding operating system. The Prometheus/OpenMetrics check is included in the Datadog Agent package, so you don’t need to install anything else on your containers or hosts.

Configuration

If running the Agent as a binary on a host, configure your Prometheus/OpenMetrics check as any other Agent integration. If running the Agent as a DaemonSet in Kubernetes, configure your Prometheus/OpenMetrics check using Autodiscovery.

First, edit the prometheus.d/conf.yaml file in the conf.d/ folder at the root of your Agent’s configuration directory to configure your Datadog-Prometheus integration. Then restart the Agent to start collecting your metrics. Find below the minimum required configuration needed to enable the integration and see the sample prometheus.d/conf.yaml for all available configuration options.

init_config:

instances:
  - prometheus_url: 'localhost:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT>'
    namespace: '<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>'
    metrics:
      - '<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>'

The Agent detects if it’s running on Docker and automatically searches all containers labels for Datadog-Prometheus labels. Autodiscovery expects labels to look like these examples, depending on the file type:

Dockerfile

LABEL "com.datadoghq.ad.check_names"='["prometheus"]'
LABEL "com.datadoghq.ad.init_configs"='["{}"]'
LABEL "com.datadoghq.ad.instances"='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

docker-compose.yaml

labels:
  com.datadoghq.ad.check_names: '["prometheus"]'
  com.datadoghq.ad.init_configs: '["{}"]'
  com.datadoghq.ad.instances: '["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

docker run command

-l com.datadoghq.ad.check_names='["prometheus"]' -l com.datadoghq.ad.init_configs='["{}"]' -l com.datadoghq.ad.instances='["{\"prometheus_url\":\"http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> \",\"namespace\":\"<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>\",\"metrics\":[\"<PROMETHEUS_METRIC_TO_FETCH>: <DATADOG_NEW_METRIC_NAME>\"]}"]'

Like all Datadog Agent integrations, the Datadog-Prometheus integration can be configured using Autodiscovery. This allows you to send a given container/pod exposed Prometheus metrics to Datadog by applying the following annotations to it:

# (...)
metadata:
#(...)
  annotations:
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: |
        ["prometheus"]
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: |
        [{}]
      ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: |
        [
          {
            "prometheus_url": "http://%%host%%:<PROMETHEUS_PORT>/<PROMETHEUS_ENDPOINT> ",
            "namespace": "<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>",
            "metrics": [{"<METRIC_TO_FETCH>":"<NEW_METRIC_NAME>"}]
          }
        ]
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'

With the following configuration placeholder values:

  • <PROMETHEUS_PORT>: Port to connect to in order to access the Prometheus endpoint.
  • <PROMETHEUS_ENDPOINT>: URL for the metrics served by the container, in Prometheus format.
  • <METRICS_NAMESPACE_PREFIX_FOR_DATADOG>: Set namespace to be prefixed to every metric when viewed in Datadog.
  • <METRIC_TO_FETCH>: Prometheus metrics key to be fetched from the Prometheus endpoint.
  • <NEW_METRIC_NAME>: Optional parameter which, if set, transforms the <METRIC_TO_FETCH> metric key to <NEW_METRIC_NAME> in Datadog.

Find below the full list of parameters that can be used for your instances:

NameTypeNecessityDefault value Description
prometheus_urlstringrequirednoneThe URL where your application metrics are exposed by Prometheus.
namespacestringrequirednoneThe namespace to be appended before all metrics namespaces. Your metrics are collected in the form namespace.metric_name.
metricslist of key:value elementsrequirednoneList of <METRIC_TO_FETCH>: <NEW_METRIC_NAME> pairs for metrics to be fetched from the Prometheus endpoint.
<NEW_METRIC_NAME> is optional. It transforms the name in Datadog if set.
This list should contain at least one metric.
prometheus_metrics_prefixstringoptionalnonePrefix for exposed Prometheus metrics.
health_service_checkbooleanoptionaltrueSend a service check reporting on the health of the Prometheus endpoint.
The check is named <NAMESPACE>.prometheus.health.
label_to_hostnamestringoptionalnoneOverride the hostname with the value of one label.
label_joinsobjectoptionalnoneThe label join allows you to target a metric and retrieve its label via a 1:1 mapping.
labels_mapperlist of key:value elementoptionalnoneThe label mapper allows you to rename some labels.
Format: <LABEL_TO_RENAME>: <NEW_LABEL_NAME>.
type_overrideslist of key:value elementoptionalnoneType override allows you to override a type in the Prometheus payload
or type an untyped metric (they’re ignored by default).
Supported <METRIC_TYPE>s are gauge, counter, histogram, and summary.
tagslist of key:value elementoptionalnoneList of tags to attach to every metric, event, and service check emitted by this integration.

Learn more about tagging.
send_histogram_bucketsbooleanoptionaltrueSet send_histograms_buckets to true to send the histograms bucket.
send_monotonic_counterbooleanoptionaltrueTo send counters as monotonic counter

see the relevant issue in GitHub.
exclude_labelslist of stringoptionalnoneList of labels to be excluded.
ssl_certstringoptionalnoneIf your Prometheus endpoint is secured, here are the settings to configure it:
Can either be: only the path to the certificate and thus you should specify the private key
, or it can be the path to a file containing both the certificate and the private key.
ssl_private_keystringoptionalnoneNeeded if the certificate does not include the private key.
WARNING: The private key to your local certificate must be unencrypted.
ssl_ca_certstringoptionalnoneThe path to the trusted CA used for generating custom certificates. Set this to false to disable SSL certificate
verification.
prometheus_timeoutintegeroptional10Set a timeout in seconds for the Prometheus query.
max_returned_metricsintegeroptional2000The check limits itself to 2000 metrics by default. Increase this limit if needed.

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Further Reading