Prometheus and OpenMetrics metrics collection from a host

Prometheus and OpenMetrics metrics collection from a host

Collect your exposed Prometheus and OpenMetrics metrics from your application running on your hosts using the Datadog Agent, and the Datadog-OpenMetrics or Datadog-Prometheus integrations.

Overview

Starting with version 6.5.0, the Agent includes OpenMetrics and Prometheus checks capable of scraping Prometheus endpoints. Datadog recommends using the OpenMetrics check since it is more efficient and fully supports Prometheus text format. For more advanced usage of the OpenMetricsCheck interface, including writing a custom check, see the Developer Tools section. Use the Prometheus check only when the metrics endpoint does not support a text format.

This page explains the basic usage of these checks, enabling you to import all your Prometheus exposed metrics within Datadog.

Setup

Installation

Install the Datadog Agent for your corresponding operating system. OpenMetrics and Prometheus checks are included in the Datadog Agent package, so you don’t need to install anything else on your containers or hosts.

Configuration

To collect your exposed metrics:

  1. Edit the openmetrics.d/conf.yaml file in the conf.d/ folder at the root of your Agent’s configuration directory. See the sample openmetrics.d/conf.yaml for all available configuration options. This is the minimum required configuration needed to enable the integration::

    init_config:
    
    instances:
        - prometheus_url: 'localhost:<PORT>/<ENDPOINT>'
          namespace: '<NAMESPACE>'
          metrics:
              - '<METRIC_TO_FETCH>': '<DATADOG_METRIC_NAME>'
    

    With the following configuration placeholder values:

    PlaceholderDescription
    <PORT>Port to connect to in order to access the Prometheus endpoint.
    <ENDPOINT>URL for the metrics served by the container, in Prometheus format.
    <NAMESPACE>Set namespace to be prefixed to every metric when viewed in Datadog.
    <METRIC_TO_FETCH>Prometheus metrics key to be fetched from the Prometheus endpoint.
    <DATADOG_METRIC_NAME>Optional parameter which, if set, transforms the <METRIC_TO_FETCH> metric key to <DATADOG_METRIC_NAME> in Datadog.
    If you choose not to use this option, pass a list of strings rather than key:value pairs.
  2. Restart the Agent to start collecting your metrics.

Parameters available

Find below the full list of parameters that can be used for your instances:

NameTypeNecessityDefault valueDescription
prometheus_urlstringrequirednoneThe URL where your application metrics are exposed by Prometheus/OpenMetrics.
namespacestringrequirednoneThe namespace to be appended before all metrics namespaces. Your metrics are collected in the form namespace.metric_name.
metricslist of strings or key:value elementsrequirednoneList of <METRIC_TO_FETCH>: <NEW_METRIC_NAME> pairs for metrics to be fetched from the Prometheus endpoint.
<NEW_METRIC_NAME> is optional. It transforms the name in Datadog if set. This list should contain at least one metric.
prometheus_metrics_prefixstringoptionalnonePrefix for exposed Prometheus/OpenMetrics metrics.
health_service_checkbooleanoptionaltrueSend a service check reporting on the health of the Prometheus endpoint. The check is named <NAMESPACE>.prometheus.health.
label_to_hostnamestringoptionalnoneOverride the hostname with the value of one label.
label_joinsobjectoptionalnoneThe label join allows you to target a metric and retrieve its label via a 1:1 mapping.
labels_mapperlist of key:value elementoptionalnoneThe label mapper allows you to rename some labels. Format: <LABEL_TO_RENAME>: <NEW_LABEL_NAME>.
type_overrideslist of key:value elementoptionalnoneType override allows you to override a type in the Prometheus payload or type an untyped metric (they’re ignored by default).
Supported <METRIC_TYPE>s are gauge, monotonic_count, histogram, and summary.
tagslist of key:value elementoptionalnoneList of tags to attach to every metric, event, and service check emitted by this integration.
Learn more about tagging.
send_distribution_bucketsbooleanoptionalfalseSet send_distribution_buckets to true to send and convert OpenMetrics histograms to Distribution metrics.
send_histograms_buckets must be set to true (default value).
send_distribution_counts_as_monotonicbooleanoptionalfalseSet send_distribution_counts_as_monotonic to true to send OpenMetrics histogram/summary counts as monotonic counts.
send_histograms_bucketsbooleanoptionaltrueSet send_histograms_buckets to true to send the histograms bucket.
send_monotonic_counterbooleanoptionaltrueo send counts as monotonic counts see the relevant issue in GitHub.
exclude_labelslist of stringoptionalnoneList of labels to be excluded.
ssl_certstringoptionalnoneIf your Prometheus endpoint is secured, here are the settings to configure it:
Can either be: only the path to the certificate and thus you should specify the private key, or it can be the path to a file containing both the certificate and the private key.
ssl_private_keystringoptionalnoneNeeded if the certificate does not include the private key.
WARNING: The private key to your local certificate must be unencrypted.
ssl_ca_certstringoptionalnoneThe path to the trusted CA used for generating custom certificates.
prometheus_timeoutintegeroptional10Set a timeout in seconds for the Prometheus/OpenMetrics query.
max_returned_metricsintegeroptional2000The check limits itself to 2000 metrics by default. Increase this limit if needed.
bearer_token_authbooleanoptionalfalseSet bearer_token_auth to true to add a bearer token authentication header. Note: If bearer_token_path is not set, /var/run/secrets/kubernetes.io/serviceaccount/token is used as the default path.
bearer_token_pathstringoptionalnoneThe path to a Kubernetes service account bearer token file (make sure the file exists and is mounted correctly). Note: Set bearer_token_auth to true to enable adding the token to HTTP headers for authentication.

Note: All parameters but send_distribution_buckets and send_distribution_counts_as_monotonic are supported by both OpenMetrics check and Prometheus check.

Getting started

Simple metric collection

To get started with collecting metrics exposed by Prometheus, follow these steps:

  1. Follow the Prometheus Getting Started documentation to start a local version of Prometheus that monitors itself.

  2. Install the Datadog Agent for your platform.

  3. Edit the openmetrics.d/conf.yaml file in the conf.d/ folder at the root of your Agent’s configuration directory with the following content:

    init_config:
    
    instances:
        - prometheus_url: http://localhost:9090/metrics
          namespace: 'documentation_example'
          metrics:
              - promhttp_metric_handler_requests_total: prometheus.handler.requests.total
    
  4. Restart the Agent.

  5. Go into your Metric summary page to see the collected metrics: prometheus_target_interval_length_seconds*

From custom to official integration

By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!

Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.

Further Reading