Overview
Extract custom metrics from any OpenMetrics or Prometheus endpoints.
All the metrics retrieved by this integration are considered
custom metrics.
The integration is compatible with both the Prometheus exposition format as well as with the OpenMetrics specification.
Setup
Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.
This integration has a latest mode (enabled by setting openmetrics_endpoint to point to the target endpoint) and a legacy mode (enabled by setting prometheus_url instead). To get all the most up-to-date features, Datadog recommends enabling the latest mode. For more information, see Latest and Legacy Versioning For OpenMetrics-based Integrations.
Installation
The OpenMetrics check is packaged with the Datadog Agent v6.6.0 or later.
Configuration
Edit the openmetrics.d/conf.yaml file at the root of your Agent’s configuration directory. See the sample openmetrics.d/conf.yaml for all available configuration options. This is the latest OpenMetrics check example as of Datadog Agent version 7.32.0. If you previously implemented this integration, see the legacy example.
For each instance, the following parameters are required:
| Parameter | Description |
|---|
openmetrics_endpoint | The URL where your application metrics are exposed in Prometheus or OpenMetrics format (must be unique). |
namespace | The namespace to prepend to all metrics. |
metrics | A list of metrics to retrieve as custom metrics. Add each metric to the list as metric_name or metric_name: renamed to rename it. The metrics are interpreted as regular expressions. Use ".*" as a wildcard (metric.*) to fetch all matching metrics. Note: Regular expressions can potentially send a lot of custom metrics. |
Starting in Datadog Agent v7.32.0, in adherence to the OpenMetrics specification standard, counter names ending in _total must be specified without the _total suffix. For example, to collect promhttp_metric_handler_requests_total, specify the metric name promhttp_metric_handler_requests. This submits to Datadog the metric name appended with .count, promhttp_metric_handler_requests.count.
This check has a limit of 2000 metrics per instance. The number of returned metrics is indicated when running the Datadog Agent status command. You can specify the metrics you are interested in by editing the configuration. To learn how to customize the metrics to collect, see Prometheus and OpenMetrics Metrics Collection.
If you need to monitor more metrics, contact Datadog support.
Validation
Run the Agent’s status subcommand and look for openmetrics under the Checks section.
Data Collected
Metrics
All metrics collected by the OpenMetrics check are forwarded to Datadog as custom metrics.
Events
The OpenMetrics check does not include any events.
Service Checks
The OpenMetrics check does not include any service checks.
Troubleshooting
High custom metrics billing
OpenMetrics configurations with generic wildcard values for the metrics option have significant impact on custom metrics billing.
Datadog recommends using specific metric names or partial metric name matches for more precise collection.
Missing untyped metrics
By default, the integration skips metrics that come without a type on a Prometheus exposition. If you want to collect untyped metrics, you must explicitly specify their type in the metrics mapping, for example:
metrics:
- "metric_without_type":
"type": "gauge"
Remember that metric names can be specified as regular expressions, making it possible to specify the type for a set of metrics without listing all of them individually.
Errors parsing the OpenMetrics payload with Agent 7.46 and above
Starting with version 3.0.0 of this integration, which is shipped by default with Agent 7.46 and above, the integration sends by default the Accept header set to application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1. Previous versions set the Accept header to text/plain. The integration then dynamically determines which scraper to use based on the Content-type it receives from the server.
If you see errors scraping the OpenMetrics endpoint with this new version because the scraper is stricter than before, manually set the Accept header that the integration sends to text/plain using the headers option in the configuration file. For instance:
## All options defined here are available to all instances.
#
init_config:
...
instances:
- openmetrics_endpoint: <OPENMETRICS_ENDPOINT>
...
headers:
Accept: text/plain
With this configuration, the endpoint returns the Content-type set to text/plain, causing the integration to use the previous scraper.
The OpenMetrics integration reports an error parsing the payload when the system you are monitoring sends data that does not match their Content-type header. Setting the integration to accept text/plain content creates a workaround to address the problem in the short term.
To fix the root cause of the problem, reach out to the maintainers of the upstream system. Submit a bug report and ask them to fix the system so the payload and the Content-type set in the header match.
Need help? Contact Datadog support.
Further Reading
