--- title: OpenMetrics description: OpenMetrics is an open standard for exposing metric data breadcrumbs: Docs > Integrations > OpenMetrics --- # OpenMetrics Supported OS Integration version7.3.0 ## Overview{% #overview %} Extract custom metrics from any OpenMetrics or Prometheus endpoints. {% alert level="warning" %} All the metrics retrieved by this integration are considered [custom metrics](https://docs.datadoghq.com/developers/metrics/custom_metrics). {% /alert %} The integration is compatible with both the [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format) as well as with the [OpenMetrics specification](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#suffixes). **Minimum Agent version:** 6.6.0 ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/agent/kubernetes/integrations/) for guidance on applying these instructions. This integration has a latest mode (enabled by setting `openmetrics_endpoint` to point to the target endpoint) and a legacy mode (enabled by setting `prometheus_url` instead). To get all the most up-to-date features, Datadog recommends enabling the latest mode. For more information, see [Latest and Legacy Versioning For OpenMetrics-based Integrations](https://docs.datadoghq.com/integrations/guide/versions-for-openmetrics-based-integrations). ### Installation{% #installation %} The OpenMetrics check is packaged with the [Datadog Agent v6.6.0 or later](https://docs.datadoghq.com/getting_started/integrations/prometheus/?tab=docker#configuration). ### Configuration{% #configuration %} Edit the `conf.d/openmetrics.d/conf.yaml` file at the root of your [Agent's configuration directory](https://docs.datadoghq.com/agent/guide/agent-configuration-files/#agent-configuration-directory). See the [sample openmetrics.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example) for all available configuration options. This is the latest OpenMetrics check example as of Datadog Agent version 7.32.0. If you previously implemented this integration, see the [legacy example](https://github.com/DataDog/integrations-core/blob/7.30.x/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example). For each instance, the following parameters are required: | Parameter | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `openmetrics_endpoint` | The URL where your application metrics are exposed in Prometheus or OpenMetrics format (must be unique). | | `namespace` | The namespace to prepend to all metrics. | | `metrics` | A list of metrics to retrieve as custom metrics. Add each metric to the list as `metric_name` or `metric_name: renamed` to rename it. The metrics are interpreted as regular expressions. Use `".*"` as a wildcard (`metric.*`) to fetch all matching metrics. **Note**: Regular expressions can potentially send a lot of custom metrics. | Starting in Datadog Agent v7.32.0, in adherence to the [OpenMetrics specification standard](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#suffixes), counter names ending in `_total` must be specified without the `_total` suffix. For example, to collect `promhttp_metric_handler_requests_total`, specify the metric name `promhttp_metric_handler_requests`. This submits to Datadog the metric name appended with `.count`, `promhttp_metric_handler_requests.count`. This check has a limit of 2000 metrics per instance. The number of returned metrics is indicated when running the Datadog Agent [status command](https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information). You can specify the metrics you are interested in by editing the configuration. To learn how to customize the metrics to collect, see [Prometheus and OpenMetrics Metrics Collection](https://docs.datadoghq.com/getting_started/integrations/prometheus/). If you need to monitor more metrics, contact [Datadog support](https://docs.datadoghq.com/help/). ### Validation{% #validation %} [Run the Agent's status subcommand](https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information) and look for `openmetrics` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} All metrics collected by the OpenMetrics check are forwarded to Datadog as custom metrics. ### Events{% #events %} The OpenMetrics check does not include any events. ### Service Checks{% #service-checks %} The OpenMetrics check does not include any service checks. ## Troubleshooting{% #troubleshooting %} ### High custom metrics billing{% #high-custom-metrics-billing %} OpenMetrics configurations with generic wildcard values for the `metrics` option have significant impact on custom metrics billing. Datadog recommends using specific metric names or partial metric name matches for more precise collection. ### Missing untyped metrics{% #missing-untyped-metrics %} By default, the integration skips metrics that come without a type on a Prometheus exposition. If you want to collect untyped metrics, you must explicitly specify their type in the `metrics` mapping, for example: ```yaml metrics: - "": "type": "gauge" ``` If you are collecting metrics through annotations, refer to the following example: ```json "metrics": [ { "": { "type": "gauge" } } ] ``` Remember that metric names can be specified as regular expressions, making it possible to specify the type for a set of metrics without listing all of them individually. ### Errors parsing the OpenMetrics payload with Agent 7.46{% #errors-parsing-the-openmetrics-payload-with-agent-746 %} The version of this integration shipped with version 7.46 of the Agent gives preference by default to the OpenMetrics format when requesting metrics from the metrics endpoint. It does so by setting the `Accept` header to `application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1`. This was done in combination with dynamically determining which scraper to use based on the `Content-Type` it receives from the server, to reduce the need for manual setup. Previous versions defaulted to `text/plain`, which normally results in the server returning metrics in the Prometheus exposition format. This means that updating to this version of the integration may result in switching from the Prometheus format to the OpenMetrics format. Although the behavior should remain the same in most circumstances, some applications return metrics in a format that is not fully OpenMetrics-compliant, despite setting the `Content-Type` to signal the use of the OpenMetrics standard format. This may cause our integration to report errors while parsing the metrics payload. If you see parsing errors when scraping the OpenMetrics endpoint with this new version, you can force the use of the less strict Prometheus format by manually setting the `Accept` header that the integration sends to `text/plain` using the `headers` option in the [configuration file](https://github.com/DataDog/integrations-core/blob/7.46.x/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example#L537-L546). For instance: ```yaml ## All options defined here are available to all instances. # init_config: ... instances: - openmetrics_endpoint: ... headers: Accept: text/plain ``` Need help? Contact [Datadog support](https://docs.datadoghq.com/help/). ## Further Reading{% #further-reading %} - [Kubernetes Prometheus and OpenMetrics metrics collection](https://docs.datadoghq.com/agent/openmetrics/) - [Custom OpenMetrics Check](https://docs.datadoghq.com/developers/openmetrics/)