Collect your exposed Prometheus and OpenMetrics metrics from your application running inside Kubernetes by using the Datadog Agent, and the Datadog-OpenMetrics or Datadog-Prometheus integrations.
Starting with version 6.5.0, the Agent includes OpenMetrics and Prometheus checks capable of scraping Prometheus endpoints. Datadog recommends using the OpenMetrics check since it is more efficient and fully supports Prometheus text format. For more advanced usage of the OpenMetricsCheck interface, including writing a custom check, see the Developer Tools section. Use the Prometheus check only when the metrics endpoint does not support a text format.
This page explains the basic usage of these checks, which enable you to scrape custom metrics from Prometheus endpoints.
For an explanation of how Prometheus and OpenMetrics metrics map to Datadog metrics, see the Mapping Prometheus Metrics to Datadog Metrics guide.
Deploy the Datadog Agent in your Kubernetes cluster. OpenMetrics and Prometheus checks are included in the Datadog Agent package, so you don’t need to install anything else on your containers or hosts.
Configure your OpenMetrics or Prometheus check using Autodiscovery, by applying the following annotations to your OpenMetrics/Prometheus pod:
# (...)
metadata:
#(...)
annotations:
ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: |
["openmetrics"]
ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: |
[{}]
ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: |
[
{
"prometheus_url": "http://%%host%%:%%port%%/<PROMETHEUS_ENDPOINT> ",
"namespace": "<METRICS_NAMESPACE_PREFIX_FOR_DATADOG>",
"metrics": [{"<METRIC_TO_FETCH>":"<NEW_METRIC_NAME>"}]
}
]
spec:
containers:
- name: '<CONTAINER_IDENTIFIER>'
With the following configuration placeholder values:
| Placeholder | Description |
|---|---|
<PROMETHEUS_PORT> | Port to connect to in order to access the Prometheus endpoint. |
<PROMETHEUS_ENDPOINT> | URL for the metrics served by the container, in Prometheus format. |
<METRICS_NAMESPACE_PREFIX_FOR_DATADOG> | Set namespace to be prefixed to every metric when viewed in Datadog. |
<METRIC_TO_FETCH> | Prometheus metrics key to be fetched from the Prometheus endpoint. |
<NEW_METRIC_NAME> | Optional parameter which, if set, transforms the <METRIC_TO_FETCH> metric key to <NEW_METRIC_NAME> in Datadog. If you choose not to use this option, pass a list of strings rather than key:value pairs. |
For a full list of available parameters for instances, including namespace and metrics, see the table in the Prometheus host collection guide.
Use this Prometheus DaemonSet prometheus.yaml to launch a Prometheus pod with already the Autodiscovery configuration in it:
Autodiscovery configuration:
# (...)
spec:
replicas: 2
selector:
matchLabels:
app: prometheus
purpose: example
template:
metadata:
labels:
app: prometheus
purpose: example
annotations:
ad.datadoghq.com/prometheus-example.check_names: |
["openmetrics"]
ad.datadoghq.com/prometheus-example.init_configs: |
[{}]
ad.datadoghq.com/prometheus-example.instances: |
[
{
"prometheus_url": "http://%%host%%:%%port%%/metrics",
"namespace": "documentation_example_kubernetes",
"metrics": [ {"promhttp_metric_handler_requests_total": "prometheus.handler.requests.total"}]
}
]
# (...)
Command to create the Prometheus pod:
kubectl create -f prometheus.yaml
Go into your Metric summary page to see the collected metrics: prometheus_target_interval_length_seconds*
By default, all metrics retrieved by the generic Prometheus check are considered custom metrics. If you are monitoring off-the-shelf software and think it deserves an official integration, don’t hesitate to contribute!
Official integrations have their own dedicated directories. There’s a default instance mechanism in the generic check to hardcode the default configuration and metrics metadata. For example, reference the kube-proxy integration.
Additional helpful documentation, links, and articles:
