---
title: Prometheus Destination
description: >-
  Learn how to send metrics to Prometheus using the Observability Pipelines
  Worker.
breadcrumbs: Docs > Observability Pipelines > Destinations > Prometheus Destination
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Prometheus Destination
Available for:
{% icon name="icon-metrics" /%}
 Metrics 
## Overview{% #overview %}

Use Observability Pipelines' Prometheus destination to send metrics to Prometheus.

## Set up destination{% #set-up-destination %}

{% alert level="danger" %}
For Secrets Management: Only enter the identifiers for the endpoint URL and, if applicable, the username and password for basic authorization, the bearer token, and the TLS key pass. Do not enter the actual values.
{% /alert %}

Configure the Prometheus destination when you [set up a pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md). You can set up a pipeline in the [UI](https://app.datadoghq.com/observability-pipelines), using the [API](https://docs.datadoghq.com/api/latest/observability-pipelines.md), or with [Terraform](https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline). The steps in this section are configured in the UI.

After you select the Prometheus destination in the pipeline UI:

1. Enter the identifier for your Remote Write Endpoint URL. An example of an endpoint URL the identifier references: `http://localhost:9090/api/v1/write`. If you leave the identifier field blank, the default is used.
1. Select your authorization strategy (None, Basic, or Bearer). If you selected:
   - Basic: Enter the identifier for your username and password. If you leave it blank, the default is used.
   - Bearer: Enter the identifier for your bearer token. If you leave it blank, the default is used.

{% alert level="info" %}
If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with `DD_OP_`. For example, if you entered `PASSWORD_1` for a password identifier, the environment variable for that password is `DD_OP_PASSWORD_1`.
{% /alert %}

### Optional settings{% #optional-settings %}

#### Default namespace{% #default-namespace %}

Enter the default namespace for any metrics sent. This namespace is only used if a metric has no existing namespace. It is added as a prefix to the metric name, separated by an underscore (`_`). The namespace must follow the [Prometheus naming convention](https://prometheus.io/docs/practices/naming/).

#### Tenant ID{% #tenant-id %}

Enter the tenant ID to add to outgoing requests. This field supports [template syntax](https://docs.datadoghq.com/observability_pipelines/destinations.md#template-syntax), but the template must have a literal prefix, such as `prefix-{{ tenant_id }}` or `prefix/{{ tenant_id }}`. Templates without a literal prefix, such as `{{ tenant_id }}`, are rejected; the Worker logs an error, and the pipeline isn't started.

#### Enable TLS{% #enable-tls %}

Toggle the switch to **Enable TLS**.

- If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
- Enter the following certificate and key files:
  - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  - (Optional) `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  - (Optional) `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  - **Notes**:
    - The configuration data directory `/var/lib/observability-pipelines-worker/config/` is automatically appended to the file paths. See [Advanced Worker Configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md) for more information.
    - The file must be readable by the `observability-pipelines-worker` group and user.

- (Optional) Enter the server name to use for certificate validation. If left blank, the hostname from the endpoint URL is used.

#### Buffering{% #buffering %}

Toggle the switch to enable **Buffering Options**. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn't create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See [Destination buffers](https://docs.datadoghq.com/observability_pipelines/scaling_and_performance/buffering_and_backpressure.md#destination-buffers) for more information.

- If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
- To configure a buffer on your destination:
  1. Select the buffer type you want to set (**Memory** or **Disk**).
  1. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     1. Maximum disk buffer size is 5 TB.
        - **Note**: For Worker versions 2.20.x and older, the maximum disk buffer size is 500 GB.
  1. In the **Behavior on full buffer** dropdown menu, select whether you want to **block** events or **drop new events** when the buffer is full.

## Allow out-of-order samples{% #allow-out-of-order-samples %}

The Worker doesn't always send metrics in the correct order for a given series because it doesn't reorder metrics. For example, if the first batch of metrics contains metrics with timestamps: `10:03`, `10:04`, `10:05` and the second batch contains metrics with timestamps: `10:01`, `10:02`, `10:06`, the Worker does not reorder those metrics before sending them out.

Because the Prometheus remote write receiver rejects out-of-order samples, the Worker logs a Bad Request (`400`) error and the entire second batch of metrics gets dropped, even if the remote write receiver accepted some of the valid metrics in the batch.

Datadog recommends setting your remote write receiver to allow out-of-order samples to prevent out-of-order samples from getting dropped.

## Troubleshooting{% #troubleshooting %}

### Debug error logs{% #debug-error-logs %}

If you see `400` or `500` error logs from this destination, you can enable debug logs to see the response returned by the server. To enable logs for this HTTP-based destination only and not every Worker module, set `VECTOR_LOG` to `info,vector::sinks::util::http=debug`:

```
docker run -i -e DD_API_KEY=<DATADOG_API_KEY> \
   -e DD_OP_PIPELINE_ID=<PIPELINE_ID> \
   -e VECTOR_LOG=info,vector::sinks::util::http=debug \
   datadog/observability-pipelines-worker run
```

See [Enable debug logs](https://docs.datadoghq.com/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md#enable-debug-logs) for instruction on enabling full debug logs.

## Secret defaults{% #secret-defaults %}

These are the defaults used for secret identifiers and environment variables.

{% tab title="Secrets Management" %}

- Remote Write endpoint URL identifier:
  - References the Remote Write endpoint URL. An example of an endpoint URL the identifier references: `http://localhost:9090/api/v1/write`.
  - The default identifier is `DESTINATION_PROMETHEUS_REMOTE_WRITE_ENDPOINT`.
- Prometheus TLS passphrase identifier (when TLS is enabled):
  - The default identifier is `DESTINATION_PROMETHEUS_REMOTE_WRITE_KEY_PASS`.
- If you are using basic authentication:
  - Prometheus username identifier:
    - The default identifier is `DESTINATION_PROMETHEUS_REMOTE_WRITE_USERNAME`.
  - Prometheus password identifier:
    - The default identifier is `DESTINATION_PROMETHEUS_REMOTE_WRITE_PASSWORD`.
- If you are using bearer authentication:
  - Prometheus bearer token identifier:
    - The default identifier is `DESTINATION_PROMETHEUS_REMOTE_WRITE_BEARER_TOKEN`.

{% /tab %}

{% tab title="Environment Variables" %}

- Prometheus endpoint URL:
  - The default environment variable is `DD_OP_DESTINATION_PROMETHEUS_REMOTE_WRITE_ENDPOINT`.
- Prometheus TLS passphrase (when enabled):
  - The default environment variable is `DD_OP_DESTINATION_PROMETHEUS_REMOTE_KEY_PASS`.
- If you are using basic authentication:
  - Prometheus endpoint authentication username and password.
  - The default environment variable is `DD_OP_DESTINATION_PROMETHEUS_REMOTE_USERNAME` and `DD_OP_DESTINATION_PROMETHEUS_REMOTE_PASSWORD`.
- If you are using bearer authentication:
  - Prometheus endpoint bearer token.
  - The default environment variable is `DD_OP_DESTINATION_PROMETHEUS_REMOTE_BEARER_TOKEN`.

{% /tab %}