For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/observability_pipelines/configuration.md. A documentation index is available at /llms.txt.

Configuration

This product is not supported for your selected Datadog site. ().

Overview

The pipelines page with a source going to two processors groups and two destinations

Observability Pipelines lets you collect and process logs and metrics (PREVIEW indicates an early access version of a major product or feature that you can opt into before its official release.Glossary) within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:

  • Source: Receives data from a tool like the Datadog Agent.
  • Processors: Transform, enrich, or filter data.
  • Destinations: Where data is sent (for example, Datadog, Amazon S3, Splunk, Google Security Operations, and Microsoft Sentinel).

Build and deploy pipelines to collect, transform, and route your data using one of these methods:

Pipeline types

There are two types of pipelines:

Use one of the logs templates to create a log pipeline.

  • Archive Logs
  • Dual Ship Logs
  • Generate Log-based Metrics
  • Log Enrichment
  • Log Volume Control
  • Sensitive Data Redaction
  • Split Logs

See Set Up Pipelines for more information on setting up a source, processors, and destinations.

Metric Tag Governance is in Preview. Fill out the form to request access.

Use the Metric Tag Governance template to create a metrics pipeline.

See Set Up Pipelines for more information on setting up a source, processors and destination.

Metrics data

Metrics sent to Observability Pipelines include the following:

  • name: The metric name.
  • kind: There are two kinds of metrics:
    • absolute metrics: Represents the current value of a measurement at the time it is reported.
    • incremental metrics: Represents the change in a measurement since the last reported value, which the system aggregates over time.
  • value: The metric type:
    • counter
    • gauge
    • distribution
    • histogram
  • timestamp: The date and time the metric is created.
  • tags: Includes tags such as host.

The counter metric type is the only incremental metric. gauge, distribution, and histogram metric types are absolute metrics.

An example of a metric:

{
  "name":"datadog.agent.retry_queue_duration.bytes_per_sec",
  "tags":{
    "agent":"core",
    "domain":"https://7-72-3-app.agent.datadoghq.com",
    "host":"COMP-YGVQDJG75L",
    "source_type_name":"System",
    "env:prod"
  },
  "timestamp":"2025-11-28T13:03:09Z",
  "kind":"absolute",
  "gauge":{"value":454.1372767857143}
}

Metric types

The available metric types:

Metric typeDescriptionExample
COUNTERThe total number of event occurrences in one time interval. Can be reset to zero, but cannot be decreased.You want to count the number of logs with status:error.
GAUGEA snapshot of a value at the time it is reported.You want to track the latest CPU utilization for each host.
HISTOGRAMStatistical aggregations (avg, min, max, count, median, percentiles) computed per host by the Datadog Agent in one time interval, then sent to Datadog.You want per-host request latency aggregations from each web server.
DISTRIBUTIONRaw values sent to Datadog so percentile aggregations are computed server-side, globally across every host reporting the metric in one time interval.You want the global p95 latency of an API endpoint, calculated across every host serving it.

See Metric Types for more information.

Further reading