This product is not supported for your selected Datadog site. ().

Overview

The pipelines page with a source going to two processors groups and two destinations

Observability Pipelines lets you collect and process logs and metrics (PREVIEW indicates an early access version of a major product or feature that you can opt into before its official release.Glossary) within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:

  • Source: Receives data from a tool like the Datadog Agent.
  • Processors: Transform, enrich, or filter data.
  • Destinations: Where data is sent (for example, Datadog, Amazon S3, Splunk, Google Security Operations, and Microsoft Sentinel).

Build and deploy pipelines to collect, transform, and route your data using one of these methods:

Pipeline types

There are two types of pipelines:

Use one of the logs templates to create a log pipeline.

  • Archive Logs
  • Dual Ship Logs
  • Generate Log-based Metrics
  • Log Enrichment
  • Log Volume Control
  • Sensitive Data Redaction
  • Split Logs

See Set Up Pipelines for more information on setting up a source, processors, and destinations.

Metric Tag Governance is in Preview. Fill out the form to request access.

Use the Metric Tag Governance template to create a metrics pipeline.

See Set Up Pipelines for more information on setting up a source, processors and destination.

Metrics data

Metrics sent to Observability Pipelines include the following:

  • name: The metric name.
  • kind: There are two kinds of metrics:
    • absolute metrics: Represents the current value of a measurement at the time it is reported.
    • incremental metrics: Represents the change in a measurement since the last reported value, which the system aggregates over time.
  • value: The metric type:
    • counter
    • gauge
    • distribution
    • histogram
  • timestamp: The date and time the metric is created.
  • tags: Includes tags such as host.

The counter metric type is the only incremental metric. gauge, distribution, and histogram metric types are absolute metrics.

An example of a metric:

{
  "name":"datadog.agent.retry_queue_duration.bytes_per_sec",
  "tags":{
    "agent":"core",
    "domain":"https://7-72-3-app.agent.datadoghq.com",
    "host":"COMP-YGVQDJG75L",
    "source_type_name":"System",
    "env:prod"
  },
  "timestamp":"2025-11-28T13:03:09Z",
  "kind":"absolute",
  "gauge":{"value":454.1372767857143}
}

Metric types

The available metric types:

Metric typeDescriptionExample
COUNTERRepresents the total number of event occurrences in one time interval. This value can be reset to zero, but cannot be decreased.You want to count the number of logs with status:error.
GAUGERepresents a snapshot of events in one time interval.You want to measure the latest CPU utilization per host for all logs in the production environment.
DISTRIBUTIONRepresent the global statistical distribution of a set of values calculated across your entire distributed infrastructure in one time interval.You want to measure the average time it takes for an API call to be made.
HISTOGRAMRepresents the statistical distribution of a set of values calculated in one time interval.You want to measure response time distributions for a service or endpoint.

See Metric Types for more information.

Further reading