Observability Pipelines

A graphic showing different data sources on the left that flows into three hexagons named transform, reduce, and route, with arrows pointing to different destinations for the modified data

What is Observability Pipelines?

Observability Pipelines is a monitoring solution built on Vector, an open source tool that enables you to monitor and manage all of your telemetry pipelines at scale. Vector is deployed as an aggregator within your infrastructure to collect, transform, and route all of your logs, metrics, and traces to any destination.

Add your Datadog API key to your Vector configuration to connect it to Observability Pipelines. Use Observability Pipelines to monitor your Vector pipelines and identify bottlenecks and latencies, fine-tune performance, monitor data delivery, and more.

With Observability Pipelines, you can also:

  • Control your data volume before routing to manage costs.
  • Route data anywhere to reduce vendor lock-in and simplify migrations.
  • Meet residency requirements and redact sensitive data to stay more compliant.
  • Enrich, structure, and transform your events to make them more useful.

Build performant and reliable data pipelines with complete visibility and simplified management using Observability Pipelines.

Get started

  1. Install Vector using the quick start method, your preferred package manager, or based on your specific platform or operating system.
  2. Set up Vector configurations to collect, transform and route your data.
  3. Connect Vector to Observability Pipelines with your Datadog API.

Explore Observability Pipelines

Now that you are sending configuration data to Observability Pipelines, start getting insights into your Vector pipelines:

Monitor the health of your Vector pipelines

Get a holistic view of all of your pipelines’ topologies and monitor key performance indicators, such as average load, error rate, and throughput for each of your flows.

The configuration map showing data coming from http, splunk_hec, and datadog, and flowing into different transforms and then sent to different destinations

Quickly identify bottlenecks and optimize performance

Dive into specific Vector components to understand how observability data is flowing into your pipeline to troubleshoot and pinpoint performance bottlenecks and to optimize your pipeline.

The S3 source configuration side panel showing graphs for events in and out per second, percentage of errors, and load average percentage

Ensure data delivery and reduce latency.

Find out if data is reaching its destination and get full visibility into any latency issues to meet SLIs and SLOs.

The Observability Pipelines page showing a list of active and inactive pipelines with columns for created date, number of hosts, version, events in, bytes in, and error rate

Further Reading