Observability Pipelines
What is Observability Pipelines and the Observability Pipelines Worker?
Observability Pipelines Worker
Observability Pipelines Worker is an on-premise end-to-end data pipeline solution designed to collect, process, and route logs and metrics from any source to any destination. You can deploy Observability Pipelines as an aggregator within your infrastructure for central processing, collecting data from multiple upstream sources, and performing cross-host aggregation and analysis. With the Observability Pipelines Worker, you can also:
- Control your data volume before routing to manage costs.
- Route data anywhere to reduce vendor lock-in and simplify migrations.
- Transform logs and metrics by adding, parsing, enriching, and removing fields and tags.
- Redact sensitive data from your telemetry data.
Observability Pipelines
Using Datadog, you can monitor, build, and manage all of your Observability Pipelines Worker deployments at scale.
Add your Datadog API key to your Observability Pipelines configuration to monitor your pipelines in Datadog: Identify bottlenecks and latencies, fine-tune performance, monitor data delivery, and more.
Get started
- Install the Observability Pipelines Worker.
- Set up configurations to collect, transform and route your data.
Explore Observability Pipelines
Start exploring and getting insights into your Observability Pipelines:
Monitor the health of your pipelines
Get a holistic view of all of your pipelines’ topologies and monitor key performance indicators, such as average load, error rate, and throughput for each of your flows.
Dive into specific configuration components to understand how observability data is flowing into your pipeline to troubleshoot and pinpoint performance bottlenecks and to optimize your pipeline.
Ensure data delivery and reduce latency.
Find out if data is reaching its destination and get full visibility into any latency issues to meet SLIs and SLOs.
Further Reading
Additional helpful documentation, links, and articles: