Best Practices for OPW Aggregator Architecture

Overview

The Observability Pipelines Worker’s (OPW) aggregator architecture deploys the Observability Pipelines Worker as a standalone service for centralized data processing and routing:

Deploy Observability Pipelines Worker into your infrastructure, like any other service to intercept and manipulate data, and then forward it to your destinations. Each Observability Pipelines Worker instance operates independently, so that you can scale the architecture with a simple load balancer.

This guide walks you through the recommended aggregator architecture for new Observability Pipelines Worker users. Specifically, these topics include:

• Optimizing the instance so you can horizontally scale the Observability Pipelines Worker aggregator.
• Starting points to estimate your resource capacity for capacity planning and scaling the Observability Pipelines Worker.
• Determining your network topology and configurations for the Observability Pipelines Worker.
• Achieving high durability and high availability.
• Using the Observability Pipelines Worker as part of your disaster recovery.
• More advanced configurations for deploying multiple aggregators, publish-subscribe systems, and global aggregation.