Overview

When setting up a pipeline to send logs from a specific source to Observability Pipelines, you often need to decide how to process and manage those logs.

Questions such as the following might come up:

• Which logs from this source are important?
• Which logs can safely be dropped?
• Should repetitive logs be sampled?
• Which fields should be parsed or formatted for the destination?

Making these decisions typically requires coordination across multiple teams and detailed knowledge of each log source.

Observability Pipelines Packs provide predefined configurations to help you make these decisions quickly and consistently. Packs apply Datadog-recommended best practices for specific log sources such as Akamai, AWS CloudTrail, Cloudflare, Fastly, Palo Alto Firewall, and Zscaler.

What Packs do

Each Pack includes source-specific configurations that define:

• Fields that can safely be removed to reduce payload size
• Logs that can be dropped, such as duplicate events or health checks
• Logs that should be retained or parsed, such as errors or security detections
• Formatting and normalization rules to align logs across different destinations and environments

By using Packs, you can apply consistent parsing, filtering, and routing logic for each log source without creating configurations manually.

Why use Packs

Packs help teams:

• Reduce ingestion volume and costs by filtering or sampling repetitive, low-value events
• Maintain consistency in parsing and field mapping across environments and destinations
• Accelerate setup by applying ready-to-use configurations for common sources

Setup

To set up packs:

1. Navigate to the Pipelines page.
2. Click Packs.
3. Click the pack you want to set up.
4. You can either create a new pipeline from the pack or add the pack to an existing pipeline.
   • If you clicked Add to New Pipeline, in the new pipeline that was created:
     • Click the processor group that was added to see the individual processors that the pack added and edit them as needed. See Processors for more information.
     • See Set Up Pipelines for information on setting up the rest of the pipeline.
   • If you clicked Add to Existing Pipeline:
     1. Select the pipeline you want to add the pack to.
     2. Click Add to Existing Pipeline.
        1. The pack is added to the last processor group in your pipeline.
        2. Click on the group to review the individual processors and edit them as needed. See Processors for more information.

Available packs

Search or filter the table by source or destination. Select a pack name for details.

Further reading

Additional helpful documentation, links, and articles:

Rehydrate archived logs in any SIEM or logging vendor with Observability PipelinesBLOG