Overview

Use the Observability Pipelines Worker to send your processed logs and metrics (PreviewPREVIEW indicates an early access version of a major product or feature that you can opt into before its official release.Glossary) to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See Event batching for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See Template syntax for more information.

Select a destination in the left navigation menu to see more information about it.

Destinations

These are the available destinations:

Template syntax

Logs are often stored in separate indexes based on log data, such as the service or environment the logs are coming from or another log attribute. In Observability Pipelines, you can use template syntax to route your logs to different indexes based on specific log fields.

When the Observability Pipelines Worker cannot resolve the field with the template syntax, the Worker defaults to a specified behavior for that destination. For example, if you are using the template {{application_id}} for the Amazon S3 destination’s Prefix field, but there isn’t an application_id field in the log, the Worker creates a folder called OP_UNRESOLVED_TEMPLATE_LOGS/ and publishes the logs there.

The following table lists the destinations and fields that support template syntax, and what happens when the Worker cannot resolve the field:

Example

If you want to route logs based on the log’s application ID field (for example, application_id) to the Amazon S3 destination, use the event fields syntax in the Prefix to apply to all object keys field.

Syntax

Event fields

Use {{ <field_name> }} to access individual log event fields. For example:

{{ application_id }}

Strftime specifiers

Use strftime specifiers for the date and time. For example:

year=%Y/month=%m/day=%d

Escape characters

Prefix a character with \ to escape the character. This example escapes the event field syntax:

\{{ field_name }}

This example escapes the strftime specifiers:

year=\%Y/month=\%m/day=\%d/

Event batching

Observability Pipelines destinations send events in batches to the downstream integration. A batch of events is flushed when one of the following parameters is met:

• Maximum number of events
• Maximum number of bytes
• Timeout (seconds)

For example, if a destination’s parameters are:

• Maximum number of events = 2
• Maximum number of bytes = 100,000
• Timeout (seconds) = 5

And the destination receives 1 event in a 5-second window, it flushes the batch at the 5-second timeout.

If the destination receives 3 events within 2 seconds, it flushes a batch with 2 events and then flushes a second batch with the remaining event after 5 seconds. If the destination receives 1 event that is more than 100,000 bytes, it flushes this batch with the 1 event.

*Destination does not batch events.