Overview

Use the Observability Pipelines Worker to send your processed logs to different destinations.

Select and set up your destinations when you set up a pipeline. This is step 4 in the pipeline setup process:

1. Navigate to Observability Pipelines.
2. Select a template.
3. Select and set up your source.
4. Select and set up your destinations.
5. Set up your processors.
6. Install the Observability Pipelines Worker.

Template syntax

Logs are often stored in separate indexes based on log data, such as the service or environment the logs are coming from or another log attribute. In Observability Pipelines, you can use template syntax to route your logs to different indexes based on specific log fields.

When the Observability Pipelines Worker cannot resolve the field with the template syntax, the Worker defaults to a specified behavior for that destination. For example, if you are using the template {{application_id}} for the Amazon S3 destination’s Prefix field, but there isn’t an application_id field in the log, the Worker creates a folder called OP_UNRESOLVED_TEMPLATE_LOGS/ and publishes the logs there.

The following table lists the destinations and fields that support template syntax, and what happens when the Worker cannot resolve the field:

Example

If you want to route logs based on the log’s application ID field (for example, application_id) to the Amazon S3 destination, use the event fields syntax in the Prefix to apply to all object keys field.

Syntax

Event fields

Use {{ <field_name> }} to access individual log event fields. For example:

{{ application_id }}

Strftime specifiers

Use strftime specifiers for the date and time. For example:

year=%Y/month=%m/day=%d

Escape characters

Prefix a character with \ to escape the character. This example escapes the event field syntax:

\{{ field_name }}

This example escapes the strftime specifiers:

year=\%Y/month=\%m/day=\%d/

Event batching

Observability Pipelines destinations send events in batches to the downstream integration. A batch of events is flushed when one of the following parameters is met:

• Maximum number of events
• Maximum number of bytes
• Timeout (seconds)

For example, if a destination’s parameters are:

• Maximum number of events = 2
• Maximum number of bytes = 100,000
• Timeout (seconds) = 5

And the destination receives 1 event in a 5-second window, it flushes the batch at the 5-second timeout.

If the destination receives 3 events within 2 seconds, it flushes a batch with 2 events and then flushes a second batch with the remaining event after 5 seconds. If the destination receives 1 event that is more than 100,000 bytes, it flushes this batch with the 1 event.

Note: The rsyslog and syslog-ng destinations do not batch events.