Overview

Observability Pipelines’ processors enable you to add, edit, and remove log fields. Remapping attributes or rewriting values ensures your logs are processed and standardized properly. For the majority of processing use cases, use the Edit Fields processor to add, remap, or remove fields from your logs. For advanced use cases, use the Custom Processor to conditionally modify fields or rewrite a field’s value.

In Datadog, reserved attributes are log fields that are set aside for specific processing in the platform. Reserved attributes include host, source, status, service, trace_id, and message. Reserved attributes are applied when routing logs to the following Observability Pipelines destinations:

• Datadog Logs
• Amazon S3 (for Log Archives)
• Azure Blob Storage (for Log Archives)
• Google Cloud Storage (for Log Archives)

There are restrictions in Observability Pipelines on how you can modify reserved attributes. For example, reserved attributes cannot be renamed using the Rename Field processor, but must be remapped instead. This guide walks you through the steps to remap the value of reserved attributes.

Remap the value of reserved attributes

To change or override the value of an existing reserved attribute field, Datadog recommends two approaches using Observability Pipelines. The first uses the Edit Fields processor, and the second uses the Custom Processor.

Use an Edit Fields processor for basic field assignments

1. Use a Remove field processor to drop the reserved attribute from the log.
2. Use an Add field processor to add the reserved attribute back to the log with your correct field name and value assignment.

Note: In terms of the processor order, the Add Field processor should go immediately after the Remove Field processor to ensure correct field remapping.

Example

The Remove field processor image below removes the improperly named service field from the log.

The Add field processor image below re-adds the service field back with the correct value.

Use the Custom Processor for dynamic or manual assignments

Use the Custom Processor to rewrite the reserved attribute’s value.

Dynamically assign the value using template syntax to reference another field’s value.

The following Custom Processor script rewrites the service field and dynamically assigns the value of app_id to the service field’s value.

.service = {{.app_id}}

In the below example image, the input shows service with the value wrongstatus. After processing the log with the script, the output shows service with the value of streaming-service, which is app_id’s value.

Manually rewrite the value of an attribute with a static name

The following Custom Processor script sets the status field to the static value info.

.status = "info"

In the below example image, the input shows status with the value wrongstatus. After processing the log with the script, the output shows status with info as assigned.

Further reading

Additional helpful documentation, links, and articles:

Learn more about the Edit Fields processorDOCUMENTATION Learn more about the Custom Processor processorDOCUMENTATION