Processors

Docs > Log Management > Log Configuration > Processors

Overview

The processors outlined in this documentation are specific to cloud-based logging environments. To parse, structure, and enrich on-premises logs, see Observability Pipelines.

A processor executes within a Pipeline to complete a data-structuring action and generate attributes to enrich your logs.

Processors

In log configuration settings, you can configure processors such as the Grok parser or date remapper to help extract, create, and remap attributes to enrich your logs and enhance faceted search.

Notes:

Structured logs should be shipped in a valid format. If the structure contains invalid characters for parsing, these should be stripped at the Agent level using the mask_sequences feature.
As a best practice, it is recommended to use at most 20 processors per pipeline.

Processor types

Select a processor type to learn more:

Arithmetic Processor: Add a new attribute to a log with the result of a formula applied to existing numeric attributes.

Array Processor: Extract, aggregate, or transform values from JSON arrays in your logs.

Attribute Remapper: Remap source attributes or tags to another target attribute or tag.

Category Processor: Add a new attribute to a log based on a search query match, for grouping and classification.

Decoder Processor: Translate binary-to-text encoded fields (such as Base64 or Hex) into their original representation.

GeoIP Parser: Extract continent, country, subdivision, or city information from an IP address attribute.

Grok Parser: Create custom parsing rules to extract and structure data from log messages or specific attributes.

Log Date Remapper: Assign one or more attributes as the official timestamp for your logs.

Log Message Remapper: Assign one or more attributes as the official message for your logs.

Log Status Remapper: Assign one or more attributes as the official severity status for your logs.

Lookup Processor: Map a log attribute to a human-readable value from a Reference Table or mapping table.

OCSF Processor: Normalize security logs to the Open Cybersecurity Schema Framework (OCSF).

Service Remapper: Assign one or more attributes as the official service for your logs.

Span Remapper: Define a correlation between application spans and logs.

String Builder Processor: Build a new attribute from a template of existing attributes and raw strings.

Threat Intel Processor: Enrich logs with Threat Intelligence attributes by matching against an indicator of compromise (IoC) table.

Trace Remapper: Define a correlation between application traces and logs.

URL Parser: Extract query parameters and other components from a URL attribute.

User-Agent Parser: Parse a user-agent attribute to extract OS, browser, device, and other user data.

Further reading

Additional helpful documentation, links, and articles:

Discover Datadog PipelinesDOCUMENTATION

Logging without Limits*DOCUMENTATION

Learn how to explore your logsDOCUMENTATION

Tips and tricks: Add business data to logs from retail endpointsVIDEO

Build and Manage Log PipelinesLEARNING CENTER

Process Logs Out of the Box with Integration PipelinesLEARNING CENTER