The reduce processor groups multiple log events into a single log, based on the fields specified and the merge strategies selected. Logs are grouped at 10-second intervals. After the interval has elapsed for the group, the reduced log for that group is sent to the next step in the pipeline.
To set up the reduce processor:
- Define a filter query. Only logs that match the specified filter query are processed. Reduced logs and logs that do not match the filter query are sent to the next step in the pipeline.
- In the Group By section, enter the field you want to group the logs by.
- Click Add Group by Field to add additional fields.
- In the Merge Strategy section:
- In On Field, enter the name of the field you want to merge the logs on.
- Select the merge strategy in the Apply dropdown menu. This is the strategy used to combine events. See the following Merge strategies section for descriptions of the available strategies.
- Click Add Merge Strategy to add additional strategies.
Merge strategies
These are the available merge strategies for combining log events.
| Name | Description |
|---|
| Array | Appends each value to an array. |
| Concat | Concatenates each string value, delimited with a space. |
| Concat newline | Concatenates each string value, delimited with a newline. |
| Concat raw | Concatenates each string value, without a delimiter. |
| Discard | Discards all values except the first value that was received. |
| Flat unique | Creates a flattened array of all unique values that were received. |
| Longest array | Keeps the longest array that was received. |
| Max | Keeps the maximum numeric value that was received. |
| Min | Keeps the minimum numeric value that was received. |
| Retain | Discards all values except the last value that was received. Works as a way to coalesce by not retaining `null`. |
| Shortest array | Keeps the shortest array that was received. |
| Sum | Sums all numeric values that were received. |
Filter query syntax
Each processor has a corresponding filter query in their fields. Processors only process logs that match their filter query. And for all processors except the filter processor, logs that do not match the query are sent to the next step of the pipeline. For the filter processor, logs that do not match the query are dropped.
For any attribute, tag, or key:value pair that is not a reserved attribute, your query must start with @. Conversely, to filter reserved attributes, you do not need to append @ in front of your filter query.
For example, to filter out and drop status:info logs, your filter can be set as NOT (status:info). To filter out and drop system-status:info, your filter must be set as NOT (@system-status:info).
Filter query examples:
NOT (status:debug): This filters for only logs that do not have the status DEBUG.status:ok service:flask-web-app: This filters for all logs with the status OK from your flask-web-app service.- This query can also be written as:
status:ok AND service:flask-web-app.
host:COMP-A9JNGYK OR host:COMP-J58KAS: This filter query only matches logs from the labeled hosts.@user.status:inactive: This filters for logs with the status inactive nested under the user attribute.
Learn more about writing filter queries in Datadog’s Log Search Syntax.
