Overview
Use the category processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Then, use categories to create groups for an analytical view (for example, URL groups, machine groups, environments, and response time buckets).
Notes:
- The syntax of the query is the one in the Log Explorer search bar. This query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
- Once the log has matched one of the processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
- The names of the categories must be unique.
- Once defined in the category processor, you can map categories to log status using the log status remapper.
Use cases
Use the Category Processor to categorize your log events based on their content. Common use cases are:
- Categorizing success and failure logs, for example, Auth0 logs.
- Categorizing status of logs based on the log content, for example, IIS logs.
Before and after state of logs
Before:
{
"http": {
"method": "GET",
"url": "/v1/orders",
"status_code": 500
},
"message": "Order processing failed: timeout while contacting payment provider",
"timestamp": 1696945536000
}
Category Processor
Create a Category Processor to set a log status category based on http.status_code, then use a Log Status Remapper to apply it as the official log status.
After processing:
{
"http": {
"method": "GET",
"url": "/v1/orders",
"status_code": 500
},
"message": "Order processing failed: timeout while contacting payment provider",
"log_status": "error",
"timestamp": 1696945536000
}
The Category Processor added a new attribute called log_status with the status of the log derived from the HTTP status code.
API
Use the Datadog Log Pipeline API endpoint with the following category processor JSON payload:
{
"type": "category-processor",
"name": "Assign a custom value to the <TARGET_ATTRIBUTE> attribute",
"is_enabled": true,
"categories": [
{"filter": {"query": "<QUERY_1>"}, "name": "<VALUE_TO_ASSIGN_1>"},
{"filter": {"query": "<QUERY_2>"}, "name": "<VALUE_TO_ASSIGN_2>"}
],
"target": "<TARGET_ATTRIBUTE>"
}
| Parameter | Type | Required | Description |
|---|
type | String | Yes | Type of the processor. |
name | String | No | Name of the processor. |
is_enabled | Boolean | No | If the processor is enabled or not. Default: false. |
categories | Array of Object | Yes | Array of filters to match or not a log and their corresponding name to assign a custom value to the log. |
target | String | Yes | Name of the target attribute which value is defined by the matching category. |
Further reading
Additional helpful documentation, links, and articles:
