A Processor executes within a Pipeline to complete a data-structuring action (Remapping an attribute, Grok parsing, etc.) on a log.
Note: Structured logs should be shipped in a valid format. If the structure contains invalid characters for parsing, these should be stripped at the Agent level using the mask_sequences feature.
As a best practice, it is recommended to use at most 20 Processors per Pipeline.
Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section. As a best practice, it is recommended to use at most 10 parsing rules within a grok Processor.
Define the Grok processor in the Datadog Log configuration page:
Up to five samples can be saved with the processor, and each sample can be up to 5000 characters in length. All samples show a status (match
or no match
), which highlights if one of the parsing rules of the grok parser matches the sample. Select a sample by clicking on it to trigger its evaluation against the parsing rule and display the result at the bottom of the screen.
Click Parse my logs to kickstart a set of three parsing rules for the logs flowing through the underlying pipeline. Fine tune attribute naming from there, and add new rules for other type of logs if needed. This feature requires that the corresponding logs are being indexed, and actually flowing in—you can temporarily deactivate or sample down exclusion filters to make this work for you.
Use the Datadog Log Pipeline API endpoint with the following Grok parser JSON payload:
{
"type": "grok-parser",
"name": "Parsing Log message",
"is_enabled": true,
"source": "message",
"samples": ["sample log 1", "sample log 2"],
"grok": {"support_rules": "<SUPPORT_RULES>", "match_rules": "<MATCH_RULES>"}
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false . |
source | String | yes | Name of the log attribute to parse, default: message . |
samples | Array of Strings | no | List of (up to 5) sample logs for this grok parser. |
grok.support_rules | String | yes | List of Support rules for your grok parser. |
grok.match_rules | String | yes | List of Match rules for your grok parser. |
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes:
timestamp
date
_timestamp
Timestamp
eventTime
published_date
If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp:
Note:
Define the Log Date remapper processor in the Datadog Log configuration page:
Use the Datadog Log Pipeline API endpoint with the following Log Date Remapper JSON payload:
{
"type": "date-remapper",
"name": "Define <SOURCE_ATTRIBUTE> as the official Date of the log",
"is_enabled": false,
"sources": ["<SOURCE_ATTRIBUTE_1>"]
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
sources | Array of Strings | yes | Array of source attributes. |
Use this Processor if you want to assign some attributes as the official status. For example, it can transform this log:
Into this log:
Each incoming status value is mapped as follows:
emerg
—map to error (3)Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.
Define the Log status remapper processor in the Datadog Log configuration page:
Use the Datadog Log Pipeline API endpoint with the following Log Status Remapper JSON payload:
{
"type": "status-remapper",
"name": "Define <SOURCE_ATTRIBUTE> as the official status of the log",
"is_enabled": true,
"sources": ["<SOURCE_ATTRIBUTE>"]
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
sources | Array of Strings | yes | Array of source attributes. |
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.
Define the Log Service remapper processor in the Datadog Log configuration page:
Use the Datadog Log Pipeline API endpoint with the following Log Service Remapper JSON payload:
{
"type": "service-remapper",
"name": "Define <SOURCE_ATTRIBUTE> as the official log service",
"is_enabled": true,
"sources": ["<SOURCE_ATTRIBUTE>"]
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
sources | Array of Strings | yes | Array of source attributes. |
The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.
Define the Log Message remapper processor in the Datadog Log configuration page:
Use the Datadog Log Pipeline API endpoint with the following Log Message Remapper JSON payload:
{
"type": "message-remapper",
"name": "Define <SOURCE_ATTRIBUTE> as the official message of the log",
"is_enabled": true,
"sources": ["msg"]
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
sources | Array of Strings | yes | Array of source attributes, default: msg |
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. It can transforms this log:
Into this log:
Constraints on the tag/attribute name are explained in the Tagging documentation. Some additional constraints are applied as :
or ,
are not allowed in the target tag/attribute name.
If the target of the remapper is an attribute, the remapper can also try to cast the value to a new type (String
, Integer
or Double
). If the cast is not possible, the original type is kept (note: The decimal separator for Double
need to be .
)
Define the remapper processor in the Datadog Log configuration page. For example, here, it remaps user
to user.firstname
Use the Datadog Log Pipeline API endpoint with the following Remapper JSON payload:
{
"type": "attribute-remapper",
"name": "Remap <SOURCE_ATTRIBUTE> to <TARGET_ATTRIBUTE>",
"is_enabled": true,
"source_type": "attribute",
"sources": ["<SOURCE_ATTRIBUTE>"],
"target": "<TARGET_ATTRIBUTE>",
"target_type": "tag",
"target_format": "integer",
"preserve_source": false,
"override_on_conflict": false
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
source_type | String | no | Defines if the sources are from log attribute or tag , default: attribute |
sources | Array of Strings | yes | Array of source attributes or tags |
target | String | yes | Final attribute or tag name to remap the sources to. |
target_type | String | no | Defines if the target is a log attribute or a tag , default: attribute |
target_format | String | no | Defines if the attribute value should be cast to another type. possible value: auto , string , long or integer , default: auto . When set to auto , no cast is applied. |
preserve_source | Boolean | no | Remove or preserve the remapped source element, default: false |
override_on_conflict | Boolean | no | Override or not the target element if already set, default: false |
This Processor extracts query parameters and other important parameters from a URL. When setup, the following attributes are produced:
Define the URL parser processor in the Datadog Log configuration page:
{
"type": "url-parser",
"name": "Parse the URL from http.url attribute.",
"is_enabled": true,
"sources": ["http.url"],
"target": "http.url_details"
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
sources | Array of Strings | no | Array of source attributes, default: http.url |
target | String | yes | Name of the parent attribute that contains all the extracted details from the sources , default: http.url_details |
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing. When set up, the following attributes are produced:
Note: If your logs contain encoded User-Agents (for example, IIS logs), configure this Processor to decode the URL before parsing it.
Define the User-Agent processor in the Datadog Log configuration page:
Use the Datadog Log Pipeline API endpoint with the following User-Agent parser JSON payload:
{
"type": "user-agent-parser",
"name": "Parses <SOURCE_ATTRIBUTE> to extract all its User-Agent information",
"is_enabled": true,
"sources": ["http.useragent"],
"target": "http.useragent_details",
"is_encoded": false
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false . |
sources | Array of Strings | no | Array of source attributes, default: http.useragent . |
target | String | yes | Name of the parent attribute that contains all the extracted details from the sources , default: http.useragent_details . |
is_encoded | Boolean | no | Define if the source attribute is url encoded or not, default: false . |
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view (for example, URL groups, machine groups, environments, and response time buckets).
Note:
Define the Category Processor in the Datadog Log configuration page. For example, to categorize your web access logs based on the status code range value (“OK” for a response code between 200 and 299, “Notice” for a response code between 300 and 399, …) add this Processor:
It produces the following result:
Use the Datadog Log Pipeline API endpoint with the following Category processor JSON payload:
{
"type": "category-processor",
"name": "Assign a custom value to the <TARGET_ATTRIBUTE> attribute",
"is_enabled": true,
"categories": [
{"filter": {"query": "<QUERY_1>"}, "name": "<VALUE_TO_ASSIGN_1>"},
{"filter": {"query": "<QUERY_2>"}, "name": "<VALUE_TO_ASSIGN_2>"}
],
"target": "<TARGET_ATTRIBUTE>"
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
categories | Array of Object | yes | Array of filters to match or not a log and their corresponding name to assign a custom value to the log. |
target | String | yes | Name of the target attribute which value is defined by the matching category. |
Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators: -
, +
, *
, /
.
By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.
Notes:
-
needs to be space split in the formula as it can also be contained in attribute names.0.1234567891
, the actual value stored for the attribute is 0.123456789
.Define the Arithmetic Processor in the Datadog Log configuration page:
Use the Datadog Log Pipeline API endpoint with the following Arithmetic processor JSON payload:
{
"type": "arithmetic-processor",
"name": "<PROCESSOR_NAME>",
"is_enabled": true,
"expression": "<ARITHMETIC_OPERATION>",
"target": "<TARGET_ATTRIBUTE>",
"is_replace_missing": false
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false . |
expression | String | yes | Arithmetic operation between one or more log attributes. |
target | String | yes | Name of the attribute that contains the result of the arithmetic operation. |
is_replace_missing | Boolean | no | If true , it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing. Default: false . |
Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax: %{attribute_path}
.
Notes:
Define the string builder processor on the Datadog log configuration page:
Example
With the following log:
{
"http": {
"method": "GET",
"status_code": 200,
"url": "https://app.datadoghq.com/users"
},
"array_ids": [123, 456, 789],
"array_users": [
{"first_name": "John", "last_name": "Doe"},
{"first_name": "Jack", "last_name": "London"}
]
}
You can use the template: Request %{http.method} %{http.url} was answered with response %{http.status_code}
, which returns the result:
Request GET https://app.datadoghq.com/users was answered with response 200
Objects:
In the example log http
is an object and cannot be used in a block (%{http}
fails), whereas %{http.method}
, %{http.status_code}
, or %{http.url}
returns the corresponding value.
Arrays:
Blocks can be used on arrays of values or on a specific attribute within an array. For the example log, adding the block %{array_ids}
returns:
123,456,789
Whereas %{array_users}
does not return anything because it is a list of objects.
However, %{array_users.first_name}
returns a list of first_name
contained in the array:
John,Jack
Use the Datadog Log Pipeline API endpoint with the following string builder processor JSON payload:
{
"type": "string-builder-processor",
"name": "<PROCESSOR_NAME>",
"is_enabled": true,
"template": "<STRING_BUILDER_TEMPLATE>",
"target": "<TARGET_ATTRIBUTE>",
"is_replace_missing": true
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | Yes | Type of the processor. |
name | String | No | Name of the processor. |
is_enabled | Boolean | No | If the processor is enabled or not, defaults to false . |
template | String | Yes | A formula with one or more attributes and raw text. |
target | String | Yes | The name of the attribute that contains the result of the template. |
is_replace_missing | Boolean | No | If true , it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes. |
The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.
Most elements contains a name
and iso_code
(or code
for continent) attribute. subdivision
is the first level of subdivision that the country uses such as “States” for the United States or “Departments” for France.
Find below an example of the GeoIP Parser that extracts gelocation from the network.client.ip
attribute and stores it into the network.client.geoip
attribute:
Note: This processor uses GeoLite2 data created by MaxMind.
Use the Datadog Log Pipeline API endpoint with the following Geo-IP parser JSON payload:
{
"type": "geo-ip-parser",
"name": "Parse the geolocation elements from network.client.ip attribute.",
"is_enabled": true,
"sources": ["network.client.ip"],
"target": "network.client.geoip"
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false |
sources | Array of Strings | no | Array of source attributes, default: network.client.ip |
target | String | yes | Name of the parent attribute that contains all the extracted details from the sources , default: network.client.geoip |
Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in an Enrichment Table (beta) or the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.
The processor performs the following actions:
You can fill the mapping table by selecting an enrichment table or manually by entering a list of source_key,target_value
pairs, or uploading a CSV file.
The size limit for the mapping table is 100Kb. This limit applies across all Lookup Processors on the platform, however, Enrichment tables support larger file sizes.
Use the Datadog Log Pipeline API endpoint with the following Lookup Processor JSON payload:
{
"type": "lookup-processor",
"name": "<PROCESSOR_NAME>",
"is_enabled": true,
"source": "<SOURCE_ATTRIBUTE>",
"target": "<TARGET_ATTRIBUTE>",
"lookup_table": ["key1,value1", "key2,value2"],
"default_lookup": "<DEFAULT_TARGET_VALUE>"
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | yes | If the processor is enabled or not. Default: false |
source | String | yes | Source attribute used to perform the lookup. |
target | String | yes | Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list. |
lookup_table | Array of strings | yes | Mapping table of values for the source attribute and their associated target attribute values, formatted as [ “source_key1,target_value1”, “source_key2,target_value2” ] |
default_lookup | String | no | Value to set the target attribute if the source value is not found in the list. |
There are two ways to improve correlation between application traces and logs:
Follow the documentation on how to inject a Trace ID in the application logs and by default log integrations take care of all the rest of the setup.
Use the Trace remapper processor to define a log attribute as its associated trace ID.
Define the Trace remapper processor in the Datadog Log configuration page. Enter the Trace ID attribute path in the Processor tile as follows:
Use the Datadog Log Pipeline API endpoint with the following Trace remapper JSON payload:
{
"type": "trace-id-remapper",
"name": "Define dd.trace_id as the official trace id associate to this log",
"is_enabled": true,
"sources": ["dd.trace_id"]
}
Parameter | Type | Required | Description |
---|---|---|---|
type | String | yes | Type of the processor. |
name | String | no | Name of the processor. |
is_enabled | Boolean | no | If the processors is enabled or not, default: false . |
sources | Array of Strings | no | Array of source attributes, default: dd.trace_id . |
Additional helpful documentation, links, and articles: