Grok Parser

Docs > Log Management > Log Configuration > Processors > Grok Parser

Overview

Create custom grok rules to parse the full message or a specific attribute of your raw event. As a best practice, limit your grok parser to 10 parsing rules. For more information on Grok syntax and parsing rules, see Parsing.

Use cases

The grok parser is mainly used to parse attributes from the message of your log. For example, NGINX logs have a message containing multiple pieces of information you might want to extract.

After creating a grok rule, the parser can write the IP address, user, request timestamp, request method, URL, version, status code, and bytes.

Setup

Define the Grok processor on the Pipelines page. To configure Grok parsing rules:

Expand a pipeline and click Add… > Grok Parser.
Enter a name in Name the processor.
Under Define parsing rules > Log samples, review the log samples, or click Add a new sample to add your own (up to 10 total, 5000 characters each).
Note: Datadog pulls the sample logs from the five highest-volume log patterns matching your pipeline filter.
Under Define parsing rules > Parsing rules, click Tap to Parse to generate rules that match your samples. You can also write your own rules. For Grok syntax, see Parsing.
Tap to Parse does not consume AI Credits.
Tap to Parse is not available for your selected Datadog site ().
(Optional) Expand Advanced Settings to parse a specific attribute instead of the default message attribute, or define helper rules. See Advanced settings.
Select a sample to evaluate it against your parsing rules and display the result in the Preview parsing panel.
Each sample displays one of the following results:
- Match: A rule matches the sample and extracts fields.
- Match with a warning: A rule matches the sample but extracts no fields.
- No Match: No rule matches the sample.
Click Create.

Before and after state of logs

Example: Parsing nginx access logs

Before (raw log):

192.168.1.1 - john [10/Oct/2023:13:55:36 +0000] "GET /api/users HTTP/1.1" 200 1234

Grok parsing rule:

access.common %{ipOrHost:network.client.ip} %{notSpace:http.ident} %{notSpace:http.auth} \[%{httpdate:date}\] "(?>%{word:http.method} |)%{notSpace:http.url}(?: HTTP/%{number:http.version}|)" %{number:http.status_code} (?>%{number:network.bytes_written}|-)

After processing:

{
 "network": {
   "client": {
     "ip": "192.168.1.1"
   },
   "bytes_written": 1234
 },
 "http": {
   "ident": "-",
   "auth": "john",
   "method": "GET",
   "url": "/api/users",
   "version": "1.1",
   "status_code": 200
 },
 "date": 1696945536000
}

The Grok Parser transforms unstructured log messages into structured JSON attributes that can be queried, filtered, and analyzed in the Log Explorer.

API

Use the Datadog Log Pipeline API endpoint with the following Grok parser JSON payload:

{
  "type": "grok-parser",
  "name": "Parsing Log message",
  "is_enabled": true,
  "source": "message",
  "samples": ["sample log 1", "sample log 2"],
  "grok": {"support_rules": "<SUPPORT_RULES>", "match_rules": "<MATCH_RULES>"}
}

Parameter	Type	Required	Description
`type`	String	Yes	Type of the processor.
`name`	String	No	Name of the processor.
`is_enabled`	Boolean	No	If the processor is enabled or not. Default: `false`.
`source`	String	Yes	Name of the log attribute to parse. Default: `message`.
`samples`	Array of strings	No	List of (up to 5) sample logs for this grok parser.
`grok.support_rules`	String	Yes	List of Support rules for your grok parser.
`grok.match_rules`	String	Yes	List of Match rules for your grok parser.