Overview

Create custom grok rules to parse the full message or a specific attribute of your raw event. As a best practice, limit your grok parser to 10 parsing rules. For more information on Grok syntax and parsing rules, see Parsing.

Use cases

The grok parser is mainly used to parse attributes from the message of your log. For example, NGINX logs have a message containing multiple pieces of information you might want to extract.

After creating a grok rule, the parser can write the IP address, user, request timestamp, request method, URL, version, status code, and bytes.

Setup

Define the Grok processor on the Pipelines page. To configure Grok parsing rules:

1. Click Parse my logs to automatically generate a set of three parsing rules based on the logs flowing through the pipeline. Note: This feature requires that the corresponding logs are indexed and actively flowing in. You can temporarily deactivate or sample down exclusion filters to allow the feature to detect logs.
2. Log Samples: Add up to five sample logs (up to 5000 characters each) to test your parsing rules.
3. Define parsing rules: Write your parsing rules in the rule editor. As you define rules, the Grok parser provides syntax assistance:
   • Matcher suggestions: Type a rule name followed by %{. A dropdown appears with available matchers (such as word, integer, ip, date). Select a matcher from the list to insert it into your rule.
     

     MyParsingRule %{
     

   • Filter suggestions: When adding a filter with :, a dropdown shows compatible filters for the selected matcher.
4. Test your rules: Select a sample by clicking on it to trigger its evaluation against the parsing rule and display the result at the bottom of the screen. All samples show a status (match or no match), which highlights if one of the parsing rules of the grok parser matches the sample.

Before and after state of logs

192.168.1.1 - john [10/Oct/2023:13:55:36 +0000] "GET /api/users HTTP/1.1" 200 1234

access.common %{ipOrHost:network.client.ip} %{notSpace:http.ident} %{notSpace:http.auth} \[%{httpdate:date}\] "(?>%{word:http.method} |)%{notSpace:http.url}(?: HTTP/%{number:http.version}|)" %{number:http.status_code} (?>%{number:network.bytes_written}|-)

{
 "network": {
   "client": {
     "ip": "192.168.1.1"
   },
   "bytes_written": 1234
 },
 "http": {
   "ident": "-",
   "auth": "john",
   "method": "GET",
   "url": "/api/users",
   "version": "1.1",
   "status_code": 200
 },
 "date": 1696945536000
}

API

Use the Datadog Log Pipeline API endpoint with the following Grok parser JSON payload:

{
  "type": "grok-parser",
  "name": "Parsing Log message",
  "is_enabled": true,
  "source": "message",
  "samples": ["sample log 1", "sample log 2"],
  "grok": {"support_rules": "<SUPPORT_RULES>", "match_rules": "<MATCH_RULES>"}
}

Further reading

Additional helpful documentation, links, and articles:

Discover Datadog PipelinesDOCUMENTATION Learn more about parsingDOCUMENTATION