Network Performance Monitoring is now generally available! Network Monitoring is now available!

Parsing

Overview

Datadog automatically parses JSON-formatted logs. For other formats, Datadog allows you to enrich your logs with the help of Grok Parser. The Grok syntax provides an easier way to parse logs than pure regular expressions. The Grok Parser enables you to extract attributes from semi-structured text messages.

Grok comes with reusable patterns to parse integers, IP addresses, hostnames, etc.

You can write parsing rules with the %{MATCHER:EXTRACT:FILTER} syntax:

  • Matcher: rule (possibly a reference to another token rule) that describes what to expect (number, word, notSpace, etc.)

  • Extract (optional): an identifier representing the capture destination for the piece of text matched by the Matcher.

  • Filter (optional): a post-processor of the match to transform it.

Example for a classic unstructured log:

john connected on 11/08/2017

With the following parsing rule:

MyParsingRule %{word:user} connected on %{date("MM/dd/yyyy"):connect_date}

After processing, the following structured log is generated:

Note:

  • If you have multiple parsing rules in a single Grok parser:
    • Only one can match any given log. The first one that matches, from top to bottom, is the one that does the parsing.
    • Each rule can reference parsing rules defined above itself in the list.
  • You must have unique rule names within the same Grok parser.
  • The rule name must contain only: alphanumeric characters, _, and .. It must start with an alphanumeric character.

Matcher and Filter

Here is a list of all the matchers and filters natively implemented by Datadog:

PatternUsage
date("pattern"[, "timezoneId"[, "localeId"]])Matches a date with the specified pattern and parses to produce a Unix timestamp. See the date Matcher examples.
regex("pattern")Matches a regex. Check the regex Matcher examples.
dataMatches any string including spaces and newlines. Equivalent to .*.
notSpaceMatches any string until the next space.
boolean("truePattern", "falsePattern")Matches and parses a Boolean, optionally defining the true and false patterns (defaults to true and false, ignoring case).
numberStrMatches a decimal floating point number and parses it as a string.
numberMatches a decimal floating point number and parses it as a double precision number.
numberExtStrMatches a floating point number (with scientific notation support) and parses it as a string.
numberExtMatches a floating point number (with scientific notation support) and parses it as a double precision number.
integerStrMatches an integer number and parses it as a string.
integerMatches an integer number and parses it as an integer number.
integerExtStrMatches an integer number (with scientific notation support) and parses it as a string.
integerExtMatches an integer number (with scientific notation support) and parses it as an integer number.
wordMatches characters from a-z, A-Z, 0-9, including the _ (underscore) character.
doubleQuotedStringMatches a double-quoted string.
singleQuotedStringMatches a single-quoted string.
quotedStringMatches a double-quoted or single-quoted string.
uuidMatches a UUID.
macMatches a MAC address.
ipv4Matches an IPV4.
ipv6Matches an IPV6.
ipMatches an IP (v4 or v6).
hostnameMatches a hostname.
ipOrHostMatches a hostname or IP.
portMatches a port number.
PatternUsage
numberParses a match as double precision number.
integerParses a match as an integer number.
booleanParses ‘true’ and ‘false’ strings as booleans ignoring case.
nullIf("value")Returns null if the match is equal to the provided value.
jsonParses properly formatted JSON.
rubyhashParses properly formatted Ruby hash (e.g. {name => "John", "job" => {"company" => "Big Company", "title" => "CTO"}}).
useragent([decodeuricomponent:true/false])Parses a user-agent and returns a JSON object that contains the device, OS, and the browser represented by the Agent. Check the User Agent processor.
querystringExtracts all the key-value pairs in a matching URL query string (e.g. ?productId=superproduct&promotionCode=superpromo).
decodeuricomponentDecodes URI components. For instance, it transforms %2Fservice%2Ftest into /service/test.
lowercaseReturns the lower-cased string.
uppercaseReturns the upper-cased string.
keyvalue([separatorStr[, characterWhiteList [, quotingStr]])Extracts key value pattern and returns a JSON object. See key-value Filter examples.
scale(factor)Multiplies the expected numerical value by the provided factor.
array([[openCloseStr, ] separator][, subRuleOrFilter)Parses a string sequence of tokens and returns it as an array.
urlParses a URL and returns all the tokenized members (domain, query params, port, etc.) in a JSON object. More info on how to parse URLs.

Advanced Settings

At the bottom of your Grok processor tiles, there is an Advanced Settings section:

  • Use the Extract from field to apply your Grok processor on a given attribute instead of the default message attribute.

  • Use the Helper Rules field to define tokens for your parsing rules. Helper rules help you to factorize Grok patterns across your parsing rules. This is useful when you have several rules in the same Grok parser that use the same tokens.

Example for a classic unstructured log:

john id:12345 connected on 11/08/2017 on server XYZ in production

Use the following parsing rule:

MyParsingRule %{user} %{connection} %{server}

With the following helpers:

user %{word:user.name} id:%{integer:user.id}
connection connected on %{date("MM/dd/yyyy"):connect_date}
server on server %{notSpace:server.name} in %{notSpace:server.env}

Examples

Some examples demonstrating how to use parsers:

Key value

This is the key-value core filter: keyvalue([separatorStr[, characterWhiteList[, quotingStr]]]) where:

  • separatorStr: defines the separator. Defaults to =.
  • characterWhiteList: defines additional non-escaped value chars. Defaults to \\w.\\-_@. Used only for non-quoted values (e.g. key=@valueStr).
  • quotingStr: defines quotes. Default behavior detects quotes (<>, "", etc.).

Note: If you define a keyvalue filter on a data object, and this filter is not matched, then an empty JSON {} is returned (e.g. input: key:=valueStr, parsing rule: rule_test %{data::keyvalue("=")}, output: {}).

Use filters such as keyvalue() to more-easily map strings to attributes:

Log:

user=john connect_date=11/08/2017 id=123 action=click

Rule

rule %{data::keyvalue}

You don’t need to specify the name of your parameters as they are already contained in the log. If you add an extract attribute my_attribute in your rule pattern you will see:

If = is not the default separator between your key and values, add a parameter in your parsing rule with a separator.

Log:

user: john connect_date: 11/08/2017 id: 123 action: click

Rule

rule %{data::keyvalue(": ")}

If logs contain special characters in an attribute value, such as / in a url for instance, add it to the whitelist in the parsing rule:

Log:

url=https://app.datadoghq.com/event/stream user=john

Rule:

rule %{data::keyvalue("=","/:")}

Other examples:

Raw stringParsing ruleResult
key=valueStr%{data::keyvalue}{“key”: “valueStr}
key=<valueStr>%{data::keyvalue}{“key”: “valueStr”}
key:valueStr%{data::keyvalue(":")}{“key”: “valueStr”}
key:“/valueStr”%{data::keyvalue(":", "/")}{“key”: “/valueStr”}
key:={valueStr}%{data::keyvalue(":=", "", "{}")}{“key”: “valueStr”}

Multiple QuotingString example: When multiple quotingstring are defined, the default behavior is replaced with a defined quoting character. The key-value always matches inputs without any quoting characters, regardless of what is specified in quotingStr. When quoting characters are used, the characterWhiteList is ignored as everything between the quoting characters is extracted.

Log:

  key1:=valueStr key2:=</valueStr2> key3:="valueStr3"

Rule:

  rule %{data::keyvalue(":=","","<>")}

Result:

  {
    "key1": "valueStr",
    "key2": "/valueStr2"
  }

Parsing dates

The date matcher transforms your timestamp in the EPOCH format (unit of measure millisecond).

Raw stringParsing ruleResult
14:20:15%{date("HH:mm:ss"):date}{“date”: 51615000}
02:20:15 PM%{date("hh:mm:ss a"):date}{“date”: 51615000}
11/10/2014%{date("dd/MM/yyyy"):date}{“date”: 1412978400000}
Thu Jun 16 08:29:03 2016%{date("EEE MMM dd HH:mm:ss yyyy"):date}{“date”: 1466065743000}
Tue Nov 1 08:29:03 2016%{date("EEE MMM d HH:mm:ss yyyy"):date}{“date”: 1466065743000}
06/Mar/2013:01:36:30 +0900%{date("dd/MMM/yyyy:HH:mm:ss Z"):date}{“date”: 1362501390000}
2016-11-29T16:21:36.431+0000%{date("yyyy-MM-dd'T'HH:mm:ss.SSSZ"):date}{“date”: 1480436496431}
2016-11-29T16:21:36.431+00:00%{date("yyyy-MM-dd'T'HH:mm:ss.SSSZZ"):date}{“date”: 1480436496431}
06/Feb/2009:12:14:14.655%{date("dd/MMM/yyyy:HH:mm:ss.SSS"):date}{“date”: 1233922454655}
Thu Jun 16 08:29:03 20161%{date("EEE MMM dd HH:mm:ss yyyy","Europe/Paris"):date}{“date”: 1466058543000}
2007-08-31 19:22:22.427 ADT%{date("yyyy-MM-dd HH:mm:ss.SSS z"):date}{“date”: 1188598942427}

1 Use this format if you perform your own localizations and your timestamps are not in UTC. Timezone IDs are pulled from the TZ Database. For more information, see TZ database names.

Note: Parsing a date doesn’t set its value as the log official date. For this use the Log Date Remapper in a subsequent Processor.

Conditional pattern

If you have logs with two possible formats which differ in only one attribute, set a single rule using conditionals with (<REGEX_1>|<REGEX_2>).

Log:

john connected on 11/08/2017
12345 connected on 11/08/2017

Rule: Note that “id” is an integer and not a string.

MyParsingRule (%{integer:user.id}|%{word:user.firstname}) connected on %{date("MM/dd/yyyy"):connect_date}

Results:

Optional attribute

Some logs contain values that only appear part of the time. In this case, make attribute extraction optional with ()?.

Log:

john 1234 connected on 11/08/2017

Rule:

MyParsingRule %{word:user.firstname} (%{integer:user.id} )?connected on %{date("MM/dd/yyyy"):connect_date}

Note: A rule will not match if you include a space after the first word in the optional section.

Nested JSON

Use the json filter to parse a JSON object nested after a raw text prefix:

Log:

Sep 06 09:13:38 vagrant program[123]: server.1 {"method":"GET", "status_code":200, "url":"https://app.datadoghq.com/logs/pipelines", "duration":123456}

Rule:

parsing_rule %{date("MMM dd HH:mm:ss"):timestamp} %{word:vm} %{word:app}\[%{number:logger.thread_id}\]: %{notSpace:server} %{data::json}

Regex

Use the regex matcher to match any substring of your log message.

Log:

john_1a2b3c4 connected on 11/08/2017

Rule:

MyParsingRule %{regex("[a-z]*"):user.firstname}_%{regex("[a-zA-Z0-9]*"):user.id} .*

Further Reading