Overview

By default, when Datadog’s Intake API receives a log, an INFO status generates and appends itself as the status attribute.

This default status may not always reflect the actual status contained in the log itself. This guide walks you through how to override the default value with the actual status.

Raw logs

If your raw logs are not showing the correct status in Datadog, extract the correct log status from the raw logs and remap it to the right status.

Extract the status value with a parser

Use a Grok parser to define a rule with the word() matcher and extract the actual log status.

1. Navigate to Logs Pipelines and click on the pipeline processing the logs.
2. Click Add Processor.
3. Select Grok Parser for the processor type.
4. Use the word() matcher to extract the status and pass it into a custom log_status attribute.

For example, the log may look like:

WARNING: John disconnected on 09/26/2017

Add a rule like:

MyParsingRule %{word:log_status}: %{word:user.name} %{word:action}.*

The output for MyParsingRule’s extraction:

{
  "action": "disconnected",
  "log_status": "WARNING",
  "user": {
    "name": "John"
  }
}

Define a log status remapper

The log_status attribute contains the correct status. Add a Log Status remapper to make sure the status value in the log_status attribute overrides the default log status.

1. Navigate to Logs Pipelines and click on the pipeline processing the logs.
2. Click Add Processor.
3. Select Status remapper as the processor type.
4. Enter a name for the processor.
5. Add log_status to the Set status attribute(s) section.
6. Click Create.

Modifications of a pipeline impacts new logs only because all the processing is done in the intake process.

JSON logs

JSON logs are automatically parsed in Datadog. Because the log status attribute is a reserved attribute, it goes through pre-processing operations for JSON logs.

In this example, the actual status of the log is the value of the logger_severity attribute, not the default INFO log status.

To make sure the logger_severity attribute value overrides the default log status, add logger_severity to the list of status attributes.

1. Navigate to Logs Pipelines and click on the pipeline processing the logs.
2. Hover over Preprocessing for JSON Logs, and click the pencil icon.
3. Add logger_severity to the list of status attributes. The status remapper looks for every reserved attribute in the order they are listed. To ensure the status comes from the logger_severity attribute, place it first in the list.
4. Click Save.

Modifications of a pipeline impacts new logs only because all the processing is done in the ingestion process. New logs are correctly configured with the logger_severity attribute value.

In order for the remapping to work, you must adhere to the status formats specified in the Processors documentation.

Further Reading

Additional helpful documentation, links, and articles:

Learn how to remap custom severity values to the official log statusDOCUMENTATION Learn how to process your logsDOCUMENTATION Learn about parsingDOCUMENTATION Learn how to investigate a log parsing issueDOCUMENTATION