The Service Map for APM is here!

Processing

Processing

Overview

To access the processing panel use the left Logs menu:

Pipelines panel

Log processing allows you to have full control over how your logs are processed with Datadog Pipelines and Processors.

Pipelines and Processors can be applied to any type of logs:

Thanks to this you do not need to change how you log, and you don’t need to deploy changes to any server-side processing rules, everything is happening and can be configured directly in the Datadog processing page.

The other benefit to implement a log processing strategy is to implement an attribute naming convention for your organization.

Log Processing

Integration logs

For integration logs, we automatically install a Integration Pipeline that takes care of parsing your logs and adds the corresponding facet in your Explorer view as on this example for ELB logs:

ELB log post processing

Custom logs

However we know that log formats can be totally custom which is why you can define custom processing rules. With any log syntax, you can extract all your attributes and, when necessary, remap them to more global or canonical attributes.

So for instance with custom processing rules you can transform this log:

Log pre processing

Into this one:

Log post processing

Consult the dedicated Pipelines documentation pages to learn more on how to perform actions only on some subset of your logs with the Pipeline filters.

To discover the full list of Processors available, refer to the dedicated Processor documentation page.

If you want to learn more about pure parsing possibilities of the Datadog application, follow the parsing training guide. We also have a parsing best practice and a parsing troubleshooting guide that might be interesting for you.

Reserved attributes

If your logs are formatted as JSON, be aware that some attributes are reserved for use by Datadog:

date attribute

By default Datadog generates a timestamp and appends it in a date attribute when logs are received. However, if a JSON formatted log file includes one of the following attributes, Datadog interprets its value as the the log’s official date:

  • @timestamp
  • timestamp
  • _timestamp
  • Timestamp
  • eventTime
  • date
  • published_date
  • syslog.timestamp

You can also specify alternate attributes to use as the source of a log’s date by setting a log date remapper Processor

Note: Datadog rejects a log entry if its official date is older than 6 hours in the past.

The recognized date formats are: ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

message attribute

By default, Datadog ingests the value of message as the body of the log entry. That value is then highlighted and displayed in the logstream, where it is indexed for full text search.

status attribute

Each log entry may specify a status level which is made available for faceted search within Datadog. However, if a JSON formatted log file includes one of the following attributes, Datadog interprets its value as the the log’s official status:

  • status
  • severity
  • level
  • syslog.severity

If you would like to remap some status existing in the status attribute, you can do so with the log status remapper

host attribute

Using the Datadog Agent or the RFC5424 format automatically sets the host value on your logs. However, if a JSON formatted log file includes the following attribute, Datadog interprets its value as the the log’s host:

  • host
  • hostname
  • syslog.hostname

service attribute

Using the Datadog Agent or the RFC5424 format automatically sets the service value on your logs. However, if a JSON formatted log file includes the following attribute, Datadog interprets its value as the the log’s service:

  • service
  • syslog.appname

Edit reserved attributes

You can now control the global hostname, service, timestamp, and status main mapping that are applied before the processing Pipelines. This is particularly helpful if logs are sent in JSON or from an external Agent.

Reserved Attribute

To change the default values for each of the reserved attributes, go to the Pipeline page and edit the Reserved Attribute mapping:

Reserved Attribute Tile

Further Reading