Overview

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes:

• timestamp
• date
• _timestamp
• Timestamp
• eventTime
• published_date

If your logs have dates in an attribute that are not in this list, use the log date remapper processor to define their date attribute as the official log timestamp:

If your logs don’t have a timestamp that conforms to the formats listed above, use the grok processor to extract the epoch time from the timestamp to a new attribute. The date remapper uses the newly defined attribute.

To see how a custom date and time format can be parsed in Datadog, see Parsing dates.

Notes:

• Log events can be submitted up to 18 hours in the past and two hours in the future.
• As of ISO 8601-1:2019, the basic format is T[hh][mm][ss] and the extended format is T[hh]:[mm]:[ss]. Earlier versions omitted the T (representing time) in both formats.
• If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.
• If multiple log date remapper processors are applied to a given log within the pipeline, the last one (according to the pipeline’s order) is taken into account.

Use cases

The Log Date Remapper is typically used to:

• Select another attribute as the date to be used for the log event, if it doesn’t match our default date attributes. For example, logs coming from Akamai Stream use reqTimeMillis as the timestamp. Use a Log Date Remapper to set that attribute as the log timestamp.
• Select an attribute after grok parsing. For example, logs coming from Ruby contain the log timestamp in the message attribute which is only accessible after parsing it.

Before and after state of logs

Before logs are parsed, Datadog references its own ingestion time as the log timestamp, which may not match the actual event time found in the log message.

After applying the Log Date Remapper, Datadog uses the custom timestamp from the specified attribute, aligning the log’s timestamp in Datadog with the true event time recorded in the log message. This ensures historical accuracy when analyzing or searching your logs.

API

Use the Datadog Log Pipeline API endpoint with the following log date remapper JSON payload:

{
  "type": "date-remapper",
  "name": "Define <SOURCE_ATTRIBUTE> as the official Date of the log",
  "is_enabled": false,
  "sources": ["<SOURCE_ATTRIBUTE_1>"]
}

Further reading

Additional helpful documentation, links, and articles:

Discover Datadog PipelinesDOCUMENTATION Learn more about parsing datesDOCUMENTATION