To access the processing panel use the left
Thanks to this you do not need to change how you log, and you don’t need to deploy changes to any server-side processing rules, everything is happening and can be configured directly in the Datadog processing page.
The other benefit to implement a log processing strategy is to implement an attribute naming convention for your organization.
For integration logs, we automatically install a Integration Pipeline that takes care of parsing your logs and adds the corresponding facet in your Explorer view as on this example for ELB logs:
However we know that log formats can be totally custom which is why you can define custom processing rules. With any log syntax, you can extract all your attributes and, when necessary, remap them to more global or canonical attributes.
So for instance with custom processing rules you can transform this log:
Into this one:
To discover the full list of Processors available, refer to the dedicated Processor documentation page.
If you want to learn more about pure parsing possibilities of the Datadog application, follow the parsing training guide. We also have a parsing best practice and a parsing troubleshooting guide that might be interesting for you.
If your logs are formatted as JSON, be aware that some attributes are reserved for use by Datadog:
By default Datadog generates a timestamp and appends it in a date attribute when logs are received. However, if a JSON formatted log file includes one of the following attributes, Datadog interprets its value as the the log’s official date:
You can also specify alternate attributes to use as the source of a log’s date by setting a log date remapper Processor
Note: Datadog rejects a log entry if its official date is older than 6 hours in the past.
Each log entry may specify a status level which is made available for faceted search within Datadog. However, if a JSON formatted log file includes one of the following attributes, Datadog interprets its value as the the log’s official status:
If you would like to remap some status existing in the
status attribute, you can do so with the log status remapper
Using the Datadog Agent or the RFC5424 format automatically sets the host value on your logs. However, if a JSON formatted log file includes the following attribute, Datadog interprets its value as the the log’s host:
Using the Datadog Agent or the RFC5424 format automatically sets the service value on your logs. However, if a JSON formatted log file includes the following attribute, Datadog interprets its value as the the log’s service:
You can now control the global hostname, service, timestamp, and status main mapping that are applied before the processing Pipelines. This is particularly helpful if logs are sent in JSON or from an external Agent.
To change the default values for each of the reserved attributes, go to the Pipeline page and edit the
Reserved Attribute mapping:
Additional helpful documentation, links, and articles: