Indexes are located on the Configuration page in the Indexes section. Double click on them or click on the edit button to see more information about the number of logs that were indexed in the past 3 days, as well as the retention period for those logs:
Index filters give dynamic control over what goes into your indexes.
For example, if some logs were captured only for troubleshooting purposes, you may only want to index those logs with errors and warnings. This can easily be achieved with exclusion filters.
To define a new index filter, click on the “add” button:
To configure an exclusion filter:
Save the filter.
Note: If a log matches several exclusion filters, only the first exclusion filter rule is applied. A log is not sampled or excluded multiple times by different exclusion filters.
Order matters for exclusion filters. Contrary to how several pipelines can process a log, if a log matches several exclusion filters, only the first exclusion filter rule is applied.
Reorder your pipeline to make sure the proper exclusion filters apply to your log. For instance, you probably want to set up the filters ordered by least inclusive to most inclusive queries..
To reorder your exclusion filter, drag and drop them into your preferred order.
If not all logs are worth indexing on a daily basis, they might still be important in certain situations. Debug logs, for instance, are not always useful, but during complex troubleshooting or a production release, they can become very helpful.
Instead of changing your application logging level or using a complex internal filtering tool, you can change what is indexed directly with Datadog index filters.
Enable or disable them in one click on the Pipeline page:
It is also possible to have multiple indexes with different retention periods (currently in private beta). Logs enter the first index whose filter they match on, so it is important to order your indexes carefully.
For example, if you create a first index filtered to the
status:notice attribute, a second index filtered to the
status:error attribute, and a final one without any filter (the equivalent of
*), all your notice logs would go to the first index, all your error logs to the second index, and the rest would go to the final one.
Multiple indexes also provide the ability to define access rules on the data contained in each index. More information available in the role base access control documentation.
Additional helpful documentation, links, and articles: