The Service Map for APM is here!

Logging without Limits

Overview

Sometimes the amount of log events generated by your infrastructure is too large and/or fluctuates a lot, raising the issue of choosing which logs should be sent to a Log Management solution and which should be sent to an archive. But filtering your logs before sending them inevitably leads to gaps in coverage, and often filters out valuable data.

Datadog log management removes these limitations by decoupling log ingestion from indexation, which makes it possible to cost-effectively collect, process, and archive all your logs. Datadog Index Filters avoid complex Agent level configuration and control what you want to index dynamically. You can now:

  • Ingest all your log events without server side filtering
  • Process and enrich all of them
  • Live Tail over the whole infrastructure
  • Dynamically decide what to include or exclude from your indexes to control your costs
  • Get alerted when volumes grows unexpectedly over an index
  • Archive all enriched logs

This flexibility is critical in some exceptional situations such as outages, when you can disable specific filters to send more data. The inverse is true as well; if you over-consume because of a seasonal reason (Black Friday, Christmas, etc…) you can decide to selectively reduce some volume to avoid overages.

Index details

Indexes are located at the bottom of the pipeline page. Double click on them or click on the edit button to see more information about the number of logs that were indexed in the past 3 days, and the retention period for those logs:

Indexed logs can be used for faceted searching, Log Analytics, dashboarding, and monitoring.

Setup Log Monitors on volumes

Get notified at any moment if the volumes in any scope (service, availibility-zone, etc…) of your infrastructure is growing unexpectedly:

  1. Go in the Datadog Log Explorer view
  2. Build a search query that represents the volume to monitor.
  3. Click on Export to monitor.
  4. Define the rate you would like to set as warning or error.
  5. Define an explicit notification: The volume on this service just got too high. Define an additional exclusion filter or increase the sampling rate to put it back under control.

Exclusion Filters

Index Filters give dynamic control over what goes into your indexes.

For example, if some logs were captured only for troubleshooting purposes, you may only care to index those logs with errors and warnings. This can easily be achieved with exclusion filters.

To define a new Index Filter click on the “add” button:

To configure an exclusion filter:

  1. Define the name of your filter
  2. Define the query for logs to exclude from your index Note: It is possible to use any attribute or tag in the Index filter query, even those that are not facets. If you are filtering by non-faceted attributes or tags, be sure to hit “enter/return” from the query bar
  3. Save the filter

Example

The following filter removes all logs that have a fast response time. We use the duration attribute and filter all logs that have a value below 100ms.

{
    "http": {
        "url": "https://app.datadoghq.com/logs",
        "status_code": "200"
    },
    "duration":12,
    "metadata": {
        "version": 12,
        "release": "sept18"
    }
}

Filter: @duration:<100

Container example

Container logs have a lot of metadata collected as tags. To exclude all logs coming from images that contains httpd in the image_name tag use the following filter:

Filter: image_name:*httpd*

Enable/Disable filters

If all logs are not worth indexing on a daily basis, they might still be critical in some situations. Debug logs, for instance, are not always useful but during a complex troubleshooting or during a production release they become very interesting to get better insight into what is going on.

Instead of changing your application logging level or using a complex internal filtering tool, it is now possible to change what is indexed directly with the Datadog index filters.

Enable or disable them in one click in the Pipeline page:

Further Reading