Monitor your log usage
The goal of this guide is to explain how to monitor your log usage with estimated usage metrics. This guide goes through the following steps:
- Alert on unexpected traffic spikes
- Alert when you are getting close to a budget threshold on your indexed logs
- Import the out-of-the-box Log Management usage dashboard
Alert on unexpected spikes
Logs usage metrics
By default, log usage metrics are available to track the number of ingested logs, ingested bytes, and indexed logs. These metrics are free and kept for 15 months:
See below how to leverage them in anomaly detection monitors.
Note: It is recommended to set the unit to Byte
for the datadog.estimated_usage.logs.ingested_bytes
in the metric summary page:
Anomaly detection monitors
To define anomaly detection monitors to be alerted of any unexpected indexing log spikes:
- Create a new Anomaly monitor
- Select the
datadog.estimated_usage.logs.ingested_events
metric - Add
datadog_is_excluded:false
in the from
section (to monitor indexed logs and not ingested ones) - Add the tag
service
and datadog_index
in count by (to be notified if a specific service spikes or stops sending logs in any indexes) - Set the alert condition to match your use case (for example, an evaluation window, or number of times outside the expected range)
- Set the notification message with actionable instructions:
Example of a notification with contextual links:
An unexpected amount of logs has been indexed in index {{datadog_index.name}}
1. [Check Log patterns for this service](https://app.datadoghq.com/logs/patterns?from_ts=1582549794112&live=true&to_ts=1582550694112&query=service%3A{{service.name}})
2. [Add an exclusion filter on the noisy pattern](https://app.datadoghq.com/logs/pipelines/indexes)
Estimated usage dashboard
Once you begin ingesting logs, an out-of-the-box dashboard summarizing your log usage metrics is automatically installed in your account.
Note: The metrics used in this dashboard are estimates and might differ from official billing numbers.
To access this dashboard, go to Dashboards > Dashboards List and search for Log Management - Estimated Usage
.
Monitor indexed logs with fixed threshold
Get notified if the indexed log volumes in any scope (service
, availability-zone
, etc…) of your infrastructure are growing unexpectedly:
- Go to the Datadog Log Explorer view.
- Build a search query that represents the volume to monitor. Keep the query empty to monitor all the logs from that index.
- Click on Export to monitor.
- Define the rate you would like to set as warning or error.
- Define an explicit notification:
The volume on this service just got too high. Define an additional exclusion filter or increase the sampling rate to get it back under control.
Alert on indexes reaching their daily quota
It is also possible to set up a daily quota on indexes to prevent indexing more than a given number of logs per day. When doing this, Datadog recommends that you set the above monitor to alert when 80% of this quota is reached within the past 24 hours.
An event is generated when the daily quota is reached. Set up a monitor to be notified when this happens:
Events generated when the daily quota is reached have the datadog_index
tag by default, displaying the index name. You can optionally create a facet on the datadog_index
tag, enabling its use in the group by
step for multi alerts, as shown in the screenshot above.
Here is an example of what the notification would look like in Slack:
Further Reading
Additional helpful documentation, links, and articles: