Cloud-based applications can generate logs at a rate of millions per minute. But because your logs are not all and equally valuable at any moment, Datadog Logging without Limits™ provides flexibility by decoupling log ingestion and indexing.
This guide identifies key components of Logging Without Limits™ such as Patterns, Exclusion Filters, Custom log-based metrics, and Monitors that can help you better organize Log Explorer and monitor your KPIs over time.
Your most logged service contains several logs, some of which may be irrelevant for troubleshooting. For example, you may want to investigate every 4xx and 5xx response code log, but excluded every 200 response code log from Log Explorer to expedite troubleshooting during a major outage or event. By identifying the corresponding service first, you can quickly track down which service status produces the most logs and is best to exclude from the Log Explorer view.
To identify your most logged service status:
serviceand limit to
serviceto group by
status. This generates a top statuses list for your service.
Note: These steps are applicable to any high volume logging query to generate a top list. You can group by any facet, such as
Now that you have identified your most logging service status, switch to the patterns view, located next to the graph view in the top left of Log Explorer, to automatically see your log patterns for the selected context.
A context is composed of a time range and a search query. Each pattern comes with highlights to get you straight to its characteristic features. A mini graph displays a rough timeline for the volume of its logs to help you identify how that pattern differs from other patterns. Sections of logs that vary within the pattern are highlighted to help you quickly identify differences across log lines.
Click on the log pattern that you would like to exclude to see a sample of underlying logs.
The patterns view is helpful when identifying and filtering noisy patterns. It shows the number of logs matching a pattern, split by service and status. Click on the first pattern to view a detailed log of events relating to your status. A contextual panel will populate with information about your noisiest status pattern.
The pattern context panel lists every instance (event) of a log pattern and creates a custom search query based on the selected pattern. Use this query in an exclusion filter to remove those logs from your index.
To create an exclusion filter:
</>option to the right of the search query and copy the search query.
Note: If a log matches several exclusion filters, only the first exclusion filter rule is applied. A log is not sampled or excluded multiple times by different exclusion filters.
In this example, the service status
Updating recommendations with customer_id=* & url=shops/*/* is filtered with an exclusion filter. Removing any high volume logging pattern similar to this one from Log Explorer will help you drill down and identify issues quicker. However, these logs are only removed from the Log Explorer view. They are still ingested, and available to view in Live Tail, sent to log archives, or used to generate metrics.
Exclusion filters can be disabled at any time by toggling the disable option to the right of the filter. They can also be modified and removed by hovering over the filter and selecting the edit or delete option.
Once a log pattern is excluded from Log Explorer, you can still track KPIs over time at the ingest level by creating a new custom log-based metric.
To generate a new log-based metric based on your log pattern:
service:web-store status:info "updating recommendations with customer_id" "url shops")
*to generate a count of all logs matching your query or enter a measure (e.g.,
@duration) to aggregate a numeric value and create its corresponding count, min, max, sum, and avg aggregated metrics.
<KEY>:<VALUE>format. Log-based metrics are considered custom metrics. Avoid grouping by unbounded or extremely high cardinality attributes like timestamps, user IDs, request IDs, or session IDs to avert impacting your billing.
Anomaly detection is an algorithmic feature that identifies when a metric is behaving differently than it has in the past. Creating an anomaly detection monitor for your excluded logs will alert you of any changes based on your set alert conditions.
To set an anomaly detection monitor:
When an anomaly is detected, an alert will be sent to all who are tagged. This alert can also be found in Monitors -> Triggered Monitors.
In this guide, you learned how to use Logging without Limits™ to:
To learn more about Logging Without Limits™ and how to better utilize features like Log Explorer, Live Tail, and Log Patterns, view the links below.
Additional helpful documentation, links, and articles: