Datadog’s Logging without Limits* lets you dynamically decide what to include or exclude from your indexes for storage and query, at the same time many types of logs are meant to be used for telemetry to track trends, such as KPIs, over long periods of time. Log-based metrics are a cost-efficient way to summarize log data from the entire ingest stream. This means that even if you use exclusion filters to limit what you store for exploration, you can still visualize trends and anomalies over all of your log data at 10s granularity for 15 months.
With log-based metrics, you can generate a count metric of logs that match a query or a distribution metric of a numeric value contained in the logs, such as request duration.
You can also create metrics from an Analytics search by selecting the “Generate new metric” option from the Export menu.
*to generate a count of all logs matching your query or enter a log attribute (e.g.,
@network.bytes_written) to aggregate a numeric value and create its corresponding
avgaggregated metrics. If the log attribute facet is a measure, the value of the metric is the value of the log attribute.
group by: By default, metrics generated from logs will not have any tags unless explicitly added. Any attribute or tag dimension that exists in your logs (for example,
env) can be used to create metric tags. Metric tags names are equal to the originating attribute or tag name, without the @. Log-based metrics are considered custom metrics. Avoid grouping by unbounded or extremely high cardinality attributes like timestamps, user IDs, request IDs, or session IDs to avert impacting your billing. See the Log Security page for more information on using this feature for HIPAA-enabled customers.
Note: Data points for Log-based metrics are generated at ten second intervals.
After a metric is created, the following fields can be updated:
To change the metric type or name, a new metric must be created.
Usage metrics are estimates of your current Datadog usage in near real-time. They enable you to:
Log Management usage metrics come with three tags that can be used for more granular monitoring:
|Indicates the routing query that matches a log to an intended index.|
|Indicates whether or not a log matches an exclusion query.|
|The service attribute of the log event.|
status tag is available on the
datadog.estimated_usage.logs.ingested_events metric to reflect the log status (
Additional helpful documentation, links, and articles: