With Tracing without Limits™, you can generate metrics from 100% of ingested spans, regardless of whether they are indexed by a retention filter.
You can pair these metrics with retention filters and Analytics monitors, or use them on their own.
Use custom metrics for specific fixed queries and comparisons, while creating retention filters to allow arbitrary querying and investigation of the retained trace and its flamegraph.
For example, you may want to use custom metrics to visualize anomalies, create dashboards and monitors, and see trends across any parameters that are important to your business context. All generated metrics are available for 15 months as Datadog custom metrics.
|Reason||Custom Metrics Generated from Spans||Retention Filters|
|Retention Period||15 months||15 days|
|Anomaly Detection||Create an Anomaly Monitor based on generated metrics.||Use Analytics to compare behavior over the past 15 days, and drill into complete traces to investigate root cause.|
|Investigation of matching traces with full context||N/A - Custom Metrics will not result in any retention of associated traces.||Keep exactly the traces relevant to your business context with retention filters.|
|Granularity of behavior||Create custom metrics for important endpoints or other low-cardinality groups.||Use Search and Analytics for specific endpoints, or use the ‘Group By’ option in Analytics.|
|Forecasting or complex mathematics||Create a Forecast monitor based on generated metrics.||N/A|
Define the metric query: Start by adding a query for filtering to your required dataset. The query syntax is the same as APM Search and Analytics.
Define the field you want to track: Select
* to generate a count of all spans matching your query or enter an attribute (for example,
@cassandra_row_count) to aggregate a numeric value and create its corresponding count, minimum, maximum, sum, and average aggregated metrics. If the attribute type is a measure, the value of the metric is the value of the span attribute.
Specify the group-by dimension: By default, metrics generated from spans will not have any tags unless explicitly added. Any attribute or tag that exists in your spans can be used to create metric tags.
Check the Live Analytics and Search Query preview: You can view the impact of your query in real-time on the data visualization, and the matching spans considered for your query in a live preview.
Name your metric: Metric names must follow the metric naming convention. Metric names that start with
trace.* are not permitted and will not be saved.
Important Note: Span-based metrics are considered custom metrics and billed accordingly. Avoid grouping by unbounded or extremely high cardinality attributes like timestamps, user IDs, request IDs, or session IDs to avoid impacting your billing.
After a metric is created, only two fields can be updated:
|Stream filter query||Change the set of matching spans to be aggregated into metrics.|
|Aggregation groups||Update the tags to manage the cardinality of generated metrics.|
Note: To change the metric type or name, create a new metric and delete the old one.
Additional helpful documentation, links, and articles: