Generate metrics from 100% of ingested spans, regardless of whether the spans are indexed by a retention filter.
Use custom metrics for specific fixed queries and comparisons, while creating retention filters to allow arbitrary querying and investigation of the retained trace and its flame graph.
Billing Note: Metrics created from ingested spans are billed as Custom Metrics.
For example, you may want to use custom metrics to visualize anomalies, create dashboards and monitors, and see trends across any parameters that are important to your business context. All generated metrics are available for 15 months as Datadog custom metrics.
Define the metric query: Start by adding a query for filtering to your required dataset. The query syntax is the same as APM Search and Analytics.
Define the field you want to track: Select * to generate a count of all spans matching your query or enter an attribute (for example, @cassandra_row_count) to aggregate a numeric value and create its corresponding count, minimum, maximum, sum, and average aggregated metrics. If the attribute type is a measure, the value of the metric is the value of the span attribute.
Note: Span attributes that are not numerical values cannot be used for aggregation. To generate a metric that counts the distinct values of a span attribute (for instance count the number of user IDs hitting a specific endpoint), add this dimension to the group by selector, and use the count_nonzero function to count the number of tag values.
Specify the group-by dimension: By default, metrics generated from spans will not have any tags unless explicitly added. Any attribute or tag that exists in your spans can be used to create metric tags.
Check the Live Analytics and Search Query preview: You can view the impact of your query in real-time on the data visualization, and the matching spans considered for your query in a live preview.
Name your metric: Metric names must follow the metric naming convention. Metric names that start with trace.* are not permitted and will not be saved.
Span-based metrics are considered custom metrics and billed accordingly. Avoid grouping by unbounded or extremely high cardinality attributes like timestamps, user IDs, request IDs, or session IDs to avoid impacting your billing.
Updating existing span-based metrics
After a metric is created, only two fields can be updated:
Field
Reason
Stream filter query
Change the set of matching spans to be aggregated into metrics.
Aggregation groups
Update the tags to manage the cardinality of generated metrics.
Note: To change the metric type or name, create a new metric and delete the old one.
Further Reading
Additional helpful documentation, links, and articles:
