Generate Metrics from Spans

Generate Metrics from Spans

This feature is currently in public beta. While this feature is in beta it is not recommended to use metrics generated from spans for monitors or production operations use cases.

Generate span-based metrics

With Tracing without Limits™, you can generate metrics from 100% of ingested spans, regardless of whether they are indexed by a retention filter.

You can pair these metrics with retention filters and Analytics monitors, or use them on their own.

Use custom metrics for specific fixed queries and comparisons, while creating retention filters to allow arbitrary querying and investigation of the retained trace and its flamegraph.

Billing Note: Metrics created from ingested spans are billed as Custom Metrics.

For example, you may want to use custom metrics to visualize anomalies, create dashboards and monitors, and see trends across any parameters that are important to your business context. All generated metrics are available for 15 months as Datadog custom metrics.

ReasonCustom Metrics Generated from SpansRetention Filters
Retention Period15 months15 days
Anomaly DetectionCreate an Anomaly Monitor based on generated metrics.Use Analytics to compare behavior over the past 15 days, and drill into complete traces to investigate root cause.
Investigation of matching traces with full contextN/A - Custom Metrics will not result in any retention of associated traces.Keep exactly the traces relevant to your business context with retention filters.
Granularity of behaviorCreate custom metrics for important endpoints or other low-cardinality groups.Use Search and Analytics for specific endpoints, or use the ‘Group By’ option in Analytics.
Forecasting or complex mathematicsCreate a Forecast monitor based on generated metrics.N/A

To generate metrics from spans, on the APM Setup and Configuration page select the Generate Metrics tab, and click the New Metric button.

Creating a span-based metric

  1. Define the metric query: Start by adding a query for filtering to your required dataset. The query syntax is the same as APM Search and Analytics.

  2. Define the field you want to track: Select * to generate a count of all spans matching your query or enter an attribute (for example, @cassandra_row_count) to aggregate a numeric value and create its corresponding count, minimum, maximum, sum, and average aggregated metrics. If the attribute type is a measure, the value of the metric is the value of the span attribute.

  3. Specify the group-by dimension: By default, metrics generated from spans will not have any tags unless explicitly added. Any attribute or tag that exists in your spans can be used to create metric tags.

  4. Check the Live Analytics and Search Query preview: You can view the impact of your query in real-time on the data visualization, and the matching spans considered for your query in a live preview.

  5. Name your metric: Metric names must follow the metric naming convention. Metric names that start with trace.* are not permitted and will not be saved.

Important Note: Span-based metrics are considered custom metrics and billed accordingly. Avoid grouping by unbounded or extremely high cardinality attributes like timestamps, user IDs, request IDs, or session IDs to avoid impacting your billing.

Updating existing span-based metrics

After a metric is created, only two fields can be updated:

Stream filter queryChange the set of matching spans to be aggregated into metrics.
Aggregation groupsUpdate the tags to manage the cardinality of generated metrics.

Note: To change the metric type or name, create a new metric and delete the old one.

Further Reading