Trace Queries Source Data
Overview
With Trace Queries, you can find entire traces based on the properties of multiple spans and the relationships between those spans within the structure of the trace. To learn more, read the Trace Queries documentation.
How Trace Queries source data
Datadog uses the Intelligent Retention Filter to index data for Trace Queries. It does so by performing:
- Flat sampling: A uniform 1% sample of ingested spans.
- Diversity sampling: A representative, diverse selection of traces to keep visibility over each environment, service, operation, and resource.
These 2 sampling mechanisms capture complete traces, meaning that all spans of a trace are always indexed to ensure the well-functioning of Trace Queries.
Note: Spans indexed by flat sampling and diversity sampling do not count towards the usage of indexed spans, and therefore, do not impact your bill.
1% flat sampling
retained_by:flat_sampled
Flat 1% sampling is applied based on the trace_id, meaning that all spans belonging to the same trace share the same sampling decision. To learn more, read the one percent flat sampling documentation.
Diversity sampling
retained_by:diversity_sampling
Every 15 minutes, diversity sampling retains at least one span and the associated trace for each combination of environment, service, operation, and resource. This occurs for the p75, p90, and p95 percentile of latencies to ensure that you can always find example traces in service and resource pages, even for low traffic endpoints. To learn more, read the diversity sampling documentation.
Impact of enabling Trace Queries
From the moment Traces Queries are enabled on your account (find the exact date in the event published in the Event Stream), the Intelligent Retention filter starts to index more data as it starts capturing complete traces .
You can query spans indexed by the Intelligent Retention filter in the Trace Explorer. As a result, you might notice a spike in the number of indexed spans in Trace Explorer queries. This change is indicated by an event overlay showing an Intelligent Retention Filter change event.
To find spans that are sampled by the 1% flat sampling or the diversity sampling methods, add a retained_by:(flat_sampled OR diversity_sampling) query parameter in the Trace Explorer.
Spans indexed by the Intelligent retention filter are excluded from APM queries in Trace Analytics monitor evaluations. Therefore, monitors are not impacted by this change.
Further reading
Additional helpful documentation, links, and articles:
