With Trace Queries, you can find entire traces based on the properties of multiple spans and the relationships between those spans within the structure of the trace. To create a trace query, you define two or more span queries and then specify the relationship within the searched-for trace structure of the spans that are returned by each span query.
You can search, filter, group, and visualize the traces from the Trace Query explorer.
With structure-based trace querying, you can answer questions such as:
Which traces include a dependency between two services (service A has a downstream call to service B)?
What API endpoints are affected by my erroring backend service?
Use Trace Queries to accelerate your investigations and find relevant traces.
Query for spans from a specific environment, service, or endpoint using the Span query syntax. Use autocomplete suggestions to view facets and recent queries.
Click Add another span query to add a span query and use it in the trace query statement.
Trace query operators
Combine multiple span queries, labeled a, b, c, and so on, into a trace query in the Traces matching field, using operators between the letters that represent each span query:
Operator
Description
Example
&&
And: Both spans are in the trace
Traces that contain spans from the service web-store and spans from the service payments-go: service:web-store && service:payments-go
||
Or: One or the other span are in the trace
Traces that contain spans from the service web-store or from the service mobile-store: service:web-store || service:mobile-store
->
Indirect relationship: Traces that contain a span matching the left query that is upstream of spans matching the right query
Traces where the service checkoutservice is upstream of the service quoteservice: service:checkoutservice -> service:quoteservice
=>
Direct relationship: Traces that contain a span matching the left query that is the direct parent of a span matching the right query
Traces where the service checkoutservice is directly calling the service shippingservice: service:checkoutservice => service:shippingservice
Trace-level filters
Filter the result set of traces further by applying filters on trace-level attributes like the number of spans or the end-to-end duration of the trace in the Where statement:
Filter
Description
Example
span_count(a)
Number of occurrences of a span
Traces that contain more than 10 calls to a mongo database: - queryA:service:web-store-mongo @db.statement:"SELECT * FROM stores - Traces matching:a - Where:span_count(a):>10
total_span_count
Number of spans in the trace
Traces that contain more than 1000 spans: Wheretotal_span_count:>1000
trace_duration
End to end trace duration
Traces for which the end-to-end execution time is more than 5 seconds : Where:trace_duration:>2s
Flow Map
The Flow Map helps you understand the request path and service dependencies from the resulting traces that match the Trace Query. Use the map to identify error paths, unusual service dependencies, or abnormally high request rates to a database.
Service nodes that match span queries are highlighted to show you which parts of the trace your query conditions are targeting.
To get more information about a single service, hover on the service’s node to see its metrics for request rate and error rate. To see metrics for the request rate and the error rate between two services, hover on an edge connecting the two services.
To filter out traces that do not contain a dependency on a particular service, click on the service’s node on the map.
Trace list
The Trace list shows up to fifty sample traces that match the query and are within the selected time range.
Hover on the Latency Breakdown to get a sense of where (in which services) time is spent during the request execution.
Note: Information displayed in the table are attributes from the root span of the trace, including the duration, which does not represent the end-to-end duration of the trace.
Analytics
Select one of the other visualizations, such as Timeseries, Top List, or Table to aggregate results over time, grouped by one or multiple dimensions. Read Span Visualizations for more information on the aggregation options.
In addition to those aggregation options, you must also select which span query (a, b, c, and so on) you want to aggregate the spans from. Select the query that matches the spans from which you’re using the tags and attributes in the aggregation options.
For example, if you query for traces that contain a span from the service web-store (query a) and a span from the service payments-go with some errors (query b), and you visualize a count of spans grouped by @merchant.tier, use spans from query a, because merchant.tier is an attribute from the spans of the service web-store, not from the service payments-go.
The flat 1% sampling is applied based on the trace_id, meaning that all spans that belong to the same trace share the same sampling decision. Spans indexed by the 1% sampling can also be queried and found in the Trace explorer.
Spans indexed by tag-based retention filters cannot be used in Trace Queries because retention filters do not guarantee that all the spans from a trace are indexed.
Note: Spans indexed by the flat 1% sampling are not counted towards your usage of indexed spans, and so do not impact your bill.
Further Reading
Additional helpful documentation, links, and articles:
