Querying spans and traces
This product is not supported for your selected
Datadog site. (
).
Overview
This page discusses using Datadog’s LLM Observability Trace Explorer to query your LLM application’s spans and traces.
Querying across spans versus traces
In LLM Observability, a span represents a unit of work representing a single operation in your LLM application. A trace represents the end-to-end operations involved in processing a request in your LLM application, often consisting of one or more nested spans. For more information about this terminology, see LLM Observability Terms and Concepts.
To toggle between searching spans and traces, use the drop-down menu on the left of the query field on LLM Observability Trace Explorer. To search across traces where the root span matches the query, select Traces. To search across all your spans, including nested child spans, select All Spans.
Some search terms are only applicable to traces. For examples, see Trace-level queries.
Query by attribute
Span attributes are key-value pairs attached directly to each span. Attributes capture details about the span’s execution, such as performance metrics, resource identifiers, or parameter values.
Attribute queries take the form @key:value. All attribute keys are prepended with @.
| Query | Match |
|---|
@duration:>5s | Spans that took more than 5 seconds to complete |
Query by tag
Span tags are key-value pairs used to group, segment, and correlate telemetry data across spans, services, or environments. Tags often indicate broader context, such as application name, environment, or deployment region, and are attached to spans to facilitate efficient search and aggregation.
Tag queries take the form key:value. In contrast to attribute keys, tag keys are not prepended with @.
| Query | Match |
|---|
ml_app:my_llm_app | Spans from an application named my_llm_app |
You can also use free text queries to search for specific keywords, phrases, or strings on any span that has an input or output pair. To use free text search, wrap your query with ".
| Query | Match |
|---|
"what's the weather" | Agent, workflow, or LLM spans that contain the string what's the weather in the input or output |
Free text queries are limited to the first 20,500 characters of a span's input or output.
Operators
You can combine multiple search terms using the Boolean operators AND (intersection), OR (union), and - (exclusion).
| Query | Match |
|---|
@duration:>5s AND -"foo" | Spans that took more than 5 seconds to complete and do not contain the string foo in the input or output |
Query syntax
The LLM Observability Trace Explorer shares the same query syntax as Datadog’s APM Trace Explorer. For query syntax details, including wildcard search, handling numerical values, escaping special characters, and more, see Trace Explorer Query Syntax.
Example queries
| Query | Match |
|---|
@status:error | Spans or traces that have a status of error |
@meta.error.type:"Max turns exceeded" | Spans or traces that have the error type Max turns exceeded |
@duration:>5s | Spans or traces that took more than 5 seconds to complete |
@trace.total_tokens:>=1000 | Traces that consumed 1000 or more total tokens |
ml_app:my_llm_app | Spans or traces from an application named my_llm_app |
"what's the weather" | Agent, workflow, or LLM spans that contain the string what's the weather in the input or output |
Evaluation queries
Use the @evaluations attribute to find spans or traces by evaluation result.
Custom evaluations
You can search spans by the results of custom evaluations. For example, if you have a custom evaluation called user_mood with categorical values happy, sad, and tired, you could use the query: @evaluations.custom.user_mood:happy.
| Query | Match |
|---|
@evaluations.custom.user_satisfaction:>5 | Spans or traces that scored higher than 5 according to a custom evaluation called user_satisfaction |
@evaluations.custom.user_mood:happy | Spans or traces that were evaluated as happy according to a custom evaluation called user_mood that has the categorical values happy, sad, and tired |
Out-of-the-box evaluations
| Query | Match |
|---|
@evaluations.failure_to_answer:"failed to answer" | Spans or traces where the LLM failed to deliver an appropriate response |
@evaluations.hallucination:"hallucination found" | Spans or traces where a hallucination is detected |
@evaluations.input_sentiment:negative | Spans or traces with negative input sentiment |
@evaluations.input_toxicity:toxic | Spans or traces where harmful or inappropriate content is detected in the input |
@evaluations.language_mismatch:mismatched | Spans or traces where the user’s prompt and the LLM’s response are in different languages |
@evaluations.output_sentiment:negative | Spans or traces with negative output sentiment |
@evaluations.output_toxicity:toxic | Spans or traces where harmful or inappropriate content is detected in the output |
@evaluations.prompt_injection:found | Spans or traces where prompt injection is detected |
@evaluations.topic_relevancy:irrelevant | Spans or traces where the prompt input diverges from the LLM application’s intended topic |
Use the @meta attribute to find spans by metadata information.
| Query | Match |
|---|
@meta.span.kind:llm | Spans of the llm span kind. |
@meta.model_provider:openai | Spans or traces where the model provider is OpenAI |
@meta.model_name:gpt-4.1 | Spans or traces where the model is GPT-4.1 |
Trace-level queries
To search traces based on attributes of its nested spans, use the @child attribute.
| Query | Match |
|---|
@child.@evaluations.hallucination:"hallucination found" | Traces with a hallucinating sub-span |
@child.@meta.span.name:retrieval AND @meta.span. kind:workflow | Workflow traces that contain a retrieval span |
Use the @trace attribute to access trace-level information, such as estimated total cost, number of LLM calls, or number of tools.
| Query | Match |
|---|
@trace.llm_calls:>3 | Traces with more than 3 LLM calls |
@trace.tool_calls:>=4 | Traces with 4 or more tool calls |
@trace.number_of_tools:<5 | Traces that call fewer than 5 different tools |
Further reading
Additional helpful documentation, links, and articles:
