- 필수 기능
- 시작하기
- Glossary
- 표준 속성
- Guides
- Agent
- 통합
- 개방형텔레메트리
- 개발자
- Administrator's Guide
- API
- Datadog Mobile App
- CoScreen
- Cloudcraft
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- APM
- Continuous Profiler
- 스팬 시각화
- 데이터 스트림 모니터링
- 데이터 작업 모니터링
- 디지털 경험
- 소프트웨어 제공
- 보안
- AI Observability
- 로그 관리
- 관리
",t};e.buildCustomizationMenuUi=t;function n(e){let t='
",t}function s(e){let n=e.filter.currentValue||e.filter.defaultValue,t='${e.filter.label}
`,e.filter.options.forEach(s=>{let o=s.id===n;t+=``}),t+="${e.filter.label}
`,t+=`This page discusses using Datadog’s LLM Observability Trace Explorer to query your LLM application’s spans and traces.
In LLM Observability, a span represents a unit of work representing a single operation in your LLM application. A trace represents the end-to-end operations involved in processing a request in your LLM application, often consisting of one or more nested spans. For more information about this terminology, see LLM Observability Terms and Concepts.
To toggle between searching spans and traces, use the drop-down menu on the left of the query field on LLM Observability Trace Explorer. To search across traces where the root span matches the query, select Traces. To search across all your spans, including nested child spans, select All Spans.
Some search terms are only applicable to traces. For examples, see Trace-level queries.
Span attributes are key-value pairs attached directly to each span. Attributes capture details about the span’s execution, such as performance metrics, resource identifiers, or parameter values.
Attribute queries take the form @key:value
. All attribute keys are prepended with @
.
Query | Match |
---|---|
@duration:>5s | Spans that took more than 5 seconds to complete |
Span tags are key-value pairs used to group, segment, and correlate telemetry data across spans, services, or environments. Tags often indicate broader context, such as application name, environment, or deployment region, and are attached to spans to facilitate efficient search and aggregation.
Tag queries take the form key:value
. In contrast to attribute keys, tag keys are not prepended with @
.
Query | Match |
---|---|
ml_app:my_llm_app | Spans from an application named my_llm_app |
You can also use free text queries to search for specific keywords, phrases, or strings on any span that has an input or output pair. To use free text search, wrap your query with "
.
Query | Match |
---|---|
"what's the weather" | Agent, workflow, or LLM spans that contain the string what's the weather in the input or output |
You can combine multiple search terms using the Boolean operators AND
(intersection), OR
(union), and -
(exclusion).
Query | Match |
---|---|
@duration:>5s AND -"foo" | Spans that took more than 5 seconds to complete and do not contain the string foo in the input or output |
The LLM Observability Trace Explorer shares the same query syntax as Datadog’s APM Trace Explorer. For query syntax details, including wildcard search, handling numerical values, escaping special characters, and more, see Trace Explorer Query Syntax.
Query | Match |
---|---|
@status:error | Spans or traces that have a status of error |
@meta.error.type:"Max turns exceeded" | Spans or traces that have the error type Max turns exceeded |
@duration:>5s | Spans or traces that took more than 5 seconds to complete |
@trace.total_tokens:>=1000 | Traces that consumed 1000 or more total tokens |
ml_app:my_llm_app | Spans or traces from an application named my_llm_app |
"what's the weather" | Agent, workflow, or LLM spans that contain the string what's the weather in the input or output |
Use the @evaluations
attribute to find spans or traces by evaluation result.
You can search spans by the results of custom evaluations. For example, if you have a custom evaluation called user_mood
with categorical values happy
, sad
, and tired
, you could use the query: @evaluations.custom.user_mood:happy
.
Query | Match |
---|---|
@evaluations.custom.user_satisfaction:>5 | Spans or traces that scored higher than 5 according to a custom evaluation called user_satisfaction |
@evaluations.custom.user_mood:happy | Spans or traces that were evaluated as happy according to a custom evaluation called user_mood that has the categorical values happy , sad , and tired |
Query | Match |
---|---|
@evaluations.failure_to_answer:"failed to answer" | Spans or traces where the LLM failed to deliver an appropriate response |
@evaluations.hallucination:"hallucination found" | Spans or traces where a hallucination is detected |
@evaluations.input_sentiment:negative | Spans or traces with negative input sentiment |
@evaluations.input_toxicity:toxic | Spans or traces where harmful or inappropriate content is detected in the input |
@evaluations.language_mismatch:mismatched | Spans or traces where the user’s prompt and the LLM’s response are in different languages |
@evaluations.output_sentiment:negative | Spans or traces with negative output sentiment |
@evaluations.output_toxicity:toxic | Spans or traces where harmful or inappropriate content is detected in the output |
@evaluations.prompt_injection:found | Spans or traces where prompt injection is detected |
@evaluations.topic_relevancy:irrelevant | Spans or traces where the prompt input diverges from the LLM application’s intended topic |
Use the @meta
attribute to find spans by metadata information.
Query | Match |
---|---|
@meta.span.kind:llm | Spans of the llm span kind. |
@meta.model_provider:openai | Spans or traces where the model provider is OpenAI |
@meta.model_name:gpt-4.1 | Spans or traces where the model is GPT-4.1 |
To search traces based on attributes of its nested spans, use the @child
attribute.
Query | Match |
---|---|
@child.@evaluations.hallucination:"hallucination found" | Traces with a hallucinating sub-span |
@child.@meta.span.name:retrieval AND @meta.span. kind:workflow | Workflow traces that contain a retrieval span |
Use the @trace
attribute to access trace-level information, such as estimated total cost, number of LLM calls, or number of tools.
Query | Match |
---|---|
@trace.llm_calls:>3 | Traces with more than 3 LLM calls |
@trace.tool_calls:>=4 | Traces with 4 or more tool calls |
@trace.number_of_tools:<5 | Traces that call fewer than 5 different tools |
추가 유용한 문서, 링크 및 기사: