OpenTelemetry Instrumentation
This product is not supported for your selected
Datadog site. (
).
Overview
By using OpenTelemetry’s standardized semantic conventions for generative AI operations, you can instrument your LLM applications with any OpenTelemetry-compatible library or framework and visualize the traces in Agent Observability.
Agent Observability supports ingesting OpenTelemetry traces that follow either the OpenTelemetry 1.37+ semantic conventions for generative AI or the supported OpenInference semantic conventions. This allows you to send LLM traces directly from OpenTelemetry-instrumented applications to Datadog without requiring the Agent Observability SDK or a Datadog Agent.
Prerequisites
Supported features
Evaluations
To send external evaluations directly to the API for OpenTelemetry spans, include the source:otel tag in the evaluation. When referencing spans, provide span_id and trace_id as decimal strings. OpenTelemetry uses hexadecimal IDs natively, so convert them to decimal before submitting evaluations. For example, use Python’s int(hex_span_id, 16) to convert a hex span ID to its decimal equivalent.
Prompt Tracking
For information on using Prompt Tracking with OpenTelemetry spans, see Prompt Tracking - OpenTelemetry Instrumentation.
Offline Experiments
You can use OpenTelemetry spans inside Agent Observability Offline Experiments. By setting DD_TRACE_OTEL_ENABLED=1, OTel spans created inside an experiment task automatically appear as children of the experiment span.
Multimodal support
Audio and images on OpenTelemetry messages are rendered in the trace view. Datadog extracts media from message parts that follow the OpenTelemetry GenAI semantic conventions, as described in Media in messages.
Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. For the fields, formats, and size limits that apply once media reaches a span, see Multimodal Support.
Span links
Use OpenTelemetry span links on your GenAI spans to express non-parent-child relationships, such as when one span’s output feeds another span’s input. When two linked spans are in the same trace, the link appears as an edge in that trace’s Execution Graph, so you can see how data flows between sibling spans (for example, a tool’s output feeding a downstream LLM call).
Use from and to attributes to indicate the direction of the data flow:
from opentelemetry import trace
from opentelemetry.trace import Link
tracer = trace.get_tracer(__name__)
# A tool span whose output feeds a downstream LLM call.
with tracer.start_as_current_span("lookup_order") as tool_span:
tool_span.set_attribute("gen_ai.operation.name", "execute_tool")
tool_ctx = tool_span.get_span_context()
# The LLM span links back to the tool span: its output became this span's input.
link = Link(context=tool_ctx, attributes={"from": "output", "to": "input"})
with tracer.start_as_current_span("chat gpt-4o", links=[link]) as llm_span:
llm_span.set_attribute("gen_ai.operation.name", "chat")
A span link that points to a span in a different trace is stored, but is not drawn in the Execution Graph, which visualizes a single trace.
Setup
1. Enable LLM Observability conversion
To enable LLM Observability conversion, set dd_llmobs_enabled=true as a resource attribute to include it in every exported span. Setting it on a single span might result in partially converted large traces. Under the standard conversion policy, an explicit dd_llmobs_enabled=false overrides organization-level enablement.
If you export traces directly to Datadog over OTLP HTTP, add DD-LLMObs-Enabled=true to OTEL_EXPORTER_OTLP_TRACES_HEADERS. This enables LLM Observability conversion for every span in the export unless dd_llmobs_enabled=false is explicitly set on a span or resource under the standard conversion policy. Use this header only for trace exports to Datadog, not for requests to your model provider.
Using environment variables
Add the dd_llmobs_enabled=true attribute to your OTEL_RESOURCE_ATTRIBUTES environment variable:
OTEL_RESOURCE_ATTRIBUTES=dd_llmobs_enabled=true
Using code
You can also set the attribute programmatically on any span in your trace:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-span") as span:
# Enable LLM Observability conversion for this entire trace
span.set_attribute("dd_llmobs_enabled", True)
2. Configure the exporter
Any method Datadog supports for ingesting OpenTelemetry traces works with Agent Observability. For the full list of supported ingestion paths, see OpenTelemetry feature compatibility. The following is one way to configure it.
To send OpenTelemetry traces to Agent Observability, configure your OpenTelemetry exporter with the following settings:
Set the following environment variables in your application:
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=
OTEL_EXPORTER_OTLP_TRACES_HEADERS=dd-api-key=<YOUR_API_KEY>,dd-otlp-source=llmobs
OTEL_RESOURCE_ATTRIBUTES=dd_llmobs_enabled=true
Replace <YOUR_API_KEY> with your Datadog API key. Add dd_llmobs_enabled=true to any existing OTEL_RESOURCE_ATTRIBUTES values. The dd-otlp-source=llmobs header identifies the source but doesn’t enable conversion. See Enabling LLM Observability conversion.
If your framework previously supported a pre-1.37 OpenTelemetry specification version, you also need to set:
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
This environment variable enables version 1.37+-compliant OpenTelemetry traces for frameworks that now support the version 1.37+ semantic conventions, but previously supported older versions (such as strands-agents).
Note:
- If you are using an OpenTelemetry library other than the default OpenTelemetry SDK, you may need to configure the endpoint, protocol, and headers differently depending on the library’s API. See your library’s documentation for the appropriate configuration method.
- When using OpenTelemetry instrumentation, some data sent to Agent Observability may also be written to the corresponding APM traces. If you are protecting sensitive data, consider also configuring a Restricted Dataset on APM to match your Agent Observability access controls. See Data Access Control for more information.
Using strands-agents
If you are using the strands-agents library, you need to set an additional environment variable to enable traces that are compliant with OpenTelemetry v1.37+:
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
This environment variable ensures that strands-agents emits traces following the OpenTelemetry v1.37+ semantic conventions for generative AI, which are required by Agent Observability.
3. Instrument the application
To generate traces compatible with Agent Observability, do one of the following:
After you enable conversion and the application starts sending eligible spans, the traces appear in the Agent Observability Traces page. To search for your traces in the UI, use the ml_app attribute, which is automatically set to the value of your OpenTelemetry root span’s service attribute.
Tested frameworks and libraries
These frameworks and libraries have been tested with Agent Observability. Frameworks that emit the supported attributes from the OpenTelemetry 1.37+ GenAI semantic conventions or OpenInference semantic conventions can send spans to Agent Observability.
Send traces from your framework or platform
The Claude Platform natively emits spans that follow the GenAI semantic conventions through OpenTelemetry export, so you don’t need an instrumentation library. Native export covers the Messages API, Claude Managed Agents, and the Claude SDKs. Spans come from two independent sources:
- API spans cover requests that Anthropic runs: Messages API requests and, for Managed Agents, each Sessions API request, agent turn, model request, and tool call. Anthropic delivers these spans to a destination that you configure in the Claude Console.
- SDK spans for each Claude SDK call, and for each tool that the SDK tool runner executes inside your application. Your application exports these with its own OpenTelemetry exporter.
If your application uses a Claude SDK, both sets of spans appear in the same trace, because the SDK propagates W3C trace context on its API requests.
Export API spans
Anthropic delivers API spans from its own infrastructure, so you add a Datadog destination in the Claude Console rather than setting OTEL_EXPORTER_OTLP_* environment variables.
In the Claude Console, go to Organization settings > OpenTelemetry and click Add destination. Complete the fields:
| Field | Value |
|---|
| Provider | Datadog |
| Site | Your Datadog site, for example US1 |
| API key | Your Datadog API key. Application keys do not work. |
| Content mode | Full content so spans carry message and tool content, or Metadata only for identifiers, timing, status, and token counts |
Selecting Datadog as the provider sets the OTLP endpoint for the site you choose, shown under OTLP details. You don’t need to enter an endpoint or headers.
Choose Full content only if your Datadog organization is permitted to store model prompts and responses. To restrict who can read them after they arrive, see Data Access Controls.
Set Workspaces on the destination to choose which workspaces export to it: All, Selected for a specific list, or All except to exclude a few.
Send a request or run a session, then check that the destination’s Delivery column reads Healthy. The spans arrive under the claude-platform service. A Managed Agents session forms this tree:
| Span | Agent Observability span kind |
|---|
anthropic.sessions.events.send | Workflow |
anthropic.session.turn | Agent, carrying the conversation input and output |
anthropic.model_request | LLM, carrying the model name and token usage |
Hosted tool calls, for example bash or read | Tool |
A destination receives spans for every surface the workspace uses, so Messages API requests arrive at the same destination without further configuration.
Export SDK spans
To also capture spans from the Claude SDK in your own application, register an OpenTelemetry tracer provider before you construct the client, and export to Datadog:
import os
from anthropic import Anthropic
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# service.name becomes the ml_app that these spans are indexed under.
provider = TracerProvider(resource=Resource.create({"service.name": "my-agent-app"}))
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint="<code class="js-region-param region-param" data-region-param="otlp_trace_endpoint"></code>",
headers={
"dd-api-key": os.environ["DD_API_KEY"],
"dd-otlp-source": "llmobs",
"DD-LLMObs-Enabled": "true",
},
)
)
)
trace.set_tracer_provider(provider)
# content_mode="content" records prompts and responses. The default is metadata only.
client = Anthropic(
open_telemetry={"tracer_provider": provider, "traces": {"content_mode": "content"}}
)
The endpoint for this exporter includes the /v1/traces path. You can also set the content mode with ANTHROPIC_OPEN_TELEMETRY_TRACES_CONTENT_MODE=content, or turn off SDK spans with ANTHROPIC_OPEN_TELEMETRY=false.
To group several calls into one conversation, set a conversation ID on the client. The SDK sends it as gen_ai.conversation.id, which Agent Observability indexes as the session:
client = client.with_options(
open_telemetry={**client.open_telemetry, "conversation_id": "conv_42"}
)
message = client.messages.create(
model="claude-opus-5",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello, Claude"}],
)
Traces appear on the Agent Observability Traces page. Search by ml_app to find them.
Avoid double-counted tokens
If you export both Claude SDK spans and Claude Platform API spans for the same workspace, Agent Observability counts tokens for the same model calls twice. The duplicate counts inflate token metrics and estimated cost. If the Claude SDK is your only source of spans, skip this step.
To count each model call once, drop the usage attributes from the Claude SDK spans before they leave your application, so the API spans are the only source of token counts:
from opentelemetry.sdk.trace import ReadableSpan
from opentelemetry.sdk.trace.export import SpanExporter
# Agent Observability reads token counts and costs from these two namespaces.
DROPPED_PREFIXES = ("gen_ai.usage.", "gen_ai.cost.")
def _without_token_usage(span):
attributes = {
key: value
for key, value in (span.attributes or {}).items()
if not key.startswith(DROPPED_PREFIXES)
}
return ReadableSpan(
name=span.name,
context=span.get_span_context(),
parent=span.parent,
resource=span.resource,
attributes=attributes,
events=span.events,
links=span.links,
kind=span.kind,
status=span.status,
start_time=span.start_time,
end_time=span.end_time,
instrumentation_scope=span.instrumentation_scope,
)
class StripClaudeTokenUsage(SpanExporter):
"""Leave token counting to the Claude Platform spans."""
def __init__(self, exporter):
self._exporter = exporter
def export(self, spans):
return self._exporter.export(
[
_without_token_usage(span)
if span.name.startswith("anthropic.")
else span
for span in spans
]
)
def shutdown(self):
return self._exporter.shutdown()
def force_flush(self, timeout_millis=30000):
return self._exporter.force_flush(timeout_millis)
Then wrap the OTLP exporter with it where you register the span processor:
provider.add_span_processor(
BatchSpanProcessor(StripClaudeTokenUsage(OTLPSpanExporter(endpoint=..., headers=...)))
)
The following example demonstrates a complete application using Strands Agents with the OpenTelemetry integration. This same approach works with any framework that supports OpenTelemetry version 1.37+ semantic conventions for generative AI.
from strands import Agent
from strands_tools import calculator, current_time
from strands.telemetry.config import StrandsTelemetry
import os
# Configure AWS credentials for Bedrock access
os.environ["AWS_PROFILE"] = "<YOUR_AWS_PROFILE>"
os.environ["AWS_DEFAULT_REGION"] = "<YOUR_AWS_REGION>"
# Enable latest GenAI semantic conventions (1.37)
os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = "gen_ai_latest_experimental"
# Configure OTLP endpoint to send traces to Agent Observability
os.environ["OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"] = "http/protobuf"
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "<code class="js-region-param region-param" data-region-param="otlp_trace_endpoint"></code>"
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = f"dd-api-key={os.getenv('DD_API_KEY')},dd-otlp-source=llmobs,DD-LLMObs-Enabled=true"
# Initialize telemetry with OTLP exporter
telemetry = StrandsTelemetry()
telemetry.setup_otlp_exporter()
# Create agent with tools
agent = Agent(tools=[calculator, current_time])
# Run the agent
if __name__ == "__main__":
result = agent("I was born in 1993, what is my age?")
print(f"Agent: {result}")
The following example demonstrates using OpenLLMetry to automatically instrument OpenAI calls with OpenTelemetry.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
import openai
from opentelemetry.sdk.resources import Resource
resource = Resource.create({
"service.name": "simple-openllmetry-test",
})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)
exporter = OTLPSpanExporter(
endpoint="<code class="js-region-param region-param" data-region-param="otlp_trace_endpoint"></code>",
headers={
"dd-api-key": "<YOUR_DATADOG_API_KEY>",
"dd-ml-app": "simple-openllmetry-test",
"dd-otlp-source": "llmobs",
"DD-LLMObs-Enabled": "true",
},
)
provider.add_span_processor(BatchSpanProcessor(exporter))
OpenAIInstrumentor().instrument()
# Make OpenAI call (automatically traced)
client = openai.OpenAI(api_key="<YOUR_OPENAI_API_KEY>")
client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "What is 15 multiplied by 7?"}]
)
provider.force_flush(timeout_millis=5000)
After running this example, search for ml_app:simple-openllmetry-test in the Agent Observability UI to find the generated trace.
The following example uses the OpenInference OpenAI instrumentation to automatically instrument OpenAI calls with OpenTelemetry.
Configure the OpenTelemetry exporter and instrument the OpenAI client:
import openai
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
resource = Resource.create({
"service.name": "simple-openinference-test",
})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)
exporter = OTLPSpanExporter(
endpoint="<code class="js-region-param region-param" data-region-param="otlp_trace_endpoint"></code>",
headers={
"dd-api-key": "<YOUR_DATADOG_API_KEY>",
"dd-ml-app": "simple-openinference-test",
"dd-otlp-source": "llmobs",
"DD-LLMObs-Enabled": "true",
},
)
provider.add_span_processor(BatchSpanProcessor(exporter))
OpenAIInstrumentor().instrument(tracer_provider=provider)
# Make OpenAI call (automatically traced)
client = openai.OpenAI(api_key="<YOUR_OPENAI_API_KEY>")
client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "What is 15 multiplied by 7?"}]
)
provider.force_flush(timeout_millis=5000)
After running this example, search for ml_app:simple-openinference-test in the Agent Observability UI to find the generated trace.
Cloudflare Agents running on Workers emit spans that follow the GenAI semantic conventions through Workers Observability. Because Workers export telemetry through a destination configured in the Cloudflare dashboard, you set the Datadog OTLP endpoint and headers on that destination instead of with OTEL_EXPORTER_OTLP_* environment variables.
Enable tracing in your Worker configuration (wrangler.jsonc), pointing at a named destination:
{
"observability": {
"traces": {
"enabled": true,
"destinations": ["datadog-agent-observability"]
}
}
}
Create the destination in the Cloudflare dashboard under Workers & Pages > Observability > Telemetry. Add a traces destination whose name matches the one in your Worker configuration, and set:
| Field | Value |
|---|
| Destination URL | <code class="js-region-param region-param" data-region-param="otlp_trace_endpoint"></code> |
| Header | dd-api-key: <YOUR_API_KEY> |
| Header | dd-otlp-source: llmobs |
| Header | DD-LLMObs-Enabled: true |
The dd-otlp-source=llmobs header identifies the source; DD-LLMObs-Enabled=true enables conversion. Replace <YOUR_API_KEY> with your Datadog API key.
Instrument model calls. Cloudflare instruments its supported agent frameworks automatically. If you call the Vercel AI SDK (generateText, streamText, generateObject, streamObject) directly, wrap it with wrapAISDK() so that model calls produce chat and invoke_agent spans with the required gen_ai.* attributes:
import { Agent } from "agents";
import * as ai from "ai";
import { wrapAISDK } from "agents/observability/ai";
import { createWorkersAI } from "workers-ai-provider";
// storeMessages/storeTools capture gen_ai.input.messages, gen_ai.output.messages,
// and tool call payloads. They are off by default.
const { generateText } = wrapAISDK(ai, { storeMessages: true, storeTools: true });
export class MyAgent extends Agent<Env> {
async onRequest(request: Request): Promise<Response> {
const workersai = createWorkersAI({ binding: this.env.AI });
const { text } = await generateText({
model: workersai("@cf/meta/llama-3.3-70b-instruct-fp8-fast"),
prompt: "What is distributed tracing?",
});
return Response.json({ text });
}
}
After you deploy the Worker and it serves traffic, the traces appear on the Agent Observability Traces page. Search by the ml_app attribute, which is set to your Worker’s service name.
The following example demonstrates how to instrument your LLM application using custom OpenTelemetry code. This approach gives you full control over the traces and spans emitted by your application.
import os
import json
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from openai import OpenAI
# Configure OpenTelemetry to send traces to Datadog
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "<code class="js-region-param region-param" data-region-param="otlp_trace_endpoint"></code>"
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = "dd-api-key=<YOUR_DATADOG_API_KEY>,dd-otlp-source=llmobs,DD-LLMObs-Enabled=true"
os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = "gen_ai_latest_experimental"
# Initialize OpenTelemetry SDK
resource = Resource(attributes={SERVICE_NAME: "simple-llm-example"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
# Make LLM call with OpenTelemetry tracing
with tracer.start_as_current_span(
"chat gpt-4o",
kind=trace.SpanKind.CLIENT,
) as span:
model = "gpt-4o"
max_tokens = 1024
temperature = 0.7
messages = [{"role": "user", "content": "Explain OpenTelemetry in one sentence."}]
# Set request attributes
span.set_attribute("gen_ai.provider.name", "openai")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.request.max_tokens", max_tokens)
span.set_attribute("gen_ai.request.temperature", temperature)
# Add input messages as event
input_messages_parts = []
for msg in messages:
input_messages_parts.append({
"role": msg["role"],
"parts": [{"type": "text", "content": msg["content"]}]
})
span.add_event(
"gen_ai.client.inference.operation.details",
{
"gen_ai.input.messages": json.dumps(input_messages_parts)
}
)
# Make actual LLM call
client = OpenAI(api_key="<YOUR_OPENAI_API_KEY>")
response = client.chat.completions.create(
model=model,
max_tokens=max_tokens,
temperature=temperature,
messages=messages
)
# Set response attributes from actual data
span.set_attribute("gen_ai.response.id", response.id)
span.set_attribute("gen_ai.response.model", response.model)
span.set_attribute("gen_ai.response.finish_reasons", [response.choices[0].finish_reason])
span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
# Add output messages as event
output_text = response.choices[0].message.content
span.add_event(
"gen_ai.client.inference.operation.details",
{
"gen_ai.output.messages": json.dumps([{
"role": "assistant",
"parts": [{"type": "text", "content": output_text}],
"finish_reason": response.choices[0].finish_reason
}])
}
)
print(f"Response: {output_text}")
# Flush spans before exit
provider.force_flush()
After running this example, search for ml_app:simple-llm-example in the Agent Observability UI to find the generated trace.
Attribute mapping reference
This section provides the mappings from OpenTelemetry GenAI semantic conventions (v1.37+), OpenLLMetry, OpenInference, and Langfuse to Datadog’s Agent Observability span schema.
Trace eligibility
You must enable conversion for a trace to be eligible. The trace must also contain at least one span with any of the following attributes:
| Attribute | Requirement |
|---|
gen_ai.operation.name | Present |
gen_ai.provider.name | Present |
gen_ai.system (deprecated) | Present |
openinference.span.kind | Non-empty value |
langfuse.observation.type | Non-empty value |
tfy.span_type | Non-empty value (TrueFoundry AI Gateway) |
Agent Observability also accepts Claude traces that match any of the following:
- Span names starting with
claude_code. - Instrumentation scope
com.anthropic.claude-platform - Instrumentation scope starting with
com.anthropic.sdk. and a non-empty suffix
Some spans in these traces may not map to Agent Observability fields. See Troubleshooting mapping warnings.
In a trace that qualifies, Agent Observability keeps only spans that have gen_ai.* attributes or a non-empty openinference.span.kind, langfuse.observation.type, or tfy.span_type. It drops all other spans.
Framework-specific compatibility
| Instrumentation | Compatibility notes |
|---|
| TrueFoundry AI Gateway | Native tfy.* attributes support span kind, model, provider, token usage, cost, and I/O mapping without requiring gen_ai.* attributes. |
| Older GenAI conventions | Deprecated attributes and pre-1.30 message events are supported. See Input and output messages. |
| Vercel AI SDK legacy telemetry | Selected model-call spans identified by ai.operationId have I/O mappings. This differs from the Cloudflare wrapper example, which emits GenAI attributes. |
| LiteLLM, Google ADK, Logfire instrumentation for OpenAI Agents, and Claude Code | Agent Observability can infer span kinds for some legacy span formats. This inference requires enablement for your organization. Contact Datadog support to confirm access. |
| Strands and Spring AI (through Micrometer) | For Strands, Agent Observability parses provider and model names. For Spring AI through Micrometer, it supports numeric attributes sent as strings and structured message content. |
Span kind inference for legacy Vercel AI SDK spans requires enablement for your organization. Contact Datadog support to confirm access. If your instrumentation can send supported gen_ai.operation.name values, use them instead of relying on span kind inference.
If a span is missing an expected attribute, or an attribute can’t be parsed, Agent Observability flags the span with a Mapping warnings indicator. See Troubleshooting mapping warnings for descriptions and fixes.
OpenTelemetry 1.37+ attribute mappings
Base span attributes
| OTLP Field | Agent Observability Field | Notes |
|---|
resource.attributes.service.name | ml_app, tags.service | |
name | name | Overridden by gen_ai.tool.name if present |
parent_span_id | parent_id | |
start_time_unix_nano | start_ns | |
end_time_unix_nano | duration | Calculated: end - start |
status.code | status | error if > 0, else ok |
status.message | meta.error.message | |
attributes.error.type | meta.error.type | |
Span kind resolution
gen_ai.operation.name | Agent Observability span.kind |
|---|
generate_content, chat, text_completion, completion | llm |
embeddings, embedding | embedding |
execute_tool | tool |
invoke_agent, create_agent | agent |
rerank, unknown, (default) | workflow |
| OTel Attribute | Agent Observability Field | Notes |
|---|
gen_ai.operation.name | meta.span.kind | See resolution table above |
gen_ai.provider.name | meta.model_provider | Falls back to gen_ai.system, then custom |
gen_ai.response.model | meta.model_name | |
gen_ai.request.model | meta.model_name | Fallback if response.model absent |
Token usage metrics
| OTel Attribute | Agent Observability Field |
|---|
gen_ai.usage.input_tokens | metrics.input_tokens |
gen_ai.usage.output_tokens | metrics.output_tokens |
gen_ai.usage.prompt_tokens | metrics.prompt_tokens |
gen_ai.usage.completion_tokens | metrics.completion_tokens |
gen_ai.usage.total_tokens | metrics.total_tokens |
Cost metrics
For supported models and providers, Datadog can estimate costs from token usage without cost attributes. Supply costs directly for unsupported models or custom pricing.
Cost attributes use two paths: supplied costs that Datadog preserves, and fallback estimates that automatic pricing can replace. Both paths populate the same metrics.estimated_*_cost fields, but Datadog preserves supplied costs despite the estimated_ prefix.
Supplied cost attributes
The following cost attributes are extensions supported by Datadog. Send non-negative amounts in USD: send 0.01 for a $0.01 cost. Automatic pricing does not overwrite these supplied cost fields, including zero. Datadog can still estimate cost fields you omit.
| OTel Attribute | Agent Observability Field |
|---|
gen_ai.usage.cost.amount | metrics.estimated_total_cost |
gen_ai.usage.cost | metrics.estimated_total_cost |
gen_ai.cost.total_cost | metrics.estimated_total_cost |
gen_ai.cost.input_cost | metrics.estimated_input_cost |
gen_ai.cost.output_cost | metrics.estimated_output_cost |
llm.cost.total | metrics.estimated_total_cost |
llm.cost.prompt | metrics.estimated_input_cost |
llm.cost.completion | metrics.estimated_output_cost |
llm.cost.prompt_details.input | metrics.estimated_non_cached_input_cost |
llm.cost.prompt_details.cache_read | metrics.estimated_cache_read_input_cost |
llm.cost.prompt_details.cache_write | metrics.estimated_cache_write_input_cost |
llm.cost.completion_details.reasoning | metrics.estimated_reasoning_output_cost |
Set gen_ai.usage.cost.currency to USD when using gen_ai.usage.cost.amount. This amount/currency format is an OpenTelemetry proposal. The other attributes use their producers’ USD conventions. Datadog ignores these USD cost attributes if they declare a currency other than USD.
If several USD attributes from the table above supply the same cost field, Datadog uses the first valid value. The precedence order is: gen_ai.usage.cost.amount, gen_ai.usage.cost, gen_ai.cost.*, then llm.cost.*. Zero is a valid supplied cost.
Fallback estimated cost attributes (nanodollars)
These Datadog-supported extensions accept optional fallback estimates in nanodollars, without unit conversion. For example, send 10000000 for a $0.01 estimate. Datadog replaces a supplied estimate when it calculates a positive cost for that field, even from partial token counts. Otherwise, Datadog keeps the supplied estimate. USD costs from the preceding table take precedence over these nanodollar estimates.
| OTel Attribute | Agent Observability Field |
|---|
gen_ai.cost.estimated_total | metrics.estimated_total_cost |
gen_ai.cost.estimated_input | metrics.estimated_input_cost |
gen_ai.cost.estimated_output | metrics.estimated_output_cost |
gen_ai.cost.estimated_cache_read_input | metrics.estimated_cache_read_input_cost |
gen_ai.cost.estimated_cache_write_input | metrics.estimated_cache_write_input_cost |
gen_ai.cost.estimated_non_cached_input | metrics.estimated_non_cached_input_cost |
gen_ai.cost.estimated_reasoning_output | metrics.estimated_reasoning_output_cost |
Request parameters
All gen_ai.request.* parameters map to meta.metadata.* with the prefix stripped.
| OTel Attribute | Agent Observability Field |
|---|
gen_ai.request.seed | metadata.seed |
gen_ai.request.frequency_penalty | metadata.frequency_penalty |
gen_ai.request.max_tokens | metadata.max_tokens |
gen_ai.request.stop_sequences | metadata.stop_sequences |
gen_ai.request.temperature | metadata.temperature |
gen_ai.request.top_k | metadata.top_k |
gen_ai.request.top_p | metadata.top_p |
gen_ai.request.choice.count | metadata.choice.count |
| OTel Attribute | Agent Observability Field | Notes |
|---|
gen_ai.tool.name | name | Overrides span name |
gen_ai.tool.call.id | metadata.tool_id | |
gen_ai.tool.description | metadata.tool_description | |
gen_ai.tool.type | metadata.tool_type | |
gen_ai.tool.definitions | meta.tool_definitions | Parsed JSON array |
gen_ai.tool.call.arguments | input.value | |
gen_ai.tool.call.result | output.value | |
Session and conversation
| OTel Attribute | Agent Observability Field | Notes |
|---|
gen_ai.conversation.id | session_id | Also added to metadata.conversation_id and tags |
When an APM trace’s top-most span is not a gen_ai span (for example, an HTTP handler that invokes several LLMs in parallel), Agent Observability produces a separate Agent Observability trace for each top-level gen_ai span in that APM trace. To keep these split traces grouped together in the UI, set gen_ai.conversation.id to the same value on each gen_ai span within the APM trace: Agent Observability groups by session_id, so the resulting traces appear together even though they have distinct Agent Observability trace IDs. This is the same attribute used for cross-request conversation grouping.
Span links
Span links you set on a GenAI span appear as span_links on the corresponding Agent Observability span.
| OTel span link field | Agent Observability Field | Notes |
|---|
trace_id | span_links[].trace_id | 128-bit trace IDs are emitted as hex. A link to a span in the same trace resolves to that span’s Agent Observability trace ID. |
span_id | span_links[].span_id | Decimal |
attributes | span_links[].attributes | Dots in attribute keys are replaced with underscores (for example, messaging.operation becomes messaging_operation). |
Links between spans in the same trace are drawn as edges in that trace’s Execution Graph.
Response attributes
| OTel Attribute | Agent Observability Field |
|---|
gen_ai.response.model | meta.model_name |
gen_ai.response.finish_reasons | metadata.finish_reasons |
Input and output messages are extracted from the following sources, in priority order:
- Direct attributes:
gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions - Span events (
meta["events"]) with name gen_ai.client.inference.operation.details - Older message events:
gen_ai.system.message, gen_ai.user.message, gen_ai.assistant.message, gen_ai.tool.message, and gen_ai.choice. These fill missing I/O fields; newer message formats take precedence.
| OTel Source | Agent Observability Field | Notes |
|---|
gen_ai.input.messages | meta.input.messages (llm) / meta.input.value (others) | |
gen_ai.output.messages | meta.output.messages (llm) / meta.output.value (others) | |
gen_ai.system_instructions | Prepended to input | Added as system role messages |
Message parts that carry media are extracted into the typed audio_parts and image_parts fields on the message:
| Part type | Behavior |
|---|
blob with mime_type and inline bytes | Extracted to image_parts or audio_parts. When the part omits modality, it is inferred from the MIME type. |
uri carrying a base64 image data URI, such as data:image/png;base64,... | Extracted to image_parts. |
uri carrying a remote URL | Recorded as the text reference [<modality>: <uri>]. The URL is not fetched. |
file with a file_id | Recorded as the text reference [<modality> file: <file_id>]. |
A positional marker such as [image blob: image/png] is also added to the message text so that media keeps its place among the other parts.
Audio reaches audio_parts through blob parts only. An audio data URI on a uri part is recorded as text, and the conventions specify blob as the part type for inline base64 data, so prefer blob for both audio and images.
For the formats the trace view renders and the size limits that apply, see Multimodal Support.
Embedding spans
| OTel Source | Agent Observability Field |
|---|
gen_ai.input.messages | meta.input.documents |
| N/A | meta.output.value = [N embedding(s) returned] |
Tags are placed directly on the span:
- Non-
gen_ai.* attributes are converted to key:value tags - Unknown
gen_ai.* keys are added with prefix stripped - Filtered out:
_dd.*, llm.*, ddtags, events, and already specifically mapped gen_ai.* keys
Any gen_ai.* attributes that are not explicitly mapped to Agent Observability span fields are placed in the LLM span's tags, with a 256 character limit per value. Values exceeding this limit are truncated. All other non-gen_ai attributes are dropped.
To add structured metadata to a span’s meta.metadata field instead of its tags, set the _dd.ml_obs.metadata attribute to a JSON object string. Its keys and values (including nested objects and arrays) are merged into meta.metadata and rendered as JSON in the UI.
import json
span.set_attribute("_dd.ml_obs.metadata", json.dumps({
"experiment": "a/b",
"config": {
"retry": {"max": 3, "backoff": "exp"},
"feature_flags": ["new_ranker", "fast_path"],
},
}))
Notes:
- Values may be arbitrarily nested; unlike tags, metadata is not subject to the 256 character per-value limit.
- Keys that collide with metadata derived from
gen_ai.* attributes (for example, temperature) are overwritten by your values, with the exception of model_name and model_provider, which are reserved. - The value must be a JSON object. A value that is not valid JSON, or that is a top-level array or scalar, is dropped.
OpenLLMetry attribute mappings
This section documents OpenLLMetry-specific attribute mappings that differ from or extend the standard OpenTelemetry GenAI semantic conventions.
Span kind resolution
llm.request.type is used as a fallback when gen_ai.operation.name is absent.
llm.request.type | Agent Observability span.kind |
|---|
chat | llm |
completion | llm |
embedding | embedding |
rerank | workflow |
unknown, (default) | workflow |
| OpenLLMetry Attribute | Agent Observability Field | Notes |
|---|
gen_ai.system | meta.model_provider | Fallback when gen_ai.provider.name absent |
Token usage metrics
| OpenLLMetry Attribute | Agent Observability Field | Notes |
|---|
llm.usage.total_tokens | metrics.total_tokens | Fallback when gen_ai.usage.total_tokens absent |
OpenLLMetry uses indexed attributes instead of JSON arrays. These are the lowest priority source and are only used when no OTel standard sources exist.
| OpenLLMetry Attribute | Description |
|---|
gen_ai.prompt.<index>.role | Message role (user, system, assistant, tool) |
gen_ai.prompt.<index>.content | Message content |
gen_ai.prompt.<index>.tool_call_id | Tool call ID for tool response messages |
Completion attributes (output)
| OpenLLMetry Attribute | Description |
|---|
gen_ai.completion.<index>.role | Message role |
gen_ai.completion.<index>.content | Message content |
gen_ai.completion.<index>.finish_reason | Completion finish reason |
Mapping
Messages are converted to OTel-compatible format and processed normally:
| OpenLLMetry Source | LLMObs Field |
|---|
gen_ai.prompt.* | meta.input.messages (llm) / meta.input.value (others) |
gen_ai.completion.* | meta.output.messages (llm) / meta.output.value (others) |
Tool calls are nested within completion attributes.
| OpenLLMetry Attribute | Maps To |
|---|
gen_ai.completion.<index>.tool_calls.<idx>.name | tool_calls[].name |
gen_ai.completion.<index>.tool_calls.<idx>.id | tool_calls[].tool_id |
gen_ai.completion.<index>.tool_calls.<idx>.arguments | tool_calls[].arguments |
When role = "tool" and tool_call_id are present, the message is converted to a tool result:
| OpenLLMetry Attribute | Maps To |
|---|
gen_ai.prompt.<index>.tool_call_id | tool_results[].tool_id |
gen_ai.prompt.<index>.content | tool_results[].result |
Embedding spans
For embedding spans, documents are extracted from prompt content attributes.
| OpenLLMetry Source | Agent Observability Field |
|---|
gen_ai.prompt.<index>.content | meta.input.documents[].text |
The following OpenLLMetry-specific attributes are filtered from tags:
gen_ai.prompt.*gen_ai.completion.*llm.*
OpenInference attribute mappings
Agent Observability recognizes an OpenInference span when the openinference.span.kind attribute is present and non-empty. The following sections document the OpenInference attributes that map to dedicated Agent Observability fields.
Span kind resolution
If both gen_ai.operation.name and openinference.span.kind are present, gen_ai.operation.name takes precedence.
openinference.span.kind | Agent Observability span.kind |
|---|
LLM | llm |
EMBEDDING | embedding |
TOOL | tool |
AGENT | agent |
RETRIEVER | retrieval |
CHAIN, RERANKER, GUARDRAIL, EVALUATOR, PROMPT, other values | workflow |
| OpenInference Attribute | Agent Observability Field | Notes |
|---|
llm.provider | meta.model_provider | Preferred OpenInference provider source |
llm.system | meta.model_provider | Fallback when llm.provider is absent |
llm.model_name | meta.model_name | |
embedding.model_name | meta.model_name | Fallback for embedding spans |
For llm and embedding spans, missing provider or model values are set to unknown.
Token usage metrics
| OpenInference Attribute | Agent Observability Field |
|---|
llm.token_count.prompt | metrics.prompt_tokens |
llm.token_count.completion | metrics.completion_tokens |
llm.token_count.total | metrics.total_tokens |
llm.token_count.prompt_details.cache_read | metrics.cache_read_input_tokens |
llm.token_count.prompt_details.cache_write | metrics.cache_write_input_tokens |
llm.token_count.completion_details.reasoning | metrics.reasoning_output_tokens |
| OpenInference Attribute | Agent Observability Field | Notes |
|---|
session.id | session_id | Also adds session_id and conversation_id tags and propagates the session to the trace root |
user.id | tags | Added as user_id:<value> |
tag.tags | tags | Each list item becomes a span tag |
llm.invocation_parameters | meta.metadata | Parsed as a JSON object |
metadata | meta.metadata | Parsed as a JSON object |
Reserved Agent Observability fields in llm.invocation_parameters and metadata do not override dedicated span fields.
| OpenInference Attribute | Agent Observability Field | Notes |
|---|
tool.name | name | Overrides the span name |
tool.id | meta.metadata.tool_id | |
tool.description | meta.metadata.tool_description | |
tool.parameters | meta.metadata.tool_parameters | |
input.value | meta.input.value | Used directly for tool, agent, and workflow spans |
output.value | meta.output.value | Used directly for tool, agent, and workflow spans |
In these attributes, <direction> is input or output.
Input and output are extracted from the following sources, in priority order:
- OpenTelemetry
gen_ai.* direct attributes and span events - OpenLLMetry indexed attributes
- OpenInference indexed attributes
- OpenInference
input.value and output.value
| OpenInference Source | Agent Observability Field |
|---|
llm.input_messages.<index>.* | meta.input.messages (llm) / meta.input.value (other span kinds) |
llm.output_messages.<index>.* | meta.output.messages (llm) / meta.output.value (other span kinds) |
input.value | Input fallback |
output.value | Output fallback |
The following indexed message attributes are supported:
| OpenInference Attribute | Mapping |
|---|
llm.<direction>_messages.<message-index>.message.role | Message role |
llm.<direction>_messages.<message-index>.message.content | Text content |
llm.<direction>_messages.<message-index>.message.contents.<content-index>.message_content.text | Ordered text content |
llm.<direction>_messages.<message-index>.message.contents.<content-index>.message_content.image.image.url | Ordered image content |
llm.<direction>_messages.<message-index>.message.tool_calls.<tool-index>.tool_call.* | Tool call ID, name, and arguments |
llm.<direction>_messages.<message-index>.message.contents.<content-index>.tool_call.* | Tool call within ordered content |
llm.<direction>_messages.<message-index>.message.tool_call_id | Tool result ID when the message role is tool |
Image content maps to an image URI while preserving its position among other message content.
Embedding spans
| OpenInference Source | Agent Observability Field |
|---|
embedding.embeddings.<index>.embedding.text | meta.input.documents[].text |
| N/A | meta.output.value = [N embedding(s) returned] |
Retrieval spans
| OpenInference Source | Agent Observability Field |
|---|
input.value | meta.input.value |
retrieval.documents.<index>.document.content | meta.output.documents[].text |
retrieval.documents.<index>.document.id | meta.output.documents[].id |
retrieval.documents.<index>.document.score | meta.output.documents[].score |
retrieval.documents.<index>.document.metadata | meta.output.documents[].metadata (parsed JSON object) |
OpenInference attributes with llm.*, retrieval.*, embedding.*, and reranker.* prefixes are excluded from tags. Specifically mapped values such as input.value, output.value, metadata, tag.tags, and tool.parameters are also excluded from duplicate tags.
Other non-empty OpenInference attributes with values of 256 characters or fewer are added as key:value tags. The tag.tags list is promoted directly to span tags.
Langfuse attribute mappings
This section documents Langfuse-specific attribute mappings for applications using Langfuse’s native OpenTelemetry instrumentation.
Detection
A span is treated as a Langfuse span when it carries a non-empty langfuse.observation.type attribute. This attribute is also used as a fallback for span kind resolution when gen_ai.operation.name is absent.
Span kind resolution
langfuse.observation.type | Agent Observability span.kind |
|---|
generation | llm |
embedding | embedding |
tool | tool |
agent | agent |
retriever | retrieval |
span, event, chain, evaluator, guardrail, (default) | workflow |
| Langfuse Attribute | Agent Observability Field | Notes |
|---|
langfuse.observation.metadata.ls_provider | meta.model_provider | |
langfuse.observation.model.name | meta.model_name | Fallback when gen_ai.response.model and gen_ai.request.model are absent |
Token usage metrics
langfuse.observation.usage_details is a JSON object. Each key maps to an Agent Observability metric, used as a fallback for any metric not already set from gen_ai.usage.* attributes:
| Langfuse Usage Key | Agent Observability Field |
|---|
input, input_tokens | metrics.input_tokens |
output, output_tokens | metrics.output_tokens |
total, total_tokens | metrics.total_tokens |
prompt_tokens | metrics.prompt_tokens |
completion_tokens | metrics.completion_tokens |
cache_creation_input_tokens | metrics.cache_write_input_tokens |
cache_read_input_tokens, cached_tokens | metrics.cache_read_input_tokens |
reasoning_tokens | metrics.reasoning_output_tokens |
langfuse.observation.input and langfuse.observation.output carry a JSON-encoded value that can be a chat-message array, a single message object, or arbitrary JSON/string content. These are the lowest-priority sources and are only used when no gen_ai.* message attributes exist.
Each message is converted to the parts-based message shape:
- A
content string becomes a text part. - A
content array of blocks converts image_url blocks to uri parts and text blocks to text parts; any other block is kept as serialized text. - A
tool_calls array on a message becomes tool_call parts. - A message with
role: tool and a tool_call_id becomes a tool_result part.
For tool, agent, and workflow spans, langfuse.observation.input/langfuse.observation.output are used directly as input.value/output.value, after the standard gen_ai.tool.call.* fallback.
Retrieval spans
For retrieval spans, langfuse.observation.input is used as the query value in meta.input.value. langfuse.observation.output is parsed as a document collection (an array of document objects, an array of strings, or a single document object) into meta.output.documents. Each document object’s text, content, or page_content key (checked in that order) maps to text, along with any id, score, and metadata keys.
Embedding spans
For embedding spans, langfuse.observation.input is parsed the same way as retrieval documents into meta.input.documents[].text, keeping only documents that carry non-empty text.
| Langfuse Attribute | Agent Observability Field | Notes |
|---|
langfuse.session.id | session_id | Fallback when gen_ai.conversation.id is absent |
langfuse.user.id | user_id: tag | Fallback when the standard user ID attribute is absent |
langfuse.observation.model.parameters | meta.metadata.* | JSON object, merged into metadata, skipping reserved keys (model_name, model_provider) |
langfuse.trace.metadata.*, langfuse.observation.metadata.* | meta.metadata.* | Prefix stripped, merged into metadata, skipping reserved keys |
langfuse.trace.tags | Appended to tags | JSON array of strings |
The following Langfuse-specific attributes are filtered from tags because they’re consumed elsewhere:
langfuse.internal.*, langfuse.observation.metadata.*, langfuse.trace.metadata.* (prefixes)langfuse.observation.input, langfuse.observation.output, langfuse.observation.model.name, langfuse.observation.model.parameters, langfuse.observation.usage_details, langfuse.observation.cost_details, langfuse.observation.completion_start_timelangfuse.trace.input, langfuse.trace.output, langfuse.trace.metadata, langfuse.trace.tagslangfuse.experiment.item.expected_output, langfuse.experiment.item.metadata, langfuse.experiment.metadata
Troubleshooting mapping warnings
At ingestion, Agent Observability detects span mapping problems such as missing input messages or an unparsable token count. It flags affected spans instead of failing silently.
Flagged spans display a Mapping warnings indicator in the span detail panel. Select the indicator to open the span’s mapping warnings. Each entry lists the affected attribute, a suggested fix, and a link to the attribute mapping reference.
Each warning also includes a short description of its downstream effect, such as skipped evaluations or unavailable cost estimation.
Mapping warnings are detected at ingestion. They do not affect billing or span retention.
Mapping warning reference
A span can display warnings not listed in the following table if Datadog adds checks. These render with a generated title based on the check that triggered the warning.
Search value shows the value stored on the span. Use it with the @collection_errors attribute to find flagged spans, for example @collection_errors:otel_warning_missing_model_name.
| Warning | Search value (@collection_errors:) | Attribute | Fix |
|---|
| Missing model name | otel_warning_missing_model_name | Expected gen_ai.response.model | Emit gen_ai.response.model directly, so the model name doesn’t need to be parsed out of gen_ai.request.model. |
| Missing model provider | otel_warning_missing_model_provider | Expected gen_ai.provider.name | Emit gen_ai.provider.name directly, so the provider doesn’t need to be inferred from gen_ai.system. |
| Malformed model identifier | otel_warning_strands_model_malformed | On gen_ai.request.model | Emit gen_ai.response.model directly, so the model name doesn’t need to be parsed out of gen_ai.request.model. |
| Malformed input | otel_warning_input_malformed | On gen_ai.input.messages | Emit gen_ai.input.messages as a valid JSON array of messages. |
| Malformed output | otel_warning_output_malformed | On gen_ai.output.messages | Emit gen_ai.output.messages as a valid JSON array of messages. |
| Malformed message | otel_warning_message_malformed | On gen_ai.input.messages | Emit each message with a role and content field. |
| Empty message content | otel_warning_invalid_parts | On gen_ai.output.messages | Emit a content string, or parts entries with a recognized type, for example text, tool_call, tool_call_response. |
| Missing embedding input | otel_warning_embedding_input_missing | Expected gen_ai.input.messages | Set gen_ai.input.messages to the embedded text. |
| Malformed embedding input | otel_warning_embedding_input_malformed | On gen_ai.input.messages | Emit gen_ai.input.messages as a valid JSON array of messages. |
| Unreadable token counts | otel_warning_token_usage_unparseable | On gen_ai.usage.input_tokens | Emit token counts as integers, not strings or objects. |
| Invalid token counts | otel_warning_token_usage_invalid | On gen_ai.usage.input_tokens | Emit non-negative integer token counts. |
| Unreadable cost metrics | otel_warning_cost_metrics_unparseable | On cost attributes | Emit cost metrics as integers or floats, not strings or objects. |
| Invalid cost metrics | otel_warning_cost_metrics_invalid | On cost attributes | Emit non-negative cost values. |
| Invalid cost currency | otel_warning_cost_currency_invalid | On USD cost attributes | Set gen_ai.usage.cost.currency to USD. This attribute is required for gen_ai.usage.cost.amount. |
| Malformed invocation parameters | otel_warning_params_malformed | On invocation parameters | Emit invocation parameters as a valid JSON object, or set them individually as gen_ai.request.* attributes (such as temperature, top_p, and max_tokens). |
| Malformed tool definitions | otel_warning_tool_definitions_malformed | On gen_ai.tool.definitions | Emit gen_ai.tool.definitions as a valid JSON array of tool definitions. |
| Malformed tool definition | otel_warning_tool_definition_entry_malformed | On gen_ai.tool.definitions | Give each tool definition a name, and make its parameters a JSON object. |
| Missing tool name | otel_warning_tool_span_name_missing | Expected gen_ai.tool.name | Set gen_ai.tool.name on tool spans. |
| Missing tool call name | otel_warning_tool_call_name_missing | On gen_ai.output.messages | Give each tool call a name. |
| Missing operation name | otel_warning_operation_missing | Expected gen_ai.operation.name | Set gen_ai.operation.name to one of chat, text_completion, embeddings, execute_tool, invoke_agent, or retriever. |
| Malformed span events | otel_warning_failed_to_parse_span_events | On events | Emit valid JSON in span events, or set gen_ai.input.messages and gen_ai.output.messages directly. |
| Unreadable document score | otel_warning_failed_to_parse_document_score | On output.documents | Emit each document score as a number. |
| Malformed document metadata | otel_warning_document_metadata_malformed | On output.documents | Emit each document’s metadata as a JSON object. |
| Unrecognized instrumentation | otel_warning_spec_version_unknown | Expected gen_ai.operation.name | Set gen_ai.operation.name and gen_ai.system so Datadog can identify the instrumentation. |
Find raw span attributes for a flagged span
To find the raw attributes Datadog received for a flagged span:
- If APM is enabled, the same data is also written to the corresponding APM trace. Open the linked APM trace to inspect the raw span attributes.
- Compare the raw attributes against the expected attribute from the mapping warning reference or the full attribute mapping reference. See Correlating Agent Observability and APM for how the two views relate.
- For a malformed warning, check whether the attribute value is valid JSON. Common causes include double-encoding, truncation, and emitting a string or other non-object value where an object is expected.
- For a missing attribute warning, check your instrumentation library version against the Tested frameworks and libraries table, and confirm
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental is set if your library requires it.
Supported semantic conventions
Agent Observability supports spans that follow the OpenTelemetry 1.37+ semantic conventions for generative AI, including:
- LLM operations with
gen_ai.provider.name, "gen_ai.operation.name", gen_ai.request.model, and other gen_ai attributes - Operation inputs/outputs on direct span attributes or via span events
- Token usage metrics (
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) - Model parameters and metadata
For the complete list of supported attributes and their specifications, see the OpenTelemetry semantic conventions for generative AI documentation.