Overview

By using OpenTelemetry’s standardized semantic conventions for generative AI operations, you can instrument your LLM applications with any OpenTelemetry-compatible library or framework and visualize the traces in Agent Observability.

Agent Observability supports ingesting OpenTelemetry traces that follow either the OpenTelemetry 1.37+ semantic conventions for generative AI or the supported OpenInference semantic conventions. This allows you to send LLM traces directly from OpenTelemetry-instrumented applications to Datadog without requiring the Agent Observability SDK or a Datadog Agent.

Prerequisites

• A Datadog API key
• An application instrumented with OpenTelemetry that emits traces following the OpenTelemetry 1.37+ semantic conventions for generative AI or the supported OpenInference semantic conventions

Supported features

Evaluations

To send external evaluations directly to the API for OpenTelemetry spans, include the source:otel tag in the evaluation. When referencing spans, provide span_id and trace_id as decimal strings. OpenTelemetry uses hexadecimal IDs natively, so convert them to decimal before submitting evaluations. For example, use Python’s int(hex_span_id, 16) to convert a hex span ID to its decimal equivalent.

Prompt Tracking

For information on using Prompt Tracking with OpenTelemetry spans, see Prompt Tracking - OpenTelemetry Instrumentation.

Experiments

You can use OpenTelemetry spans inside Agent Observability Experiments. By setting DD_TRACE_OTEL_ENABLED=1, OTel spans created inside an experiment task automatically appear as children of the experiment span.

Multimodal support

Audio and images on OpenTelemetry messages are rendered in the trace view. Datadog extracts media from message parts that follow the OpenTelemetry GenAI semantic conventions, as described in Media in messages.

Only media carried inline as base64 bytes is rendered. A remote URL is recorded as a text reference and is never fetched. For the fields, formats, and size limits that apply once media reaches a span, see Multimodal Support.

Span links

Use OpenTelemetry span links on your GenAI spans to express non-parent-child relationships, such as when one span’s output feeds another span’s input. When two linked spans are in the same trace, the link appears as an edge in that trace’s Execution Graph, so you can see how data flows between sibling spans (for example, a tool’s output feeding a downstream LLM call).

Use from and to attributes to indicate the direction of the data flow:

from opentelemetry import trace
from opentelemetry.trace import Link

tracer = trace.get_tracer(__name__)

# A tool span whose output feeds a downstream LLM call.
with tracer.start_as_current_span("lookup_order") as tool_span:
    tool_span.set_attribute("gen_ai.operation.name", "execute_tool")
    tool_ctx = tool_span.get_span_context()

# The LLM span links back to the tool span: its output became this span's input.
link = Link(context=tool_ctx, attributes={"from": "output", "to": "input"})
with tracer.start_as_current_span("chat gpt-4o", links=[link]) as llm_span:
    llm_span.set_attribute("gen_ai.operation.name", "chat")

Setup

Any method Datadog supports for ingesting OpenTelemetry traces works with Agent Observability. For the full list of supported ingestion paths, see OpenTelemetry feature compatibility. The following is one way to configure it.

To send OpenTelemetry traces to Agent Observability, configure your OpenTelemetry exporter with the following settings:

Configuration

Set the following environment variables in your application:

OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=
OTEL_EXPORTER_OTLP_TRACES_HEADERS=dd-api-key=<YOUR_API_KEY>,dd-otlp-source=llmobs

Replace <YOUR_API_KEY> with your Datadog API key.

If your framework previously supported a pre-1.37 OpenTelemetry specification version, you also need to set:

OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental

This environment variable enables version 1.37+-compliant OpenTelemetry traces for frameworks that now support the version 1.37+ semantic conventions, but previously supported older versions (such as strands-agents).

Note:

• If you are using an OpenTelemetry library other than the default OpenTelemetry SDK, you may need to configure the endpoint, protocol, and headers differently depending on the library’s API. See your library’s documentation for the appropriate configuration method.
• When using OpenTelemetry instrumentation, some data sent to Agent Observability may also be written to the corresponding APM traces. If you are protecting sensitive data, consider also configuring a Restricted Dataset on APM to match your Agent Observability access controls. See Data Access Control for more information.

Using strands-agents

If you are using the strands-agents library, you need to set an additional environment variable to enable traces that are compliant with OpenTelemetry v1.37+:

OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental

This environment variable ensures that strands-agents emits traces following the OpenTelemetry v1.37+ semantic conventions for generative AI, which are required by Agent Observability.

Instrumentation

To generate traces compatible with Agent Observability, do one of the following:

• Use an OpenTelemetry library or instrumentation package that emits spans following the OpenTelemetry 1.37+ semantic conventions for generative AI or the supported OpenInference semantic conventions.
• Create custom OpenTelemetry instrumentation that produces the required gen_ai.* or OpenInference attributes defined by your chosen convention.

After your application starts sending data, the traces automatically appear in the Agent Observability Traces page. To search for your traces in the UI, use the ml_app attribute, which is automatically set to the value of your OpenTelemetry root span’s service attribute.

Tested frameworks and libraries

These frameworks and libraries have been tested with Agent Observability. Frameworks that emit the supported attributes from the OpenTelemetry 1.37+ GenAI semantic conventions or OpenInference semantic conventions can send spans to Agent Observability.

Examples

Using Strands Agents

The following example demonstrates a complete application using Strands Agents with the OpenTelemetry integration. This same approach works with any framework that supports OpenTelemetry version 1.37+ semantic conventions for generative AI.

from strands import Agent
from strands_tools import calculator, current_time
from strands.telemetry.config import StrandsTelemetry
import os

# Configure AWS credentials for Bedrock access
os.environ["AWS_PROFILE"] = "<YOUR_AWS_PROFILE>"
os.environ["AWS_DEFAULT_REGION"] = "<YOUR_AWS_REGION>"

# Enable latest GenAI semantic conventions (1.37)
os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = "gen_ai_latest_experimental"

# Configure OTLP endpoint to send traces to Agent Observability
os.environ["OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"] = "http/protobuf"
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = ""
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = f"dd-api-key={os.getenv('DD_API_KEY')},dd-otlp-source=llmobs"

# Initialize telemetry with OTLP exporter
telemetry = StrandsTelemetry()
telemetry.setup_otlp_exporter()

# Create agent with tools
agent = Agent(tools=[calculator, current_time])

# Run the agent
if __name__ == "__main__":
    result = agent("I was born in 1993, what is my age?")
    print(f"Agent: {result}")

Custom OpenTelemetry instrumentation

The following example demonstrates how to instrument your LLM application using custom OpenTelemetry code. This approach gives you full control over the traces and spans emitted by your application.

import os
import json
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from openai import OpenAI

# Configure OpenTelemetry to send traces to Datadog
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = ""
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = "dd-api-key=<YOUR_DATADOG_API_KEY>,dd-otlp-source=llmobs"
os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = "gen_ai_latest_experimental"

# Initialize OpenTelemetry SDK
resource = Resource(attributes={SERVICE_NAME: "simple-llm-example"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Make LLM call with OpenTelemetry tracing
with tracer.start_as_current_span(
    "chat gpt-4o",
    kind=trace.SpanKind.CLIENT,
) as span:
    model = "gpt-4o"
    max_tokens = 1024
    temperature = 0.7
    messages = [{"role": "user", "content": "Explain OpenTelemetry in one sentence."}]

    # Set request attributes
    span.set_attribute("gen_ai.provider.name", "openai")
    span.set_attribute("gen_ai.request.model", model)
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.max_tokens", max_tokens)
    span.set_attribute("gen_ai.request.temperature", temperature)

    # Add input messages as event
    input_messages_parts = []
    for msg in messages:
        input_messages_parts.append({
            "role": msg["role"],
            "parts": [{"type": "text", "content": msg["content"]}]
        })

    span.add_event(
        "gen_ai.client.inference.operation.details",
        {
            "gen_ai.input.messages": json.dumps(input_messages_parts)
        }
    )

    # Make actual LLM call
    client = OpenAI(api_key="<YOUR_OPENAI_API_KEY>")
    response = client.chat.completions.create(
        model=model,
        max_tokens=max_tokens,
        temperature=temperature,
        messages=messages
    )

    # Set response attributes from actual data
    span.set_attribute("gen_ai.response.id", response.id)
    span.set_attribute("gen_ai.response.model", response.model)
    span.set_attribute("gen_ai.response.finish_reasons", [response.choices[0].finish_reason])
    span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)

    # Add output messages as event
    output_text = response.choices[0].message.content
    span.add_event(
        "gen_ai.client.inference.operation.details",
        {
            "gen_ai.output.messages": json.dumps([{
                "role": "assistant",
                "parts": [{"type": "text", "content": output_text}],
                "finish_reason": response.choices[0].finish_reason
            }])
        }
    )

    print(f"Response: {output_text}")

# Flush spans before exit
provider.force_flush()

After running this example, search for ml_app:simple-llm-example in the Agent Observability UI to find the generated trace.

Using OpenLLMetry

The following example demonstrates using OpenLLMetry to automatically instrument OpenAI calls with OpenTelemetry.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
import openai
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "simple-openllmetry-test",
})

provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

exporter = OTLPSpanExporter(
    endpoint="",
    headers={
        "dd-api-key": "<YOUR_DATADOG_API_KEY>",
        "dd-ml-app": "simple-openllmetry-test",
        "dd-otlp-source": "llmobs",
    },
)

provider.add_span_processor(BatchSpanProcessor(exporter))

OpenAIInstrumentor().instrument()

# Make OpenAI call (automatically traced)
client = openai.OpenAI(api_key="<YOUR_OPENAI_API_KEY>")
client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "What is 15 multiplied by 7?"}]
)

provider.force_flush(timeout_millis=5000)

After running this example, search for ml_app:simple-openllmetry-test in the Agent Observability UI to find the generated trace.

Using OpenInference

The following example uses the OpenInference OpenAI instrumentation to automatically instrument OpenAI calls with OpenTelemetry.

Configure the OpenTelemetry exporter and instrument the OpenAI client:

import openai
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

resource = Resource.create({
    "service.name": "simple-openinference-test",
})

provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

exporter = OTLPSpanExporter(
    endpoint="",
    headers={
        "dd-api-key": "<YOUR_DATADOG_API_KEY>",
        "dd-ml-app": "simple-openinference-test",
        "dd-otlp-source": "llmobs",
    },
)

provider.add_span_processor(BatchSpanProcessor(exporter))

OpenAIInstrumentor().instrument(tracer_provider=provider)

# Make OpenAI call (automatically traced)
client = openai.OpenAI(api_key="<YOUR_OPENAI_API_KEY>")
client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "What is 15 multiplied by 7?"}]
)

provider.force_flush(timeout_millis=5000)

After running this example, search for ml_app:simple-openinference-test in the Agent Observability UI to find the generated trace.

Attribute mapping reference

This section provides the mappings from OpenTelemetry GenAI semantic conventions (v1.37+), OpenLLMetry, OpenInference, and Langfuse to Datadog’s Agent Observability span schema.

Agent Observability requires at least one of these attributes on a span. If no span in a trace has one, the whole trace is dropped before reaching Agent Observability:

• gen_ai.operation.name
• gen_ai.provider.name
• openinference.span.kind
• langfuse.observation.type
• gen_ai.system (deprecated)

Spans without any gen_ai.* attribute are also dropped individually, even in a trace that qualifies.

If a span is missing an expected attribute, or an attribute can’t be parsed, Agent Observability flags the span with a Mapping warnings indicator. See Troubleshooting mapping warnings for descriptions and fixes.

OpenTelemetry 1.37+ attribute mappings

Base span attributes

Span kind resolution

Model information

Token usage metrics

Request parameters

All gen_ai.request.* parameters map to meta.metadata.* with the prefix stripped.

Tool attributes

Session and conversation

When an APM trace’s top-most span is not a gen_ai span (for example, an HTTP handler that invokes several LLMs in parallel), Agent Observability produces a separate Agent Observability trace for each top-level gen_ai span in that APM trace. To keep these split traces grouped together in the UI, set gen_ai.conversation.id to the same value on each gen_ai span within the APM trace: Agent Observability groups by session_id, so the resulting traces appear together even though they have distinct Agent Observability trace IDs. This is the same attribute used for cross-request conversation grouping.

Span links

Span links you set on a GenAI span appear as span_links on the corresponding Agent Observability span.

Links between spans in the same trace are drawn as edges in that trace’s Execution Graph.

Response attributes

Input and output messages

Input and output messages are extracted from the following sources, in priority order:

1. Direct attributes: gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions
2. Span events (meta["events"]) with name gen_ai.client.inference.operation.details

Media in messages

Message parts that carry media are extracted into the typed audio_parts and image_parts fields on the message:

A positional marker such as [image blob: image/png] is also added to the message text so that media keeps its place among the other parts.

Audio reaches audio_parts through blob parts only. An audio data URI on a uri part is recorded as text, and the conventions specify blob as the part type for inline base64 data, so prefer blob for both audio and images.

For the formats the trace view renders and the size limits that apply, see Multimodal Support.

Embedding spans

Tags

Tags are placed directly on the span:

• Non-gen_ai.* attributes are converted to key:value tags
• Unknown gen_ai.* keys are added with prefix stripped
• Filtered out: _dd.*, llm.*, ddtags, events, and already specifically mapped gen_ai.* keys

Custom metadata

To add structured metadata to a span’s meta.metadata field instead of its tags, set the _dd.ml_obs.metadata attribute to a JSON object string. Its keys and values (including nested objects and arrays) are merged into meta.metadata and rendered as JSON in the UI.

import json

span.set_attribute("_dd.ml_obs.metadata", json.dumps({
    "experiment": "a/b",
    "config": {
        "retry": {"max": 3, "backoff": "exp"},
        "feature_flags": ["new_ranker", "fast_path"],
    },
}))

Notes:

• Values may be arbitrarily nested; unlike tags, metadata is not subject to the 256 character per-value limit.
• Keys that collide with metadata derived from gen_ai.* attributes (for example, temperature) are overwritten by your values, with the exception of model_name and model_provider, which are reserved.
• The value must be a JSON object. A value that is not valid JSON, or that is a top-level array or scalar, is dropped.

OpenLLMetry attribute mappings

This section documents OpenLLMetry-specific attribute mappings that differ from or extend the standard OpenTelemetry GenAI semantic conventions.

Span kind resolution

llm.request.type is used as a fallback when gen_ai.operation.name is absent.

Model information

Token usage metrics

Input and output messages

OpenLLMetry uses indexed attributes instead of JSON arrays. These are the lowest priority source and are only used when no OTel standard sources exist.

Prompt attributes (input)

Completion attributes (output)

Mapping

Messages are converted to OTel-compatible format and processed normally:

Tool calls

Tool calls are nested within completion attributes.

Tool response messages

When role = "tool" and tool_call_id are present, the message is converted to a tool result:

Embedding spans

For embedding spans, documents are extracted from prompt content attributes.

Tags filtering

The following OpenLLMetry-specific attributes are filtered from tags:

• gen_ai.prompt.*
• gen_ai.completion.*
• llm.*

OpenInference attribute mappings

Agent Observability recognizes an OpenInference span when the openinference.span.kind attribute is present and non-empty. The following sections document the OpenInference attributes that map to dedicated Agent Observability fields.

Span kind resolution

If both gen_ai.operation.name and openinference.span.kind are present, gen_ai.operation.name takes precedence.

Model information

For llm and embedding spans, missing provider or model values are set to unknown.

Token usage metrics

Session, user, and metadata

Reserved Agent Observability fields in llm.invocation_parameters and metadata do not override dedicated span fields.

Tool attributes

Input and output messages

In these attributes, <direction> is input or output. Input and output are extracted from the following sources, in priority order:

1. OpenTelemetry gen_ai.* direct attributes and span events
2. OpenLLMetry indexed attributes
3. OpenInference indexed attributes
4. OpenInference input.value and output.value

The following indexed message attributes are supported:

Image content maps to an image URI while preserving its position among other message content.

Embedding spans

Retrieval spans

Tags filtering

OpenInference attributes with llm.*, retrieval.*, embedding.*, and reranker.* prefixes are excluded from tags. Specifically mapped values such as input.value, output.value, metadata, tag.tags, and tool.parameters are also excluded from duplicate tags.

Other non-empty OpenInference attributes with values of 256 characters or fewer are added as key:value tags. The tag.tags list is promoted directly to span tags.

Langfuse attribute mappings

This section documents Langfuse-specific attribute mappings for applications using Langfuse’s native OpenTelemetry instrumentation.

Detection

A span is treated as a Langfuse span when it carries a non-empty langfuse.observation.type attribute. This attribute is also used as a fallback for span kind resolution when gen_ai.operation.name is absent.

Span kind resolution

Model information

Token usage metrics

langfuse.observation.usage_details is a JSON object. Each key maps to an Agent Observability metric, used as a fallback for any metric not already set from gen_ai.usage.* attributes:

Input and output messages

langfuse.observation.input and langfuse.observation.output carry a JSON-encoded value that can be a chat-message array, a single message object, or arbitrary JSON/string content. These are the lowest-priority sources and are only used when no gen_ai.* message attributes exist.

Each message is converted to the parts-based message shape:

• A content string becomes a text part.
• A content array of blocks converts image_url blocks to uri parts and text blocks to text parts; any other block is kept as serialized text.
• A tool_calls array on a message becomes tool_call parts.
• A message with role: tool and a tool_call_id becomes a tool_result part.

Tool spans

For tool, agent, and workflow spans, langfuse.observation.input/langfuse.observation.output are used directly as input.value/output.value, after the standard gen_ai.tool.call.* fallback.

Retrieval spans

For retrieval spans, langfuse.observation.input is used as the query value in meta.input.value. langfuse.observation.output is parsed as a document collection (an array of document objects, an array of strings, or a single document object) into meta.output.documents. Each document object’s text, content, or page_content key (checked in that order) maps to text, along with any id, score, and metadata keys.

Embedding spans

For embedding spans, langfuse.observation.input is parsed the same way as retrieval documents into meta.input.documents[].text, keeping only documents that carry non-empty text.

Session, user, metadata, and tags

Tags filtering

The following Langfuse-specific attributes are filtered from tags because they’re consumed elsewhere:

• langfuse.internal.*, langfuse.observation.metadata.*, langfuse.trace.metadata.* (prefixes)
• langfuse.observation.input, langfuse.observation.output, langfuse.observation.model.name, langfuse.observation.model.parameters, langfuse.observation.usage_details, langfuse.observation.cost_details, langfuse.observation.completion_start_time
• langfuse.trace.input, langfuse.trace.output, langfuse.trace.metadata, langfuse.trace.tags
• langfuse.experiment.item.expected_output, langfuse.experiment.item.metadata, langfuse.experiment.metadata

Troubleshooting mapping warnings

At ingestion, Agent Observability detects span mapping problems such as missing input messages or an unparsable token count. It flags affected spans instead of failing silently.

Flagged spans display a Mapping warnings indicator in the span detail panel. Select the indicator to open the span’s mapping warnings. Each entry lists the affected attribute, a suggested fix, and a link to the attribute mapping reference.

Each warning also includes a short description of its downstream effect, such as skipped evaluations or unavailable cost estimation.

Mapping warnings are detected at ingestion. They do not affect billing or span retention.

Mapping warning reference

A span can display warnings not listed in the following table if Datadog adds checks. These render with a generated title based on the check that triggered the warning.

Search value shows the value stored on the span. Use it with the @collection_errors attribute to find flagged spans, for example @collection_errors:otel_warning_missing_model_name.

Find raw span attributes for a flagged span

To find the raw attributes Datadog received for a flagged span:

• If APM is enabled, the same data is also written to the corresponding APM trace. Open the linked APM trace to inspect the raw span attributes.
• Compare the raw attributes against the expected attribute from the mapping warning reference or the full attribute mapping reference. See Correlating Agent Observability and APM for how the two views relate.
• For a malformed warning, check whether the attribute value is valid JSON. Common causes include double-encoding, truncation, and emitting a string or other non-object value where an object is expected.
• For a missing attribute warning, check your instrumentation library version against the Tested frameworks and libraries table, and confirm OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental is set if your library requires it.

Supported semantic conventions

Agent Observability supports spans that follow the OpenTelemetry 1.37+ semantic conventions for generative AI, including:

• LLM operations with gen_ai.provider.name, "gen_ai.operation.name", gen_ai.request.model, and other gen_ai attributes
• Operation inputs/outputs on direct span attributes or via span events
• Token usage metrics (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens)
• Model parameters and metadata

For the complete list of supported attributes and their specifications, see the OpenTelemetry semantic conventions for generative AI documentation.

Disabling Agent Observability conversion

If you’d only like your generative AI spans to remain in APM and not appear in Agent Observability, you can disable the automatic conversion by setting the dd_llmobs_enabled attribute to false. Setting this attribute on any span in a trace prevents the entire trace from being converted to Agent Observability.

Using environment variables

Add the dd_llmobs_enabled=false attribute to your OTEL_RESOURCE_ATTRIBUTES environment variable:

OTEL_RESOURCE_ATTRIBUTES=dd_llmobs_enabled=false

Using code

You can also set the attribute programmatically on any span in your trace:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("my-span") as span:
    # Disable Agent Observability conversion for this entire trace
    span.set_attribute("dd_llmobs_enabled", False)