Agent Observability SDK Reference

Docs > Agent Observability > Agent Observability Instrumentation > Agent Observability SDK Reference

Overview

Agent Observability SDKs provide automatic instrumentation as well as manual instrumentation APIs to provide observability and insights into your LLM applications.

SDKs are available for Python, Node.js, Java, and Go (experimental). Feature coverage varies by language: sections that do not show a tab for your language are not supported by that SDK.

The Go SDK is experimental. It provides manual instrumentation only, because dd-trace-go has no LLM provider integrations.

Setup

Requirements

A Datadog API key.

The latest ddtrace package is installed (Python 3.7+ required):
```
pip install ddtrace
```

The latest dd-trace package is installed (Node.js 16+ required):
```
npm install dd-trace
```

You have downloaded the latest dd-trace-java JAR. The Agent Observability SDK is supported in dd-trace-java v1.51.0+ (Java 8+ required).

The Go SDK is experimental. Its API may change in a future release.

The latest dd-trace-go/v2 module is installed. The Agent Observability SDK is supported in dd-trace-go v2.3.0+ (v2.10.0+ recommended for the full feature set):
```
go get github.com/DataDog/dd-trace-go/v2
```
Note: dd-trace-go v2.8.0 and later require Go 1.25+. Versions v2.3.0 through v2.7.0 require Go 1.24+.

Command-line setup

Enable Agent Observability by running your application using the ddtrace-run command and specifying the required environment variables.

Note: ddtrace-run automatically turns on all Agent Observability integrations.

DD_SITE=<YOUR_DATADOG_SITE> DD_API_KEY=<YOUR_API_KEY> DD_LLMOBS_ENABLED=1 \
DD_LLMOBS_ML_APP=<YOUR_ML_APP_NAME> ddtrace-run <YOUR_APP_STARTUP_COMMAND>

Environment variables for command-line setup

DD_SITE: required - string
Destination Datadog site for LLM data submission. Your site is .
DD_LLMOBS_ENABLED: required - integer or string
Toggle to enable submitting data to Agent Observability. Should be set to 1 or true.
DD_LLMOBS_ML_APP: optional - string
The name of your LLM application, service, or project, under which all traces and spans are grouped. This helps distinguish between different applications or experiments. See Application naming guidelines for allowed characters and other constraints. To override this value for a given root span, see Tracing multiple applications. If not provided, this defaults to the value of DD_SERVICE, or the value of a propagated DD_LLMOBS_ML_APP from an upstream service.
Note: Before version ddtrace==3.14.0, this is a required field.
DD_LLMOBS_AGENTLESS_ENABLED: optional - integer or string - default: false
Only required if you are not using the Datadog Agent, in which case this should be set to 1 or true.
DD_LLMOBS_SAMPLE_RATE: optional - float - default: 1.0
The fraction of traces retained by Agent Observability. See Trace sampling.
DD_API_KEY: optional - string
Your Datadog API key. Only required if you are not using the Datadog Agent.
DD_MCP_CAPTURE_INTENT: optional - integer or string - default: false
When set to 1 or true, adds an argument to every MCP server tool requesting that the calling model describe why it chose to call the tool. The intent is recorded on the tool span.

Enable Agent Observability by running your application with NODE_OPTIONS="--import dd-trace/initialize.mjs" and specifying the required environment variables.

Note: dd-trace/initialize.mjs automatically turns on all APM integrations.

DD_SITE=<YOUR_DATADOG_SITE> DD_API_KEY=<YOUR_API_KEY> DD_LLMOBS_ENABLED=1 \
DD_LLMOBS_ML_APP=<YOUR_ML_APP_NAME> NODE_OPTIONS="--import dd-trace/initialize.mjs" node <YOUR_APP_ENTRYPOINT>

Environment variables for command-line setup

DD_SITE: required - string
The Datadog site to submit your LLM data. Your site is .
DD_LLMOBS_ENABLED: required - integer or string
Toggle to enable submitting data to Agent Observability. Should be set to 1 or true.
DD_LLMOBS_ML_APP: optional - string
The name of your LLM application, service, or project, under which all traces and spans are grouped. This helps distinguish between different applications or experiments. See Application naming guidelines for allowed characters and other constraints. To override this value for a given root span, see Tracing multiple applications. If not provided, this defaults to the value of DD_SERVICE, or the value of a propagated DD_LLMOBS_ML_APP from an upstream service.
Note: Before version dd-trace@5.66.0, this is a required field.
DD_LLMOBS_AGENTLESS_ENABLED: optional - integer or string - default: false
Only required if you are not using the Datadog Agent, in which case this should be set to 1 or true.
DD_LLMOBS_SAMPLE_RATE: optional - float - default: 1.0
The fraction of traces retained by Agent Observability. See Trace sampling.
DD_API_KEY: optional - string
Your Datadog API key. Only required if you are not using the Datadog Agent.

Enable Agent Observability by running your application with dd-trace-java and specifying the required parameters as environment variables or system properties.

DD_SITE=<YOUR_DATADOG_SITE> DD_API_KEY=<YOUR_API_KEY> \
java -javaagent:path/to/your/dd-trace-java-jar/dd-java-agent-SNAPSHOT.jar \
-Ddd.service=my-app -Ddd.llmobs.enabled=true -Ddd.llmobs.ml.app=my-ml-app -jar path/to/your/app.jar

Environment variables and system properties

You can supply the following parameters as environment variables (for example, DD_LLMOBS_ENABLED) or as Java system properties (for example, dd.llmobs_enabled).

DD_SITE or dd.site: required - string
Destination Datadog site for LLM data submission. Your site is .
DD_LLMOBS_ENABLED or dd.llmobs.enabled: required - integer or string
Toggle to enable submitting data to Agent Observability. Should be set to 1 or true.
DD_LLMOBS_ML_APP or dd.llmobs.ml.app: optional - string
The name of your LLM application, service, or project, under which all traces and spans are grouped. This helps distinguish between different applications or experiments. See Application naming guidelines for allowed characters and other constraints. To override this value for a given root span, see Tracing multiple applications. If not provided, this defaults to the value of DD_SERVICE, or the value of a propagated DD_LLMOBS_ML_APP from an upstream service.
Note: Before version 1.54.0 of dd-trace-java, this is a required field.
DD_LLMOBS_AGENTLESS_ENABLED or dd.llmobs.agentless.enabled: optional - integer or string - default: false
Only required if you are not using the Datadog Agent, in which case this should be set to 1 or true.
DD_LLMOBS_SAMPLE_RATE or dd.llmobs.sample.rate: optional - float - default: 1.0
The fraction of traces retained by Agent Observability. Requires dd-trace-java 1.66.0 or later. See Trace sampling.
DD_API_KEY or dd.api.key: optional - string
Your Datadog API key. Only required if you are not using the Datadog Agent.

The Go SDK has no command-line wrapper equivalent to ddtrace-run, because the tracer is started from your application code. Supply the following parameters as environment variables, then call tracer.Start(). See In-code setup.

DD_SITE=<YOUR_DATADOG_SITE> DD_API_KEY=<YOUR_API_KEY> DD_LLMOBS_ENABLED=1 \
DD_LLMOBS_ML_APP=<YOUR_ML_APP_NAME> <YOUR_APP_STARTUP_COMMAND>

Environment variables for command-line setup

DD_SITE: required - string
Destination Datadog site for LLM data submission. Your site is .
DD_LLMOBS_ENABLED: required - integer or string
Toggle to enable submitting data to Agent Observability. Should be set to 1 or true. Equivalent to tracer.WithLLMObsEnabled(true).
DD_LLMOBS_ML_APP: optional - string
The name of your LLM application, service, or project, under which all traces and spans are grouped. This helps distinguish between different applications or experiments. See Application naming guidelines for allowed characters and other constraints. To override this value for a given root span, see Tracing multiple applications. If not provided, this defaults to the value of DD_SERVICE. Equivalent to tracer.WithLLMObsMLApp().
DD_LLMOBS_AGENTLESS_ENABLED: optional - integer or string - default: false
Only required if you are not using the Datadog Agent, in which case this should be set to 1 or true. Equivalent to tracer.WithLLMObsAgentlessEnabled().
DD_LLMOBS_PROJECT_NAME: optional - string
The name of the project that datasets and experiments are grouped under. Equivalent to tracer.WithLLMObsProjectName().
DD_API_KEY: optional - string
Your Datadog API key. Only required if you are not using the Datadog Agent.

Note: The Go SDK does not support DD_LLMOBS_SAMPLE_RATE. See Trace sampling.

In-code setup

Instead of using command-line setup, you can also enable Agent Observability programmatically.

Use the LLMObs.enable() function to enable Agent Observability.

Do not use this setup method with the ddtrace-run command.

from ddtrace.llmobs import LLMObs
LLMObs.enable(
  ml_app="<YOUR_ML_APP_NAME>",
  api_key="<YOUR_DATADOG_API_KEY>",
  site="<YOUR_DATADOG_SITE>",
  agentless_enabled=True,
)

Parameters

ml_app: optional - string
The name of your LLM application, service, or project, under which all traces and spans are grouped. This helps distinguish between different applications or experiments. See Application naming guidelines for allowed characters and other constraints. To override this value for a given trace, see Tracing multiple applications. If not provided, this defaults to the value of DD_LLMOBS_ML_APP.
integrations_enabled - default: true: optional - boolean
A flag to enable automatically tracing LLM calls for Datadog’s supported LLM integrations. If not provided, all supported LLM integrations are enabled by default. To avoid using the LLM integrations, set this value to false.
agentless_enabled: optional - boolean - default: false
Only required if you are not using the Datadog Agent, in which case this should be set to True. This configures the ddtrace library to not send any data that requires the Datadog Agent. If not provided, this defaults to the value of DD_LLMOBS_AGENTLESS_ENABLED.
site: optional - string
The Datadog site to submit your LLM data. Your site is . If not provided, this defaults to the value of DD_SITE.
api_key: optional - string
Your Datadog API key. Only required if you are not using the Datadog Agent. If not provided, this defaults to the value of DD_API_KEY.
env: optional - string
The name of your application’s environment (examples: prod, pre-prod, staging). If not provided, this defaults to the value of DD_ENV.
service: optional - string
The name of the service used for your application. If not provided, this defaults to the value of DD_SERVICE.
sample_rate: optional - float
The fraction of traces retained by Agent Observability. Requires ddtrace 4.12.0 or later. When set, this takes precedence over DD_LLMOBS_SAMPLE_RATE. See Trace sampling.
capture_intent: optional - boolean - default: false
When set to True, adds an argument to every MCP server tool requesting that the calling model describe why it chose to call the tool. The intent is recorded on the tool span. If not provided, this defaults to the value of DD_MCP_CAPTURE_INTENT.

Do not use this setup method with the dd-trace/initialize.mjs command.

Use the init() function to enable Agent Observability.

const tracer = require('dd-trace').init({
  llmobs: {
    mlApp: "<YOUR_ML_APP_NAME>",
    agentlessEnabled: true,
  },
  site: "<YOUR_DATADOG_SITE>",
  env: "<YOUR_ENV>",
});

const llmobs = tracer.llmobs;

Options for llmobs configuration

mlApp: optional - string
The name of your LLM application, service, or project, under which all traces and spans are grouped. This helps distinguish between different applications or experiments. See Application naming guidelines for allowed characters and other constraints. To override this value for a given trace, see Tracing multiple applications. If not provided, this defaults to the value of DD_LLMOBS_ML_APP.
agentlessEnabled: optional - boolean - default: false
Only required if you are not using the Datadog Agent, in which case this should be set to true. This configures the dd-trace library to not send any data that requires the Datadog Agent. If not provided, this defaults to the value of DD_LLMOBS_AGENTLESS_ENABLED.
sampleRate: optional - number
The fraction of traces retained by Agent Observability. Requires dd-trace 5.110.0 or later. When set, this takes precedence over DD_LLMOBS_SAMPLE_RATE. See Trace sampling.

Options for general tracer configuration:

site: optional - string
The Datadog site to submit your LLM data. Your site is . If not provided, this defaults to the value of DD_SITE.
env: optional - string
The name of your application’s environment (examples: prod, pre-prod, staging). If not provided, this defaults to the value of DD_ENV.
service: optional - string
The name of the service used for your application. If not provided, this defaults to the value of DD_SERVICE.

Environment variables

Set the following values as environment variables. They cannot be configured programmatically.

DD_API_KEY: optional - string
Your Datadog API key. Only required if you are not using the Datadog Agent.

Enable Agent Observability by passing the Agent Observability options to tracer.Start().

package main

import (
	"log"

	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func main() {
	if err := tracer.Start(
		tracer.WithLLMObsEnabled(true),
		tracer.WithLLMObsMLApp("<YOUR_ML_APP_NAME>"),
		tracer.WithLLMObsAgentlessEnabled(true),
	); err != nil {
		log.Fatalf("failed to start tracer: %v", err)
	}
	defer tracer.Stop()

	// your application logic
}

Options

tracer.WithLLMObsEnabled: optional - bool
Toggle to enable submitting data to Agent Observability. If not provided, this defaults to the value of DD_LLMOBS_ENABLED.
tracer.WithLLMObsMLApp: optional - string
The name of your LLM application, service, or project, under which all traces and spans are grouped. This helps distinguish between different applications or experiments. See Application naming guidelines for allowed characters and other constraints. To override this value for a given trace, see Tracing multiple applications. If not provided, this defaults to the value of DD_LLMOBS_ML_APP.
tracer.WithLLMObsAgentlessEnabled: optional - bool - default: false
Only required if you are not using the Datadog Agent, in which case this should be set to true. If not provided, this defaults to the value of DD_LLMOBS_AGENTLESS_ENABLED.
tracer.WithLLMObsProjectName: optional - string
The name of the project that datasets and experiments are grouped under. If not provided, this defaults to the value of DD_LLMOBS_PROJECT_NAME.

Set DD_SITE, DD_API_KEY, DD_ENV, and DD_SERVICE as environment variables. DD_API_KEY is only required if you are not using the Datadog Agent.

Note: Agent mode requires Datadog Agent v7.72.0 or later when using datasets and experiments. Earlier Agent versions do not proxy the datasets and experiments endpoints correctly.

For AWS Lambda, see Trace an LLM Application in AWS Lambda.

After installing the SDK and running your application you should expect to see some data in Agent Observability from auto-instrumentation. Manual instrumentation can be used to capture custom built frameworks or operations from libraries that are not yet supported.

Trace sampling

Trace sampling is available in the Python SDK (ddtrace 4.12.0 or later), the Node.js SDK (dd-trace 5.110.0 or later), and the Java SDK (dd-trace-java 1.66.0 or later). It is not supported in the Go SDK.

Trace sampling sets the fraction of traces that Agent Observability retains. Because Agent Observability billing is based on the volume of spans you send, setting a sample rate is one way to control your Agent Observability cost. The SDK makes the sampling decision on the root span and applies it to all of that root span’s child spans, including spans created in downstream services through distributed tracing.

Sampling does not affect your Agent Observability metrics, including token and cost metrics and other operational metrics. Because unsampled spans are dropped after Datadog ingests your traces, these metrics remain based on 100% of your application’s instrumented traffic, regardless of the specified sample rate. Trace sampling is also independent of in-app controls such as automation rules and APM trace sampling, which apply after ingestion.

Configure the sample rate through either of two mechanisms:

Environment variable (DD_LLMOBS_SAMPLE_RATE): applies to both command-line setup and in-code setup. In Java, the dd.llmobs.sample.rate system property sets the same value.
In-code parameter (sample_rate in Python, sampleRate in Node.js): passed to LLMObs.enable() in Python, or under llmobs in Node.js, when you enable the SDK with in-code setup. When set, it takes precedence over DD_LLMOBS_SAMPLE_RATE. The Java and Go SDKs have no in-code equivalent.

The sample rate is a float between 0.0 (retain no traces) and 1.0 (retain all traces). The default is 1.0. Out-of-range values are ignored.

Set the sample rate with the environment variable:

DD_LLMOBS_SAMPLE_RATE=0.5 ddtrace-run <YOUR_APP_STARTUP_COMMAND>

Or pass sample_rate to LLMObs.enable(), which takes precedence over the environment variable:

from ddtrace.llmobs import LLMObs

LLMObs.enable(
  ml_app="<YOUR_ML_APP_NAME>",
  sample_rate=0.5,
)

Set the sample rate with the environment variable:

DD_LLMOBS_SAMPLE_RATE=0.5 NODE_OPTIONS="--import dd-trace/initialize.mjs" <YOUR_APP_STARTUP_COMMAND>

Or pass sampleRate under llmobs to init(), which takes precedence over the environment variable:

const tracer = require('dd-trace').init({
  llmobs: {
    mlApp: "<YOUR_ML_APP_NAME>",
    sampleRate: 0.5,
  },
});

const llmobs = tracer.llmobs;

Set the sample rate with the environment variable:

DD_LLMOBS_SAMPLE_RATE=0.5 \
java -javaagent:path/to/your/dd-trace-java-jar/dd-java-agent-SNAPSHOT.jar \
-Ddd.service=my-app -Ddd.llmobs.enabled=true -Ddd.llmobs.ml.app=<YOUR_ML_APP_NAME> \
-jar path/to/your/app.jar

Or set the equivalent dd.llmobs.sample.rate system property:

java -javaagent:path/to/your/dd-trace-java-jar/dd-java-agent-SNAPSHOT.jar \
-Ddd.service=my-app -Ddd.llmobs.enabled=true -Ddd.llmobs.ml.app=<YOUR_ML_APP_NAME> \
-Ddd.llmobs.sample.rate=0.5 -jar path/to/your/app.jar

Manual instrumentation

To capture an LLM operation a function decorator can be used to easily instrument workflows:

from ddtrace.llmobs.decorators import workflow

@workflow
def handle_user_request():
    ...

or a context-manager based approach to capture fine-grained operations:

from ddtrace.llmobs import LLMObs

with LLMObs.llm(model="gpt-4o"):
    call_llm()
    LLMObs.annotate(
        metrics={
            "input_tokens": ...,
            "output_tokens": ...,
        },
    )

For a list of available span kinds, see the Span Kinds documentation. For more granular tracing of operations within functions, see Tracing spans using inline methods.

To trace a span, use llmobs.wrap(options, function) as a function wrapper for the function you’d like to trace. For a list of available span kinds, see the Span Kinds documentation. For more granular tracing of operations within functions, see Tracing spans using inline methods.

Span Kinds

Span kinds are required, and are specified on the options object passed to the llmobs tracing functions (trace, wrap, and decorate). See the Span Kinds documentation for a list of supported span kinds.

Note: Spans with an invalid span kind are not submitted to Agent Observability.

Automatic function argument/output/name capturing

llmobs.wrap (along with llmobs.decorate for TypeScript) tries to automatically capture inputs, outputs, and the name of the function being traced. If you need to manually annotate a span, see Enriching spans. Inputs and outputs you annotate will override the automatic capturing. Additionally, to override the function name, pass the name property on the options object to the llmobs.wrap function:

function processMessage () {
  ... // user application logic
  return
}
processMessage = llmobs.wrap({ kind: 'workflow', name: 'differentFunctionName' }, processMessage)

Conditions for finishing a span for a wrapped function

llmobs.wrap extends the underlying behavior of tracer.wrap. The underlying span created when the function is called is finished under the following conditions:

If the function returns a Promise, then the span finishes when the promise is resolved or rejected.
If the function takes a callback as its last parameter, then the span finishes when that callback is called.
If t function doesn’t accept a callback and doesn’t return a Promise, then the span finishes at the end of the function execution.

The following example demonstrates the second condition, where the last argument is a callback:

Example

const express = require('express')
const app = express()

function myAgentMiddleware (req, res, next) {
  const err = ... // user application logic
  // the span for this function is finished when `next` is called
  next(err)
}
myAgentMiddleware = llmobs.wrap({ kind: 'agent' }, myAgentMiddleware)

app.use(myAgentMiddleware)

If the application does not use the callback function, it is recommended to use an inline traced block instead. See Tracing spans using inline methods for more information.

const express = require('express')
const app = express()

function myAgentMiddleware (req, res) {
  // the `next` callback is not being used here
  return llmobs.trace({ kind: 'agent', name: 'myAgentMiddleware' }, () => {
    return res.status(200).send('Hello World!')
  })
}

app.use(myAgentMiddleware)

Starting a span

There are multiple methods to start a span, based on the kind of span that you are starting. See the Span Kinds documentation for a list of supported span kinds.

All spans are started as an object instance of LLMObsSpan. Each span has methods that you can use to interact with the span and record data.

Finishing a span

Spans must be finished for the trace to be submitted and visible in the Datadog app.

To finish a span, call finish() on a span object instance. If possible, wrap the span in a try/finally block to ensure the span is submitted even if an exception occurs.

Example

    try {
        LLMObsSpan workflowSpan = LLMObs.startWorkflowSpan("my-workflow-span-name", "ml-app-override", "session-141");
        // user logic
        // interact with started span
    } finally {
      workflowSpan.finish();
    }

Starting a span

Each span kind has its own constructor in the llmobs package. See the Span Kinds documentation for a list of supported span kinds.

Every constructor takes a context.Context and returns the started span along with a new context.Context that carries the span. Pass that returned context into the next call to nest spans automatically. See Nesting spans.

span, ctx := llmobs.StartWorkflowSpan(ctx, "handle-user-request")
defer span.Finish()

Span kind	Constructor	Annotation method
`llm`	`llmobs.StartLLMSpan`	`AnnotateLLMIO`
`workflow`	`llmobs.StartWorkflowSpan`	`AnnotateTextIO`
`agent`	`llmobs.StartAgentSpan`	`AnnotateTextIO`
`tool`	`llmobs.StartToolSpan`	`AnnotateTextIO`
`task`	`llmobs.StartTaskSpan`	`AnnotateTextIO`
`embedding`	`llmobs.StartEmbeddingSpan`	`AnnotateEmbeddingIO`
`retrieval`	`llmobs.StartRetrievalSpan`	`AnnotateRetrievalIO`

Every constructor accepts the following llmobs.StartSpanOption values:

Start span options

llmobs.WithMLApp: optional - string
The name of the ML application that the operation belongs to. Overrides the ML app supplied at tracer start. See Tracing multiple applications.
llmobs.WithSessionID: optional - string
The ID of the underlying user session. See Tracking user sessions.
llmobs.WithModelName: optional - string - default: "custom"
The name of the invoked model. Ignored for span kinds other than llm and embedding.
llmobs.WithModelProvider: optional - string - default: "custom"
The name of the model provider. Ignored for span kinds other than llm and embedding.
Note: To display the estimated cost in US dollars, set the model provider to a supported provider.
llmobs.WithStartTime: optional - time.Time
An explicit start time for the span. Defaults to the current time.
llmobs.WithIntegration: optional - string
The name of the integration that produced the span.

Finishing a span

Spans must be finished for the trace to be submitted and visible in Datadog. Use defer to guarantee the span is finished even if the function returns early or panics.

span, ctx := llmobs.StartWorkflowSpan(ctx, "handle-user-request")
defer span.Finish()

Finish accepts the following llmobs.FinishSpanOption values:

llmobs.WithError: optional - error
Marks the span as errored and records the error message, type, and stack trace. See Annotating errors.
llmobs.WithFinishTime: optional - time.Time
An explicit finish time for the span. Defaults to the current time.

Retrieving the active span

llmobs.SpanFromContext returns the active Agent Observability span as an *llmobs.AnySpan. Convert it to a concrete span kind with the As* methods to reach that kind’s annotation method.

if span, ok := llmobs.SpanFromContext(ctx); ok {
	if llmSpan, ok := span.AsLLM(); ok {
		llmSpan.AnnotateLLMIO(nil, []llmobs.LLMMessage{{Role: "assistant", Content: "..."}})
	}
}

Note: The Go SDK has no decorator or function-wrapper API, so it does not automatically capture function arguments, return values, or names. Annotate inputs and outputs explicitly. See Enriching spans.

LLM calls

If you are using any LLM providers or frameworks that are supported by Datadog's LLM integrations, you do not need to manually start an LLM span to trace these operations.

If you are manually instrumenting an LLM span, you must record token counts (such as input_tokens, output_tokens, and total_tokens) yourself by annotating the span. See Enriching spans for more information.

To trace an LLM call, use the function decorator ddtrace.llmobs.decorators.llm().

Arguments

model_name: required - string
The name of the invoked LLM.
name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
model_provider: optional - string - default: "custom"
The name of the model provider.
Note: To display the estimated cost in US dollars, set model_provider to one of the following values: openai, azure_openai, or anthropic.
session_id: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
ml_app: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm

@llm(model_name="claude", name="invoke_llm", model_provider="anthropic")
def llm_call(prompt):
    completion = ... # user application logic to invoke LLM
    LLMObs.annotate(
        input_data=[{"role": "user", "content": prompt}],
        output_data=[{"role": "assistant", "content": completion}],
        metrics={"input_tokens": 4, "output_tokens": 6, "total_tokens": 10},
    )
    return completion

To trace an LLM call, specify the span kind as llm, and optionally specify the following arguments on the options object.

Arguments

modelName: optional - string - default: "custom"
The name of the invoked LLM.
name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
modelProvider: optional - string - default: "custom"
The name of the model provider.
Note: To display the estimated cost in US dollars, set modelProvider to one of the following values: openai, azure_openai, or anthropic.
sessionId: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
mlApp: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

function llmCall (prompt) {
  const completion = ... // user application logic to invoke LLM
  llmobs.annotate({
    inputData: [{ role: "user", content: prompt }],
    outputData: [{ role: "assistant", content: completion }],
    metrics: { input_tokens: 4, output_tokens: 6, total_tokens: 10 }
  })
  return completion
}
llmCall = llmobs.wrap({ kind: 'llm', name: 'invokeLLM', modelName: 'claude', modelProvider: 'anthropic' }, llmCall)

To trace an LLM call, import and call the following method with the arguments listed below:

import datadog.trace.api.llmobs.LLMObs;
LLMObs.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionID);

Arguments

spanName: optional - String
The name of the operation. If not provided, spanName defaults to the span kind.
modelName: optional - String - default: "custom"
The name of the invoked LLM.
modelProvider: optional - String - default: "custom"
The name of the model provider.
Note: To display the estimated cost in US dollars, set modelProvider to one of the following values: openai, azure_openai, or anthropic.
mlApp: optional - String
The name of the ML application that the operation belongs to. Supplying a non-null value overrides the ML app name supplied at the start of the application. See Tracing multiple applications for more information.
sessionId: optional - String
The ID of the underlying user session. See Tracking user sessions for more information.

Example

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeModel() {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    String inference = ... // user application logic to invoke LLM
    llmSpan.annotateIO(...); // record the input and output
    llmSpan.setMetrics(Map.of(
      "input_tokens", 617,
      "output_tokens", 338,
      "total_tokens", 955
    ));
    llmSpan.finish();
    return inference;
  }
}

Use llmobs.StartLLMSpan to trace an LLM call, then record the prompt and completion with AnnotateLLMIO.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func invokeModel(ctx context.Context, userInput string) string {
	span, _ := llmobs.StartLLMSpan(ctx, "invoke-llm",
		llmobs.WithModelName("gpt-5.1"),
		llmobs.WithModelProvider("openai"),
	)
	defer span.Finish()

	completion := callProvider(userInput) // user application logic to invoke LLM

	span.AnnotateLLMIO(
		[]llmobs.LLMMessage{
			{Role: "system", Content: "You are a helpful assistant"},
			{Role: "user", Content: userInput},
		},
		[]llmobs.LLMMessage{
			{Role: "assistant", Content: completion},
		},
		llmobs.WithAnnotatedMetrics(map[string]float64{
			llmobs.MetricKeyInputTokens:  50,
			llmobs.MetricKeyOutputTokens: 120,
			llmobs.MetricKeyTotalTokens:  170,
		}),
	)
	return completion
}

Tool calls and tool results

An llmobs.LLMMessage can also carry the tool calls a model requested and the results returned to it.

args, _ := json.Marshal(map[string]any{"city": "Paris"})

span.AnnotateLLMIO(
	[]llmobs.LLMMessage{{Role: "user", Content: "What is the weather in Paris?"}},
	[]llmobs.LLMMessage{{
		Role: "assistant",
		ToolCalls: []llmobs.ToolCall{
			{Name: "get_weather", Arguments: args, ToolID: "call_1", Type: "function"},
		},
	}},
)

Workflows

To trace a workflow span, use the function decorator ddtrace.llmobs.decorators.workflow().

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
session_id: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
ml_app: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

from ddtrace.llmobs.decorators import workflow

@workflow
def process_message():
    ... # user application logic
    return

To trace a workflow span, specify the span kind as workflow, and optionally specify arguments on the options object.

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
sessionId: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
mlApp: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

function processMessage () {
  ... // user application logic
  return
}
processMessage = llmobs.wrap({ kind: 'workflow' }, processMessage)

To trace a workflow span, import and call the following method with the arguments listed below:

import datadog.trace.api.llmobs.LLMObs;
LLMObs.startWorkflowSpan(spanName, mlApp, sessionID);

Arguments

spanName: optional - String
The name of the operation. If not provided, spanName defaults to the span kind.
mlApp: optional - String
The name of the ML application that the operation belongs to. Supplying a non-null value overrides the ML app name supplied at the start of the application. See Tracing multiple applications for more information.
sessionId: optional - String
The ID of the underlying user session. See Tracking user sessions for more information.

Example

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String executeWorkflow() {
    LLMObsSpan workflowSpan = LLMObs.startWorkflowSpan("my-workflow-span-name", null, "session-141");
    String workflowResult = workflowFn(); // user application logic
    workflowSpan.annotateIO(...); // record the input and output
    workflowSpan.finish();
    return workflowResult;
  }
}

Use llmobs.StartWorkflowSpan to trace a workflow span, then record its input and output with AnnotateTextIO.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func executeWorkflow(ctx context.Context, question string) string {
	span, ctx := llmobs.StartWorkflowSpan(ctx, "handle-user-request")
	defer span.Finish()

	answer := workflowFn(ctx, question) // user application logic

	span.AnnotateTextIO(question, answer)
	return answer
}

Agents

To trace an agent execution, use the function decorator ddtrace.llmobs.decorators.agent().

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
session_id: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
ml_app: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

from ddtrace.llmobs.decorators import agent

@agent
def react_agent():
    ... # user application logic
    return

To trace an agent execution, specify the span kind as agent, and optionally specify arguments on the options object.

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
sessionId: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
mlApp: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

function reactAgent () {
  ... // user application logic
  return
}
reactAgent = llmobs.wrap({ kind: 'agent' }, reactAgent)

To trace an agent execution, import and call the following method with the arguments listed below

import datadog.trace.api.llmobs.LLMObs;
LLMObs.startAgentSpan(spanName, mlApp, sessionID);

Arguments

spanName: optional - String
The name of the operation. If not provided, spanName defaults to the name of the traced function.
mlApp: optional - String
The name of the ML application that the operation belongs to. Supplying a non-null value overrides the ML app name supplied at the start of the application. See Tracing multiple applications for more information.
sessionId: optional - String
The ID of the underlying user session. See Tracking user sessions for more information.

Use llmobs.StartAgentSpan to trace an agent span, then record its input and output with AnnotateTextIO.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func runAgent(ctx context.Context, task string) string {
	span, ctx := llmobs.StartAgentSpan(ctx, "research-agent")
	defer span.Finish()

	result := agentLoop(ctx, task) // user application logic

	span.AnnotateTextIO(task, result)
	return result
}

Tool calls

To trace a tool call, use the function decorator ddtrace.llmobs.decorators.tool().

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
session_id: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
ml_app: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

from ddtrace.llmobs.decorators import tool

@tool
def call_weather_api():
    ... # user application logic
    return

To trace a tool call, specify the span kind as tool, and optionally specify arguments on the options object.

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
sessionId: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
mlApp: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

function callWeatherApi () {
  ... // user application logic
  return
}
callWeatherApi = llmobs.wrap({ kind: 'tool' }, callWeatherApi)

To trace a tool call, import and call the following method with the arguments listed below:

import datadog.trace.api.llmobs.LLMObs;
LLMObs.startToolSpan(spanName, mlApp, sessionID);

Arguments

spanName: optional - String
The name of the operation. If not provided, spanName defaults to the name of the traced function.
mlApp: optional - String
The name of the ML application that the operation belongs to. Supplying a non-null value overrides the ML app name supplied at the start of the application. See Tracing multiple applications for more information.
sessionId: optional - String
The ID of the underlying user session. See Tracking user sessions for more information.

Use llmobs.StartToolSpan to trace a tool span, then record its input and output with AnnotateTextIO.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func getWeather(ctx context.Context, city string) string {
	span, ctx := llmobs.StartToolSpan(ctx, "get_weather")
	defer span.Finish()

	result := weatherAPI(ctx, city) // user application logic

	span.AnnotateTextIO(city, result)
	return result
}

Tasks

To trace a task span, use the function decorator LLMObs.task().

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
session_id: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
ml_app: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

from ddtrace.llmobs.decorators import task

@task
def sanitize_input():
    ... # user application logic
    return

To trace a task span, specify the span kind as task, and optionally specify arguments on the options object.

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
sessionId: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
mlApp: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

function sanitizeInput () {
  ... // user application logic
  return
}
sanitizeInput = llmobs.wrap({ kind: 'task' }, sanitizeInput)

To trace a task span, import and call the following method with the arguments listed below:

import datadog.trace.api.llmobs.LLMObs;
LLMObs.startTaskSpan(spanName, mlApp, sessionID);

Arguments

spanName: optional - String
The name of the operation. If not provided, spanName defaults to the name of the traced function.
mlApp: optional - String
The name of the ML application that the operation belongs to. Supplying a non-null value overrides the ML app name supplied at the start of the application. See Tracing multiple applications for more information.
sessionId: optional - String
The ID of the underlying user session. See Tracking user sessions for more information.

Use llmobs.StartTaskSpan to trace a task span, then record its input and output with AnnotateTextIO.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func parseInput(ctx context.Context, raw string) string {
	span, ctx := llmobs.StartTaskSpan(ctx, "parse-input")
	defer span.Finish()

	parsed := parse(ctx, raw) // user application logic

	span.AnnotateTextIO(raw, parsed)
	return parsed
}

Embeddings

To trace an embedding operation, use the function decorator LLMObs.embedding().

Note: Annotating an embedding span’s input requires different formatting than other span types. See Enriching spans for more details on how to specify embedding inputs.

Arguments

model_name: required - string
The name of the invoked LLM.
name: optional - string
The name of the operation. If not provided, name is set to the name of the traced function.
model_provider: optional - string - default: "custom"
session_id: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
ml_app: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

from ddtrace.llmobs.decorators import embedding

@embedding(model_name="text-embedding-3", model_provider="openai")
def perform_embedding():
    ... # user application logic
    return

To trace an embedding operation, specify the span kind as embedding, and optionally specify arguments on the options object.

Note: Annotating an embedding span’s input requires different formatting than other span types. See Enriching spans for more details on how to specify embedding inputs.

Arguments

modelName: optional - string - default: "custom"
The name of the invoked LLM.
name: optional - string
The name of the operation. If not provided, name is set to the name of the traced function.
modelProvider: optional - string - default: "custom"
The name of the model provider.
sessionId: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
mlApp: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

function performEmbedding () {
  ... // user application logic
  return
}
performEmbedding = llmobs.wrap({ kind: 'embedding', modelName: 'text-embedding-3', modelProvider: 'openai' }, performEmbedding)

Use llmobs.StartEmbeddingSpan to trace an embedding span. AnnotateEmbeddingIO takes a slice of llmobs.EmbeddedDocument as the input and a string as the output.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func embedQuery(ctx context.Context, query string) []float64 {
	span, _ := llmobs.StartEmbeddingSpan(ctx, "embed-query",
		llmobs.WithModelName("text-embedding-3-small"),
		llmobs.WithModelProvider("openai"),
	)
	defer span.Finish()

	vector := embed(query) // user application logic

	span.AnnotateEmbeddingIO(
		[]llmobs.EmbeddedDocument{{Text: query}},
		"[0.023, -0.918, ...]",
		llmobs.WithAnnotatedMetrics(map[string]float64{
			llmobs.MetricKeyInputTokens: 8,
		}),
	)
	return vector
}

llmobs.EmbeddedDocument has the following fields:

Text: required - string
The text content of the document.
Name: optional - string
The name or title of the document.
ID: optional - string
The unique identifier of the document.
Score: optional - float64
The relevance score of the document, typically between 0.0 and 1.0.

Retrievals

To trace a retrieval span, use the function decorator ddtrace.llmobs.decorators.retrieval().

Note: Annotating a retrieval span’s output requires different formatting than other span types. See Enriching spans for more details on how to specify retrieval outputs.

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
session_id: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
ml_app: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

from ddtrace.llmobs.decorators import retrieval

@retrieval
def get_relevant_docs(question):
    context_documents = ... # user application logic
    LLMObs.annotate(
        input_data=question,
        output_data = [
            {"id": doc.id, "score": doc.score, "text": doc.text, "name": doc.name} for doc in context_documents
        ]
    )
    return

To trace a retrieval span, specify the span kind as retrieval, and optionally specify the following arguments on the options object.

Note: Annotating a retrieval span’s output requires different formatting than other span types. See Enriching spans for more details on how to specify retrieval outputs.

Arguments

name: optional - string
The name of the operation. If not provided, name defaults to the name of the traced function.
sessionId: optional - string
The ID of the underlying user session. See Tracking user sessions for more information.
mlApp: optional - string
The name of the ML application that the operation belongs to. See Tracing multiple applications for more information.

Example

The following also includes an example of annotating a span. See Enriching spans for more information.

function getRelevantDocs (question) {
  const contextDocuments = ... // user application logic
  llmobs.annotate({
    inputData: question,
    outputData: contextDocuments.map(doc => ({
      id: doc.id,
      score: doc.score,
      text: doc.text,
      name: doc.name
    }))
  })
  return
}
getRelevantDocs = llmobs.wrap({ kind: 'retrieval' }, getRelevantDocs)

Use llmobs.StartRetrievalSpan to trace a retrieval span. AnnotateRetrievalIO takes a string as the input and a slice of llmobs.RetrievedDocument as the output.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func searchDocs(ctx context.Context, query string) []Doc {
	span, _ := llmobs.StartRetrievalSpan(ctx, "search-docs")
	defer span.Finish()

	docs := vectorSearch(query) // user application logic

	span.AnnotateRetrievalIO(
		query,
		[]llmobs.RetrievedDocument{
			{Text: "Paris is the capital of France.", Name: "geography.md", ID: "doc-1", Score: 0.91},
		},
	)
	return docs
}

llmobs.RetrievedDocument has the following fields:

Text: required - string
The text content of the document.
Name: optional - string
The name or title of the document.
ID: optional - string
The unique identifier of the document.
Score: optional - float64
The relevance score of the document, typically between 0.0 and 1.0.

Nesting spans

Starting a new span before the current span is finished automatically traces a parent-child relationship between the two spans. The parent span represents the larger operation, while the child span represents a smaller nested sub-operation within it.

from ddtrace.llmobs.decorators import task, workflow

@workflow
def extract_data(document):
    preprocess_document(document)
    ... # performs data extraction on the document
    return

@task
def preprocess_document(document):
    ... # preprocesses a document for data extraction
    return

function preprocessDocument (document) {
  ... // preprocesses a document for data extraction
  return
}
preprocessDocument = llmobs.wrap({ kind: 'task' }, preprocessDocument)

function extractData (document) {
  preprocessDocument(document)
  ... // performs data extraction on the document
  return
}
extractData = llmobs.wrap({ kind: 'workflow' }, extractData)

import datadog.trace.api.llmobs.LLMObs;
import datadog.trace.api.llmobs.LLMObsSpan;

public class MyJavaClass {
  public void preprocessDocument(String document) {
  LLMObsSpan taskSpan = LLMObs.startTaskSpan("preprocessDocument", null, "session-141");
   ...   // preprocess document for data extraction
   taskSpan.annotateIO(...); // record the input and output
   taskSpan.finish();
  }

  public String extractData(String document) {
    LLMObsSpan workflowSpan = LLMObs.startWorkflowSpan("extractData", null, "session-141");
    preprocessDocument(document);
    ... // perform data extraction on the document
    workflowSpan.annotateIO(...); // record the input and output
    workflowSpan.finish();
  }
}

Span constructors return a new context.Context carrying the started span. Pass that context to the next constructor to make the new span a child of the current one.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func handleRequest(ctx context.Context) {
	workflowSpan, ctx := llmobs.StartWorkflowSpan(ctx, "workflow-1")
	defer workflowSpan.Finish()

	// child of workflow-1, because it receives the workflow's context
	llmSpan, ctx := llmobs.StartLLMSpan(ctx, "llm-1", llmobs.WithModelName("gpt-5.1"))
	defer llmSpan.Finish()

	// APM spans can be mixed into the same trace
	apmSpan, _ := tracer.StartSpanFromContext(ctx, "post-process")
	defer apmSpan.Finish()
}

Note: Parenting is carried on the context. A span started with a context that holds no parent span becomes the root of a new trace. When you launch a goroutine, pass it the context returned by the parent’s constructor, so its spans attach to the right parent.

Enriching spans

The metrics parameter here refers to numeric values attached as attributes on individual spans — not Datadog platform metrics. For certain recognized keys such as input_tokens, output_tokens, and total_tokens, Datadog uses these span attributes to generate corresponding platform metrics (such as ml_obs.span.llm.input.tokens) for use in dashboards and monitors.

The SDK provides the method LLMObs.annotate() to enrich spans with inputs, outputs, and metadata.

The LLMObs.annotate() method accepts the following arguments:

Arguments

span: optional - Span - default: the current active span
The span to annotate. If span is not provided (as when using function decorators), the SDK annotates the current active span.
input_data: optional - JSON serializable type or list of dictionaries
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: {"content": "...", "role": "...", "tool_calls": ..., "tool_results": ..., "audio_parts": ..., "image_parts": ...}, where "tool_calls" are an optional list of tool call dictionaries with required keys: "name", "arguments", and optional keys: "tool_id", "type", and "tool_results" are an optional list of tool result dictionaries with required key: "result", and optional keys: "name", "tool_id", "type" for function calling scenarios. "audio_parts" and "image_parts" are optional lists of media dictionaries for multimodal spans, each with a required "mime_type" and "content" (base64-encoded media, carried inline). Note: Embedding spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: {"text": "..."}.
output_data: optional - JSON serializable type or list of dictionaries
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: {"content": "...", "role": "...", "tool_calls": ..., "audio_parts": ..., "image_parts": ...}, where "tool_calls" are an optional list of tool call dictionaries with required keys: "name", "arguments", and optional keys: "tool_id", "type" for function calling scenarios. "audio_parts" and "image_parts" are optional lists of media dictionaries for multimodal spans, each with a required "mime_type" and "content" (base64-encoded media, carried inline). Note: Retrieval spans are a special case and require a string or a dictionary (or a list of dictionaries) with this format: {"text": "...", "name": "...", "score": float, "id": "..."}.
tool_definitions: optional - list of dictionaries
List of tool definition dictionaries for function calling scenarios. Each tool definition should have a required "name": "..." key and optional "description": "..." and "schema": {...} keys.
metadata: optional - dictionary
A dictionary of JSON serializable key-value pairs that users can add as metadata information relevant to the input or output operation described by the span (model_temperature, max_tokens, top_k, etc.).
metrics: optional - dictionary
A dictionary of JSON serializable keys and numeric values that users can add as metrics relevant to the operation described by the span (input_tokens, output_tokens, total_tokens, time_to_first_token, etc.). The unit for time_to_first_token is in seconds, similar to the duration metric which is emitted by default.
tags: optional - dictionary
A dictionary of JSON serializable key-value pairs that users can add as tags on the span. Example keys: session, env, system, and version. For more information about tags, see Getting Started with Tags.
cost_tags: optional - list of strings
A list of tag keys (already set with tags or annotated previously on the same span) to propagate as custom tags on the generated LLM cost and token metrics. Entries that don’t reference an existing tag key are skipped. See Cost monitoring for details.

Example

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import embedding, llm, retrieval, workflow

@llm(model_name="model_name", model_provider="model_provider")
def llm_call(prompt):
    resp = ... # llm call here
    LLMObs.annotate(
        span=None,
        input_data=[{"role": "user", "content": "Hello world!"}],
        output_data=[{"role": "assistant", "content": "How can I help?"}],
        metadata={"temperature": 0, "max_tokens": 200},
        metrics={"input_tokens": 4, "output_tokens": 6, "total_tokens": 10},
        tags={"host": "host_name"},
    )
    return resp

@workflow
def extract_data(document):
    resp = llm_call(document)
    LLMObs.annotate(
        input_data=document,
        output_data=resp,
        tags={"host": "host_name"},
    )
    return resp

@embedding(model_name="text-embedding-3", model_provider="openai")
def perform_embedding():
    ... # user application logic
    LLMObs.annotate(
        span=None,
        input_data={"text": "Hello world!"},
        output_data=[0.0023064255, -0.009327292, ...],
        metrics={"input_tokens": 4},
        tags={"host": "host_name"},
    )
    return

@retrieval(name="get_relevant_docs")
def similarity_search():
    ... # user application logic
    LLMObs.annotate(
        span=None,
        input_data="Hello world!",
        output_data=[{"text": "Hello world is ...", "name": "Hello, World! program", "id": "document_id", "score": 0.9893}],
        tags={"host": "host_name"},
    )
    return

@llm(model_name="gpt-realtime", model_provider="openai")
def voice_turn(user_audio_bytes):
    import base64
    resp = ... # multimodal (audio) llm call here
    LLMObs.annotate(
        span=None,
        input_data=[
            {
                "role": "user",
                "content": "Hey, how are you?",  # transcript of the input audio
                "audio_parts": [
                    {"mime_type": "audio/wav", "content": base64.b64encode(user_audio_bytes).decode("utf-8")}
                ],
            }
        ],
        output_data=[
            {
                "role": "assistant",
                "content": "Hey! I'm doing great, thanks for asking. How about you?",
                "audio_parts": [
                    {"mime_type": "audio/wav", "content": base64.b64encode(resp.audio_bytes).decode("utf-8")}
                ],
            }
        ],
    )
    return resp

@llm(model_name="gpt-4o", model_provider="openai")
def describe_image(image_bytes):
    import base64
    resp = ... # multimodal (vision) llm call here
    LLMObs.annotate(
        span=None,
        input_data=[
            {
                "role": "user",
                "content": "What is in this image?",
                "image_parts": [
                    {"mime_type": "image/png", "content": base64.b64encode(image_bytes).decode("utf-8")}
                ],
            }
        ],
        output_data=[{"role": "assistant", "content": "The image shows a golden retriever puppy."}],
    )
    return resp

Messages annotated with audio_parts or image_parts render as inline audio players and images in the trace view. For rendered examples, supported formats, size limits, and the integrations that populate these fields automatically, see Multimodal Support.

The SDK provides the method llmobs.annotate() to annotate spans with inputs, outputs, and metadata.

The LLMObs.annotate() method accepts the following arguments:

Arguments

span: optional - Span - default: the current active span
The span to annotate. If span is not provided (as when using function wrappers), the SDK annotates the current active span.
annotationOptions: required - object
An object of different types of data to annotate the span with.

The annotationOptions object can contain the following:

inputData: optional - JSON serializable type or list of objects
Either a JSON serializable type (for non-LLM spans) or a list of dictionaries with this format: {role: "...", content: "...", audioParts: [...], imageParts: [...]} (for LLM spans). audioParts and imageParts are optional lists of media objects for multimodal spans, each with a required mimeType and content (base64-encoded media, carried inline). Note: Embedding spans are a special case and require a string or an object (or a list of objects) with this format: {text: "..."}.
outputData: optional - JSON serializable type or list of objects
Either a JSON serializable type (for non-LLM spans) or a list of objects with this format: {role: "...", content: "...", audioParts: [...], imageParts: [...]} (for LLM spans). audioParts and imageParts are optional lists of media objects for multimodal spans, each with a required mimeType and content (base64-encoded media, carried inline). Note: Retrieval spans are a special case and require a string or an object (or a list of objects) with this format: {text: "...", name: "...", score: number, id: "..."}.
metadata: optional - object
An object of JSON serializable key-value pairs that users can add as metadata information relevant to the input or output operation described by the span (model_temperature, max_tokens, top_k, etc.).
metrics: optional - object
An object of JSON serializable keys and numeric values that users can add as metrics relevant to the operation described by the span (input_tokens, output_tokens, total_tokens, etc.).
tags: optional - object
An object of JSON serializable key-value pairs that users can add as tags regarding the span’s context (session, environment, system, versioning, etc.). For more information about tags, see Getting Started with Tags.
costTags: optional - array of strings
A list of tag keys (already set with tags or annotated previously on the same span) to propagate as custom tags on the generated LLM cost and token metrics. Entries that don’t reference an existing tag key are skipped. See Cost monitoring for details.

Example

function llmCall (prompt) {
  const completion = ... // user application logic to invoke LLM
  llmobs.annotate({
    inputData: [{ role: "user", content: "Hello world!" }],
    outputData: [{ role: "assistant", content: "How can I help?" }],
    metadata: { temperature: 0, max_tokens: 200 },
    metrics: { input_tokens: 4, output_tokens: 6, total_tokens: 10 },
    tags: { host: "host_name" }
  })
  return completion
}
llmCall = llmobs.wrap({ kind:'llm', modelName: 'modelName', modelProvider: 'modelProvider' }, llmCall)

function extractData (document) {
  const resp = llmCall(document)
  llmobs.annotate({
    inputData: document,
    outputData: resp,
    tags: { host: "host_name" }
  })
  return resp
}
extractData = llmobs.wrap({ kind: 'workflow' }, extractData)

function performEmbedding () {
  ... // user application logic
  llmobs.annotate(
    undefined, { // this can be set to undefined or left out entirely
      inputData: { text: "Hello world!" },
      outputData: [0.0023064255, -0.009327292, ...],
      metrics: { input_tokens: 4 },
      tags: { host: "host_name" }
    }
  )
}
performEmbedding = llmobs.wrap({ kind: 'embedding', modelName: 'text-embedding-3', modelProvider: 'openai' }, performEmbedding)

function similaritySearch () {
  ... // user application logic
  llmobs.annotate(undefined, {
    inputData: "Hello world!",
    outputData: [{ text: "Hello world is ...", name: "Hello, World! program", id: "document_id", score: 0.9893 }],
    tags: { host: "host_name" }
  })
  return
}
similaritySearch = llmobs.wrap({ kind: 'retrieval', name: 'getRelevantDocs' }, similaritySearch)

function voiceTurn (userAudioBytes) {
  const resp = ... // multimodal (audio) llm call here
  llmobs.annotate({
    inputData: [
      {
        role: "user",
        content: "Hey, how are you?", // transcript of the input audio
        audioParts: [{ mimeType: "audio/wav", content: userAudioBytes.toString("base64") }]
      }
    ],
    outputData: [
      {
        role: "assistant",
        content: "Hey! I'm doing great, thanks for asking. How about you?",
        audioParts: [{ mimeType: "audio/wav", content: resp.audioBuffer.toString("base64") }]
      }
    ]
  })
  return resp
}
voiceTurn = llmobs.wrap({ kind: 'llm', modelName: 'gpt-audio', modelProvider: 'openai' }, voiceTurn)

function describeImage (imageBytes) {
  const resp = ... // multimodal (vision) llm call here
  llmobs.annotate({
    inputData: [
      {
        role: "user",
        content: "What is in this image?",
        imageParts: [{ mimeType: "image/png", content: imageBytes.toString("base64") }]
      }
    ],
    outputData: [{ role: "assistant", content: "The image shows a golden retriever puppy." }]
  })
  return resp
}
describeImage = llmobs.wrap({ kind: 'llm', modelName: 'gpt-4o', modelProvider: 'openai' }, describeImage)

Messages annotated with audioParts or imageParts render as inline audio players and images in the trace view. For rendered examples, supported formats, size limits, and the integrations that populate these fields automatically, see Multimodal Support.

For OpenAI audio chat completions, audioParts are also captured automatically by Datadog’s LLM integrations, with no manual annotation required. The Node.js SDK does not capture imageParts automatically. Annotate them as shown above.

The SDK provides several methods to annotate spans with inputs, outputs, metrics, and metadata.

Annotating inputs and outputs

Use the annotateIO() member method of the LLMObsSpan interface to add structured input and output data to an LLMObsSpan. This includes optional arguments and LLM message objects.

Arguments

If an argument is null or empty, nothing happens. For example, if inputData is a non-empty string while outputData is null, then only inputData is recorded.

inputData: optional - String or List<LLMObs.LLMMessage>
Either a string (for non-LLM spans) or a list of LLMObs.LLMMessages for LLM spans.
outputData: optional - String or List<LLMObs.LLMMessage>
Either a string (for non-LLM spans) or a list of LLMObs.LLMMessages for LLM spans.

LLM Messages

LLM spans must be annotated with LLM Messages using the LLMObs.LLMMessage object.

The LLMObs.LLMMessage object can be instantiated by calling LLMObs.LLMMessage.from() with the following arguments:

role: required - String
A string describing the role of the author of the message.
content: required - String
A string containing the content of the message.

Example

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeChat(String userInput) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    String systemMessage = "You are a helpful assistant";
    Response chatResponse = ... // user application logic to invoke LLM
    llmSpan.annotateIO(
      Arrays.asList(
        LLMObs.LLMMessage.from("user", userInput),
        LLMObs.LLMMessage.from("system", systemMessage)
      ),
      Arrays.asList(
        LLMObs.LLMMessage.from(chatResponse.role, chatResponse.content)
      )
    );
    llmSpan.finish();
    return chatResponse;
  }
}

Adding metrics

Bulk add metrics

The setMetrics() member method of the LLMObsSpan interface accepts the following arguments to attach multiple metrics in bulk:

Arguments

metrics: required - Map<String, Number>
A map of JSON-serializable keys and numeric values that users can add to record metrics relevant to the operation described by the span (for example, input_tokens, output_tokens, or total_tokens).

Add a single metric

The setMetric() member method of the LLMObsSpan interface accepts the following arguments to attach a single metric:

Arguments

key: required - CharSequence
The name of the metric.
value: required - int, long, or double
The value of the metric.

Examples

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeChat(String userInput) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    String chatResponse = ... // user application logic to invoke LLM
    llmSpan.setMetrics(Map.of(
      "input_tokens", 617,
      "output_tokens", 338,
      "time_per_output_token", 0.1773
    ));
    llmSpan.setMetric("total_tokens", 955);
    llmSpan.setMetric("time_to_first_token", 0.23);
    llmSpan.finish();
    return chatResponse;
  }
}

Adding tags

For more information about tags, see Getting Started with Tags.

Bulk add tags

The setTags() member method of the LLMObsSpan interface accepts the following arguments to attach multiple tags in bulk:

Arguments

tags: required - Map<String, Object>
A map of JSON-serializable key-value pairs that users can add as tags to describe the span’s context (for example, session, environment, system, or version).

Add a single tag

The setTag() member method of the LLMObsSpan interface accepts the following arguments to attach a single tag:

Arguments

key: required - String
The key of the tag.
value: required - int, long, double, boolean, or String
The value of the tag.

Examples

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeChat(String userInput) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    String chatResponse = ... // user application logic to invoke LLM
    llmSpan.setTags(Map.of(
      "chat_source", "web",
      "users_in_chat", 3
    ));
    llmSpan.setTag("is_premium_user", true);
    llmSpan.finish();
    return chatResponse;
  }
}

Annotating errors

Adding a Throwable (recommended)

The addThrowable() member method of the LLMObsSpan interface accepts the following argument to attach a throwable with a stack trace:

Arguments

throwable: required - Throwable
The throwable/exception that occurred.

Adding an error message

The setErrorMessage() member method of the LLMObsSpan interface accepts the following argument to attach an error string:

Arguments

errorMessage: required - String
The message of the error.

Setting an error flag

The setError() member method of the LLMObsSpan interface accepts the following argument to indicate an error with the operation:

Arguments

error: required - boolean
true if the span errored.

Examples

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeChat(String userInput) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    String chatResponse = "N/A";
    try {
      chatResponse = ... // user application logic to invoke LLM
    } catch (Exception e) {
      llmSpan.addThrowable(e);
      throw new RuntimeException(e);
    } finally {
      llmSpan.finish();
    }
    return chatResponse;
  }
}

Annotating metadata

The setMetadata() member method of the LLMObsSpan interface accepts the following arguments:

metadata: required - Map<String, Object>
A map of JSON-serializable key-value pairs that contains metadata relevant to the input or output operation described by the span.

Example

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeChat(String userInput) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    llmSpan.setMetadata(
      Map.of(
        "temperature", 0.5,
        "is_premium_member", true,
        "class", "e1"
      )
    );
    String chatResponse = ... // user application logic to invoke LLM
    return chatResponse;
  }
}

Each span kind exposes a typed method for annotating its input and output, and every span exposes a generic Annotate method that accepts the following llmobs.AnnotateOption values.

Annotating inputs and outputs

Use the annotation method matching the span kind:

AnnotateLLMIO(input, output []llmobs.LLMMessage, opts ...) on llm spans
AnnotateEmbeddingIO(input []llmobs.EmbeddedDocument, output string, opts ...) on embedding spans
AnnotateRetrievalIO(input string, output []llmobs.RetrievedDocument, opts ...) on retrieval spans
AnnotateTextIO(input, output string, opts ...) on workflow, agent, tool, and task spans

llmobs.LLMMessage has the following fields:

Role: required - string
The role of the message author, for example system, user, or assistant.
Content: required - string
The text content of the message.
ToolCalls: optional - []llmobs.ToolCall
The tool calls made in this message.
ToolResults: optional - []llmobs.ToolResult
The results of tool calls in this message.

Adding metrics

Use llmobs.WithAnnotatedMetrics with a map[string]float64. The llmobs package exports constants for the recognized keys.

span.Annotate(llmobs.WithAnnotatedMetrics(map[string]float64{
	llmobs.MetricKeyInputTokens:            50,
	llmobs.MetricKeyOutputTokens:           120,
	llmobs.MetricKeyTotalTokens:            170,
	llmobs.MetricKeyCacheReadInputTokens:   22,
	llmobs.MetricKeyCacheWriteInputTokens:  15,
	llmobs.MetricKeyReasoningOutputTokens:  30,
	llmobs.MetricKeyTimeToFirstToken:       0.25,
	llmobs.MetricKeyBillableCharacterCount: 900,
}))

Constant	Key
`llmobs.MetricKeyInputTokens`	`input_tokens`
`llmobs.MetricKeyOutputTokens`	`output_tokens`
`llmobs.MetricKeyTotalTokens`	`total_tokens`
`llmobs.MetricKeyCacheReadInputTokens`	`cache_read_input_tokens`
`llmobs.MetricKeyCacheWriteInputTokens`	`cache_write_input_tokens`
`llmobs.MetricKeyReasoningOutputTokens`	`reasoning_output_tokens`
`llmobs.MetricKeyBillableCharacterCount`	`billable_character_count`
`llmobs.MetricKeyTimeToFirstToken`	`time_to_first_token`

Keys that are not in this list are accepted as custom metrics, including the input_cost and output_cost keys used for manual cost tracking.

Adding tags

Use llmobs.WithAnnotatedTags with a map[string]string.

span.Annotate(llmobs.WithAnnotatedTags(map[string]string{
	"team":        "nlp",
	"host":        "host_name",
	"feature":     "chatbot",
	"user_handle": "poodle@dog.com",
}))

Annotating errors

Pass llmobs.WithError to Finish to mark the span as errored. The SDK records the error message, type, and stack trace.

func invokeModel(ctx context.Context) (string, error) {
	span, _ := llmobs.StartLLMSpan(ctx, "invoke-llm")

	completion, err := invokeProvider(ctx)
	if err != nil {
		span.Finish(llmobs.WithError(err))
		return "", err
	}

	span.Finish()
	return completion, nil
}

To keep a single defer, capture the error in a named return value:

func invokeModel(ctx context.Context) (completion string, err error) {
	span, _ := llmobs.StartLLMSpan(ctx, "invoke-llm")
	defer func() { span.Finish(llmobs.WithError(err)) }()

	completion, err = invokeProvider(ctx)
	return completion, err
}

Note: llmobs.WithError(nil) is a no-op, so the deferred call above does not mark successful spans as errored.

Annotating metadata

Use llmobs.WithAnnotatedMetadata with a map[string]any to record parameters and other contextual data.

span.Annotate(llmobs.WithAnnotatedMetadata(map[string]any{
	"temperature": 0.7,
	"max_tokens":  200,
}))

Other annotation options

llmobs.WithAnnotatedSessionID: optional - string
The ID of the underlying user session. See Tracking user sessions.
llmobs.WithAnnotatedPrompt: optional - llmobs.Prompt
Structured prompt metadata. Requires dd-trace-go v2.8.0+. See Prompt tracking.
llmobs.WithAnnotatedToolDefinitions: optional - []llmobs.ToolDefinition
The tool definitions made available to the model for this call. Requires dd-trace-go v2.8.0+.
llmobs.WithAnnotatedCostTagKeys: optional - []string
Span tag keys to propagate onto the cost and token metrics. Requires dd-trace-go v2.10.0+. See Adding custom tags to cost and tokens metrics.
llmobs.WithAnnotatedIntent: optional - string
A description of why a tool was called, recorded on tool spans. Requires dd-trace-go v2.8.0+.

Annotating auto-instrumented spans

The SDK’s LLMObs.annotation_context() method returns a context manager that can be used to modify all auto-instrumented spans started while the annotation context is active.

The LLMObs.annotation_context() method accepts the following arguments:

Arguments

name: optional - str
Name that overrides the span name for any auto-instrumented spans that are started within the annotation context.
prompt: optional - dictionary
A dictionary that represents the prompt used for an LLM call. See the Prompt object documentation for the complete schema and supported keys. You can also import the Prompt object from ddtrace.llmobs.utils and pass it in as the prompt argument. Note: This argument only applies to LLM spans.
tags: optional - dictionary
A dictionary of JSON serializable key-value pairs that users can add as tags on the span. Example keys: session, env, system, and version. For more information about tags, see Getting Started with Tags.
cost_tags: optional - list of strings
A list of tag keys to propagate as custom tags on the generated LLM cost and token metrics. Each entry must reference a key present in tags at span start (supplied to the same context or a parent context); tag keys added later with LLMObs.annotate() are not retained. See Cost monitoring for details.

Example

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import workflow

@workflow
def rag_workflow(user_question):
    context_str = retrieve_documents(user_question).join(" ")

    with LLMObs.annotation_context(
        prompt = Prompt(
            id="chatbot_prompt",
            version="1.0.0",
            template="Please answer the question using the provided context: {{question}}\n\nContext:\n{{context}}",
            variables={
                "question": user_question,
                "context": context_str,
            }
        ),
        tags = {
            "retrieval_strategy": "semantic_similarity"
        },
        name = "augmented_generation"
    ):
        completion = openai_client.chat.completions.create(...)
    return completion.choices[0].message.content

The SDK’s llmobs.annotationContext() accepts a callback function that can be used to modify all auto-instrumented spans started while inside the scope of the callback function.

The llmobs.annotationContext() method accepts the following options on the first argument:

Options

name: optional - str
Name that overrides the span name for any auto-instrumented spans that are started within the annotation context.
tags: optional - object
An object of JSON serializable key-value pairs that users can add as tags on the span. Example keys: session, env, system, and version. For more information about tags, see Getting Started with Tags.
costTags: optional - array of strings
A list of tag keys to propagate as custom tags on the generated LLM cost and token metrics. Each entry must reference a key present in tags at span start (supplied to the same context or a parent context); tag keys added later with llmobs.annotate() are not retained. See Cost monitoring for details.

Example

const { llmobs } = require('dd-trace');

function ragWorkflow(userQuestion) {
    const contextStr = retrieveDocuments(userQuestion).join(" ");

    const completion = await llmobs.annotationContext({
      tags: {
        retrieval_strategy: "semantic_similarity"
      },
      name: "augmented_generation"
    }, async () => {
      const completion = await openai_client.chat.completions.create(...);
      return completion.choices[0].message.content;
    });
}

Prompt tracking

Attach structured prompt metadata to the LLM span so you can reproduce results, audit changes, and compare prompt performance across versions. When using templates, Agent Observability also provides version tracking based on template content changes.

Use LLMObs.annotation_context(prompt=...) to attach prompt metadata before the LLM call. For more details on span annotation, see Enriching spans.

Arguments

prompt: required - dictionary
A typed dictionary that follows the Prompt schema below.

Prompt structure

Supported keys:

id (str): Logical identifier for this prompt. Should be unique per ml_app. Defaults to {ml_app}-unnamed_prompt
version (str): Version tag for the prompt (for example, “1.0.0”). See version tracking for more details.
variables (Dict[str, str]): Variables used to populate the template placeholders.
template (str): Template string with placeholders (for example, "Translate {{text}} to {{lang}}").
chat_template (List[Message]): Multi-message template form. Provide a list of { "role": "<role>", "content": "<template string with placeholders>" } objects.
tags (Dict[str, str]): Tags to attach to the prompt run.
rag_context_variables (List[str]): Variable keys that contain ground-truth/context content. Used for hallucination detection.
rag_query_variables (List[str]): Variable keys that contain the user query. Used for hallucination detection.

Example: single-template prompt

from ddtrace.llmobs import LLMObs

def answer_question(text):
    # Attach prompt metadata to the upcoming LLM span using LLMObs.annotation_context()
    with LLMObs.annotation_context(prompt={
        "id": "translation-template",
        "version": "1.0.0",
        "chat_template": [{"role": "user", "content": "Translate to {{lang}}: {{text}}"}],
        "variables": {"lang": "fr", "text": text},
        "tags": {"team": "nlp"}
    }):
        # Example provider call (replace with your client)
        completion = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"Translate to fr: {text}"}]
        )
    return completion

Example: LangChain prompt templates

When you use LangChain’s prompt templating with auto-instrumentation, assign templates to variables with meaningful names. Auto-instrumentation uses these names to identify prompts.

# "translation_template" will be used to identify the template in Datadog
translation_template = PromptTemplate.from_template("Translate {text} to {language}")
chain = translation_template | llm

Use llmobs.annotationContext({ prompt: ... }, () => { ... }) to attach prompt metadata before the LLM call. For more details on span annotation, see Enriching spans.

Arguments

Options

prompt: required - object
An object that follows the Prompt schema below.

Prompt structure

Supported properties:

id (string): Logical identifier for this prompt. Should be unique per ml_app. Defaults to {ml_app}-unnamed_prompt
version (string): Version tag for the prompt (for example, “1.0.0”). See version tracking for more details.
variables (Record<string, string>): Variables used to populate the template placeholders.
template (string | List[Message]): Template string with placeholders (for example, "Translate {{text}} to {{lang}}"). Alternatively, a list of { "role": "<role>", "content": "<template string with placeholders>" } objects.
tags (Record<string, string>): Tags to attach to the prompt run.
contextVariables (string[]): Variable keys that contain ground-truth/context content. Used for hallucination detection.
queryVariables (string[]): Variable keys that contain the user query. Used for hallucination detection.

Example: single-template prompt

const { llmobs } = require('dd-trace');

function answerQuestion(text) {
    // Attach prompt metadata to the upcoming LLM span using LLMObs.annotation_context()
    return llmobs.annotationContext({
      prompt: {
        id: "translation-template",
        version: "1.0.0",
        chat_template: [{"role": "user", "content": "Translate to {{lang}}: {{text}}"}],
        variables: {"lang": "fr", "text": text},
        tags: {"team": "nlp"}
      }
    }, () => {
      // Example provider call (replace with your client)
      return openaiClient.chat.completions.create({
          model: "gpt-4o",
          messages: [{"role": "user", "content": f"Translate to fr: {text}"}]
        });
    });
}

Use llmobs.WithAnnotatedPrompt to attach prompt metadata to an LLM span. Requires dd-trace-go v2.8.0+. For more details on span annotation, see Enriching spans.

Prompt structure

llmobs.Prompt has the following fields:

ID (string): Logical identifier for this prompt. Should be unique per ML app.
Version (string): Version tag for the prompt, for example "1.0.0". See version tracking.
Label (string): Deployment label for the prompt, for example "production" or "staging".
Template (string): Template string with placeholders, for example "Translate {{text}} to {{lang}}". Mutually exclusive with ChatTemplate; if both are set, Template is dropped.
ChatTemplate ([]llmobs.LLMMessage): Multi-message template form. Mutually exclusive with Template.
Variables (map[string]string): Variables used to populate the template placeholders.
Tags (map[string]string): Tags to attach to the prompt run.
RAGContextVariables ([]string): Variable keys that contain ground-truth or context content. Used for hallucination detection.
RAGQueryVariables ([]string): Variable keys that contain the user query. Used for hallucination detection.

Example: chat-template prompt

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func answerQuestion(ctx context.Context, text string) string {
	span, _ := llmobs.StartLLMSpan(ctx, "answer-question",
		llmobs.WithModelName("gpt-5.1"),
		llmobs.WithModelProvider("openai"),
	)
	defer span.Finish()

	span.Annotate(llmobs.WithAnnotatedPrompt(llmobs.Prompt{
		ID:      "translation-template",
		Version: "1.0.0",
		ChatTemplate: []llmobs.LLMMessage{
			{Role: "user", Content: "Translate to {{lang}}: {{text}}"},
		},
		Variables: map[string]string{"lang": "fr", "text": text},
		Tags:      map[string]string{"team": "nlp"},
	}))

	completion := callProvider(text) // user application logic
	span.AnnotateLLMIO(
		[]llmobs.LLMMessage{{Role: "user", Content: "Translate to fr: " + text}},
		[]llmobs.LLMMessage{{Role: "assistant", Content: completion}},
	)
	return completion
}

Notes

Annotating a prompt is only available on LLM spans.
Place the annotation immediately before the provider call so it applies to the correct LLM span.
Use a unique prompt id to distinguish different prompts within your application.
Keep templates static by using placeholder syntax (like {{variable_name}}) and define dynamic content in the variables section.
For multiple auto-instrumented LLM calls within a block, use an annotation context to apply the same prompt metadata across calls. See Annotating auto-instrumented spans.

Version tracking

Agent Observability provides automatic versioning for your prompts when no explicit version is specified. When you provide a template or chat_template in your prompt metadata without a version tag, the system automatically generates a version by computing a hash of the template content. If you do provide a version tag, Agent Observability uses your specified version label instead of auto-generating one.

The versioning system works as follows:

Auto versioning: When no version tag is provided, Agent Observability computes a hash of the template or chat_template content to automatically generate a numerical version identifier
Manual versioning: When a version tag is provided, Agent Observability uses your specified version label exactly as provided
Version history: Both auto-generated and manual versions are maintained in the version history to track prompt evolution over time

This gives you the flexibility to either rely on automatic version management based on template content changes, or maintain full control over versioning with your own version labels.

MCP intent capture

To gain insight into why your MCP tools were called, enable intent capture on your MCP server. When enabled, the SDK adds an argument to every MCP server tool requesting that the calling model describe why it chose to call the tool. The intent is recorded on the tool span, helping you improve your tool definitions and descriptions.

Enable MCP intent capture with the DD_MCP_CAPTURE_INTENT environment variable:

DD_MCP_CAPTURE_INTENT=1 DD_SITE=<YOUR_DATADOG_SITE> DD_API_KEY=<YOUR_API_KEY> DD_LLMOBS_ENABLED=1 \
DD_LLMOBS_ML_APP=<YOUR_ML_APP_NAME> ddtrace-run <YOUR_APP_STARTUP_COMMAND>

Or, enable it programmatically with the capture_intent parameter on LLMObs.enable():

from ddtrace.llmobs import LLMObs
LLMObs.enable(
  ml_app="<YOUR_ML_APP_NAME>",
  capture_intent=True,
)

Cost monitoring

Attach token metrics (for automatic cost tracking) or cost metrics (for manual cost tracking) to your LLM/embedding spans. Token metrics allow Datadog to calculate costs using provider pricing, while cost metrics let you supply your own pricing when using custom or unsupported models. For more details, see Costs.

If you’re using automatic instrumentation, token and cost metrics appear on your spans automatically. If you’re instrumenting manually, follow the guidance below.

In this context, "token metrics" and "cost metrics" refer to numeric key-value pairs you attach to spans through the metrics parameter of the LLMObs.annotate() method. These are distinct from Datadog platform Agent Observability metrics. For recognized keys such as input_tokens, output_tokens, input_cost, and output_cost, Datadog uses these span attributes to generate corresponding platform metrics (such as ml_obs.span.llm.input.cost) for use in dashboards and monitors.

Use case: Using a common model provider

Datadog supports common model providers such as OpenAI, Azure OpenAI, Anthropic, and Google Gemini. When using these providers, you only need to annotate your LLM request with the model name, model provider, and token usage. Datadog automatically calculates the estimated cost based on the provider’s pricing.

To learn more about what each token represents and how Datadog calculates them, see How token counts are calculated.

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm

@llm(model_name="gpt-5.1", model_provider="openai")
def llm_call(prompt):
    resp = ... # llm call here
    # Annotate token metrics
    LLMObs.annotate(
        metrics={
          "input_tokens": 50,
          "output_tokens": 120,
          "total_tokens": 170,
          "non_cached_input_tokens": 13,  # optional
          "cache_read_input_tokens": 22,  # optional
          "cache_write_input_tokens": 15, # optional
        },
    )
    return resp

function llmCall (prompt) {
  const resp = ... // llm call here
  llmobs.annotate({
    metrics: {
      input_tokens: 50,
      output_tokens: 120,
      total_tokens: 170,
      non_cached_input_tokens: 13,  // optional
      cache_read_input_tokens: 22,  // optional
      cache_write_input_tokens: 15  // optional
    }
  })
  return resp
}
llmCall = llmobs.wrap({ kind: 'llm', modelName: 'gpt-5.1', modelProvider: 'openai' }, llmCall)

import datadog.trace.api.llmobs.LLMObs;
import datadog.trace.api.llmobs.LLMObsSpan;
import java.util.Map;

public class MyJavaClass {
  public String llmCall(String prompt) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("llm-call", "gpt-5.1", "openai", null, null);
    String resp = ... // llm call here
    llmSpan.setMetrics(Map.of(
      "input_tokens", 50,
      "output_tokens", 120,
      "total_tokens", 170,
      "non_cached_input_tokens", 13,  // optional
      "cache_read_input_tokens", 22,  // optional
      "cache_write_input_tokens", 15  // optional
    ));
    llmSpan.finish();
    return resp;
  }
}

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func llmCall(ctx context.Context, prompt string) string {
	span, _ := llmobs.StartLLMSpan(ctx, "llm-call",
		llmobs.WithModelName("gpt-5.1"),
		llmobs.WithModelProvider("openai"),
	)
	defer span.Finish()

	resp := callProvider(prompt) // llm call here

	span.Annotate(llmobs.WithAnnotatedMetrics(map[string]float64{
		llmobs.MetricKeyInputTokens:           50,
		llmobs.MetricKeyOutputTokens:          120,
		llmobs.MetricKeyTotalTokens:           170,
		llmobs.MetricKeyCacheReadInputTokens:  22, // optional
		llmobs.MetricKeyCacheWriteInputTokens: 15, // optional
	}))
	return resp
}

Note: dd-trace-go has no LLM provider integrations, so token counts are never captured automatically. Record them on every llm and embedding span; otherwise, Datadog cannot estimate cost.

Use case: Using a custom model

For custom or unsupported models, you must annotate the span manually with the cost data in dollars.

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm

@llm(model_name="custom_model", model_provider="model_provider")
def llm_call(prompt):
    resp = ... # llm call here
    # Annotate cost metrics
    LLMObs.annotate(
        metrics={
          "input_cost": 3,
          "output_cost": 7,
          "total_cost": 10,
          "non_cached_input_cost": 1,    # optional
          "cache_read_input_cost": 0.6,  # optional
          "cache_write_input_cost": 1.4, # optional
        },
    )
    return resp

function llmCall (prompt) {
  const resp = ... // llm call here
  llmobs.annotate({
    metrics: {
      input_cost: 3,
      output_cost: 7,
      total_cost: 10,
      non_cached_input_cost: 1,    // optional
      cache_read_input_cost: 0.6,  // optional
      cache_write_input_cost: 1.4  // optional
    }
  })
  return resp
}
llmCall = llmobs.wrap({ kind: 'llm', modelName: 'custom_model', modelProvider: 'model_provider' }, llmCall)

import datadog.trace.api.llmobs.LLMObs;
import datadog.trace.api.llmobs.LLMObsSpan;
import java.util.Map;

public class MyJavaClass {
  public String llmCall(String prompt) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("llm-call", "custom_model", "model_provider", null, null);
    String resp = ... // llm call here
    llmSpan.setMetrics(Map.of(
      "input_cost", 3,
      "output_cost", 7,
      "total_cost", 10,
      "non_cached_input_cost", 1,    // optional
      "cache_read_input_cost", 0.6,  // optional
      "cache_write_input_cost", 1.4  // optional
    ));
    llmSpan.finish();
    return resp;
  }
}

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func llmCall(ctx context.Context, prompt string) string {
	span, _ := llmobs.StartLLMSpan(ctx, "llm-call",
		llmobs.WithModelName("my-custom-model"),
		llmobs.WithModelProvider("my-company"),
	)
	defer span.Finish()

	resp := callProvider(prompt) // llm call here

	span.Annotate(llmobs.WithAnnotatedMetrics(map[string]float64{
		"input_cost":  0.0021,
		"output_cost": 0.0048,
	}))
	return resp
}

Adding custom tags to cost and tokens metrics

By default, the LLM cost and token metrics carry a fixed set of OOTB tags such as model_name, model_provider, and ml_app. To slice LLM spend by attributes specific to your application — such as team, customer, or feature — mark a subset of the span’s existing tag keys to propagate to those metrics as custom tags. For example use cases like custom dashboards and monitors, see Custom tags on cost and tokens metrics.

Each entry must be a string and must reference a key already supplied through the span’s tags parameter at the time the annotation is applied. When annotating a single span, the key can be supplied through tags in the same annotation call or in an earlier annotation on the same span. When using an annotation context, only keys present in tags at span start qualify — keys added later through individual span annotations are not retained. Entries that don’t reference an existing tag key are skipped.

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm

@llm(model_name="gpt-5.1", model_provider="openai")
def llm_call(prompt):
    resp = ... # llm call here
    LLMObs.annotate(
        metrics={"input_tokens": 50, "output_tokens": 120, "total_tokens": 170},
        tags={"team": "nlp", "customer_tier": "enterprise", "host": "host_name"},
        cost_tags=["team", "customer_tier"],
    )
    return resp

function llmCall (prompt) {
  const resp = ... // llm call here
  llmobs.annotate({
    metrics: { input_tokens: 50, output_tokens: 120, total_tokens: 170 },
    tags: { team: 'nlp', customer_tier: 'enterprise', host: 'host_name' },
    costTags: ['team', 'customer_tier']
  })
  return resp
}
llmCall = llmobs.wrap({ kind: 'llm', modelName: 'gpt-5.1', modelProvider: 'openai' }, llmCall)

Requires dd-trace-go v2.10.0+.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func llmCall(ctx context.Context, prompt string) string {
	span, _ := llmobs.StartLLMSpan(ctx, "llm-call",
		llmobs.WithModelName("gpt-5.1"),
		llmobs.WithModelProvider("openai"),
	)
	defer span.Finish()

	resp := callProvider(prompt) // llm call here

	span.Annotate(
		llmobs.WithAnnotatedMetrics(map[string]float64{
			llmobs.MetricKeyInputTokens:  50,
			llmobs.MetricKeyOutputTokens: 120,
			llmobs.MetricKeyTotalTokens:  170,
		}),
		llmobs.WithAnnotatedTags(map[string]string{
			"team":          "nlp",
			"customer_tier": "enterprise",
			"host":          "host_name",
		}),
		llmobs.WithAnnotatedCostTagKeys([]string{"team", "customer_tier"}),
	)
	return resp
}

You can also propagate tags this way through an annotation context to apply it to all auto-instrumented spans started inside the context.

with LLMObs.annotation_context(
    tags={"team": "nlp", "customer_tier": "enterprise"},
    cost_tags=["team", "customer_tier"],
):
    resp = ... # llm call here

llmobs.annotationContext({
  tags: { team: 'nlp', customer_tier: 'enterprise' },
  costTags: ['team', 'customer_tier']
}, () => {
  const resp = ... // llm call here
})

Evaluations

The Agent Observability SDK provides methods to export and submit your evaluations to Datadog.

For building reusable, class-based evaluators (BaseEvaluator, BaseSummaryEvaluator) with rich result metadata, see the Evaluation Developer Guide.

Evaluations must be joined to a single span. You can identify the target span using either of these two methods:

Tag-based joining - Join an evaluation using a unique key-value tag pair that is set on a single span. The evaluation will fail to join if the tag key-value pair matches multiple spans or no spans.
Direct span reference - Join an evaluation using the span’s unique trace ID and span ID combination.

Exporting a span

The Go SDK has no span export method. Pass the span directly to llmobs.SubmitEvaluationFromSpan instead. See Submitting evaluations.

LLMObs.export_span() can be used to extract the span context from a span. This method is helpful for associating your evaluation with the corresponding span.

Arguments

The LLMObs.export_span() method accepts the following argument:

span: optional - Span
The span to extract the span context (span and trace IDs) from. If not provided (as when using function decorators), the SDK exports the current active span.

Example

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm

@llm(model_name="claude", name="invoke_llm", model_provider="anthropic")
def llm_call():
    completion = ... # user application logic to invoke LLM
    span_context = LLMObs.export_span(span=None)
    return completion

llmobs.exportSpan() can be used to extract the span context from a span. You’ll need to use this method to associate your evaluation with the corresponding span.

Arguments

The llmobs.exportSpan() method accepts the following argument:

span: optional - Span
The span to extract the span context (span and trace IDs) from. If not provided (as when using function wrappers), the SDK exports the current active span.

Example

function llmCall () {
  const completion = ... // user application logic to invoke LLM
  const spanContext = llmobs.exportSpan()
  return completion
}
llmCall = llmobs.wrap({ kind: 'llm', name: 'invokeLLM', modelName: 'claude', modelProvider: 'anthropic' }, llmCall)

Submitting evaluations

LLMObs.submit_evaluation() can be used to submit your custom evaluation associated with a given span.

LLMObs.submit_evaluation_for is deprecated and will be removed in the next major version of ddtrace (4.0). To migrate, rename your LLMObs.submit_evaluation_for calls with LLMObs.submit_evaluation.

Note: Custom evaluations are evaluators that you implement and host yourself. These differ from out-of-the-box evaluations, which are automatically computed by Datadog using built-in evaluators. To configure out-of-the-box evaluations for your application, use the Agent Observability > Settings > Evaluations page in Datadog.

The LLMObs.submit_evaluation() method accepts the following arguments:

Arguments

label: required - string
The name of the evaluation.
metric_type: required - string
The type of the evaluation. Must be categorical, score, boolean or json.
value: required - string, numeric type, or dict
The value of the evaluation. Must be a string (metric_type==categorical), integer/float (metric_type==score), boolean (metric_type==boolean), or dict (metric_type==json).
span: optional - dictionary
A dictionary that uniquely identifies the span associated with this evaluation. Must contain span_id (string) and trace_id (string). Use LLMObs.export_span() to generate this dictionary.
span_with_tag_value: optional - dictionary
A dictionary that uniquely identifies the span associated with this evaluation. Must contain tag_key (string) and tag_value (string).
Note: Exactly one of span or span_with_tag_value is required. Supplying both, or neither, raises a ValueError.
ml_app: required - string
The name of the ML application.
timestamp_ms: optional - integer
The unix timestamp in milliseconds when the evaluation metric result was generated. If not provided, this defaults to the current time.
tags: optional - dictionary
A dictionary of string key-value pairs that users can add as tags regarding the evaluation. For more information about tags, see Getting Started with Tags.
assessment: optional - string
An assessment of this evaluation. Accepted values are pass and fail.
reasoning: optional - string
A text explanation of the evaluation result.
metadata: optional - dictionary
A dictionary containing arbitrary structured metadata associated with the evaluation result.

Example

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm

@llm(model_name="claude", name="invoke_llm", model_provider="anthropic")
def llm_call():
    completion = ... # user application logic to invoke LLM

    # joining an evaluation to a span via a tag key-value pair
    msg_id = get_msg_id()
    LLMObs.annotate(
        tags = {'msg_id': msg_id}
    )

    LLMObs.submit_evaluation(
        span_with_tag_value = {
            "tag_key": "msg_id",
            "tag_value": msg_id
        },
        ml_app = "chatbot",
        label="harmfulness",
        metric_type="score",
        value=10,
        tags={"evaluation_provider": "ragas"},
        assessment="fail",
        reasoning="Malicious intent was detected in the user instructions.",
        metadata={"details": ["jailbreak", "SQL injection"]}
    )

    # joining an evaluation to a span via span ID and trace ID
    span_context = LLMObs.export_span(span=None)
    LLMObs.submit_evaluation(
        span_context = span_context,
        ml_app = "chatbot",
        label="harmfulness",
        metric_type="score",
        value=10,
        tags={"evaluation_provider": "ragas"},
        assessment="fail",
        reasoning="Malicious intent was detected in the user instructions.",
        metadata={"details": ["jailbreak", "SQL injection"]}
    )
    return completion

llmobs.submitEvaluation() can be used to submit your custom evaluation associated with a given span.

The llmobs.submitEvaluation() method accepts the following arguments:

Arguments

span_context: required - dictionary
The span context to associate the evaluation with. This should be the output of LLMObs.export_span().
evaluationOptions: required - object
An object of the evaluation data.

The evaluationOptions object can contain the following:

label: required - string
The name of the evaluation.
metricType: required - string
The type of the evaluation. Must be one of “categorical”, “score”, “boolean” or “json”.
value: required - string or numeric type
The value of the evaluation. Must be a string (for categorical metric_type), number (for score metric_type), boolean (for boolean metric_type), or a JSON object (for json metric_type).
tags: optional - dictionary
A dictionary of string key-value pairs that users can add as tags regarding the evaluation. For more information about tags, see Getting Started with Tags.
assessment: optional - string
An assessment of this evaluation. Accepted values are pass and fail.
reasoning: optional - string
A text explanation of the evaluation result.
metadata: optional - dictionary
A JSON object containing arbitrary structured metadata associated with the evaluation result.

Example

function llmCall () {
  const completion = ... // user application logic to invoke LLM
  const spanContext = llmobs.exportSpan()
  llmobs.submitEvaluation(spanContext, {
    label: "harmfulness",
    metricType: "score",
    value: 10,
    tags: { evaluationProvider: "ragas" }
  })
  return completion
}
llmCall = llmobs.wrap({ kind: 'llm', name: 'invokeLLM', modelName: 'claude', modelProvider: 'anthropic' }, llmCall)

Use LLMObs.SubmitEvaluation() to submit your custom evaluation associated with a given span.

The LLMObs.SubmitEvaluation() method accepts the following arguments:

Arguments

llmObsSpan: required - LLMObsSpan
The span context to associate the evaluation with.
label: required - String
The name of the evaluation.
categoricalValue or scoreValue: required - String or double
The value of the evaluation. Must be a string (for categorical evaluations) or a double (for score evaluations).
tags: optional - Map<String, Object>
A dictionary of string key-value pairs used to tag the evaluation. For more information about tags, see Getting Started with Tags.

Example

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeChat(String userInput) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    String chatResponse = "N/A";
    try {
      chatResponse = ... // user application logic to invoke LLM
    } catch (Exception e) {
      llmSpan.addThrowable(e);
      throw new RuntimeException(e);
    } finally {
      llmSpan.finish();

      // submit evaluations
      LLMObs.SubmitEvaluation(llmSpan, "toxicity", "toxic", Map.of("language", "english"));
      LLMObs.SubmitEvaluation(llmSpan, "f1-similarity", 0.02, Map.of("provider", "f1-calculator"));
    }
    return chatResponse;
  }
}

The Go SDK provides two functions for submitting evaluations. Both infer the evaluation type from the Go type of the value you pass: bool becomes a boolean evaluation, string becomes categorical, and any numeric type becomes score.

Note: The Go SDK does not support json evaluations.

Direct span reference

llmobs.SubmitEvaluationFromSpan takes the span itself, so no separate export step is needed.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func evaluate(ctx context.Context) {
	span, _ := llmobs.StartLLMSpan(ctx, "invoke-llm",
		llmobs.WithModelName("claude-opus-4-6"),
		llmobs.WithModelProvider("anthropic"),
	)
	defer span.Finish()

	// ... user application logic

	llmobs.SubmitEvaluationFromSpan("correctness", "correct", span) // categorical
	llmobs.SubmitEvaluationFromSpan("relevance_score", 0.95, span)  // score
	llmobs.SubmitEvaluationFromSpan("is_safe", true, span)          // boolean
}

Tag-based joining

llmobs.SubmitEvaluationFromTag joins the evaluation to the span carrying a matching tag key-value pair. The evaluation fails to join if the pair matches multiple spans or no spans.

joinTag := llmobs.JoinTag{Key: "session_id", Value: sessionID}

llmobs.SubmitEvaluationFromTag("user_feedback", "positive", joinTag)
llmobs.SubmitEvaluationFromTag("user_rating", 4.2, joinTag)
llmobs.SubmitEvaluationFromTag("thumbs_up", true, joinTag)

Options

Both functions accept the following llmobs.EvaluationOption values:

llmobs.WithEvaluationTags: optional - []string
A list of tags, formatted as key:value strings, to attach to the evaluation.
llmobs.WithEvaluationMLApp: optional - string
The name of the ML application. Defaults to the ML app configured for the SDK.
llmobs.WithEvaluationTimestamp: optional - time.Time
The time the evaluation was generated. Defaults to the current time.

llmobs.SubmitEvaluationFromSpan("is_safe", true, span,
	llmobs.WithEvaluationTags([]string{"evaluator:custom", "env:prod"}),
	llmobs.WithEvaluationMLApp("<YOUR_ML_APP_NAME>"),
)

Note: These functions do not return an error. Failures are reported through the tracer’s debug log. Set DD_TRACE_DEBUG=1 to surface them.

Submitting end-user feedback

End-user feedback captures input from the users of your LLM application, such as thumbs-up or thumbs-down ratings, whether a user accepted an agent’s change, and free-text comments. Unlike an evaluation, feedback carries the identity of the submitter and can target a span, a trace, a session, or a customer-defined entity. For more information, see End-User Feedback.

Use LLMObs.submit_feedback() to submit end-user feedback associated with a span, trace, session, or customer-defined entity.

The LLMObs.submit_feedback() method accepts the following arguments:

Arguments

label: required - string
The name of the feedback metric. Must not contain a ..
metric_type: required - string
The type of the feedback. Must be categorical, score, boolean, json, or text.
value: required - string, numeric type, boolean, or dict
The value of the feedback. Must be a string (metric_type==categorical or metric_type==text), integer or float (metric_type==score), boolean (metric_type==boolean), or dict (metric_type==json).
submitter: required - dictionary
A dictionary that identifies who submitted the feedback. Must contain a non-empty id (string), and can contain an optional type (string), such as user.
span: optional - dictionary
A dictionary that identifies the span associated with this feedback. Use LLMObs.export_span() to generate this dictionary.
span_id: optional - string
The ID of the span associated with this feedback.
trace_id: optional - string
The ID of the trace associated with this feedback.
session_id: optional - string
The ID of the session associated with this feedback.
feedback_join_key: optional - string
A customer-defined key associated with this feedback, such as an incident ID or a ticket ID. To connect the feedback to your spans, first annotate them with a feedback_join_key tag holding the same value. See Enriching spans.
Note: Exactly one of span, span_id, trace_id, session_id, or feedback_join_key is required. Supplying more than one, or none, raises a ValueError.
ml_app: optional - string
The name of the ML application. If not provided, this defaults to the ML application configured for the SDK.
timestamp_ms: optional - integer
The Unix timestamp in milliseconds when the feedback was generated. If not provided, this defaults to the current time.
tags: optional - dictionary
A dictionary of string key-value pairs that users can add as tags regarding the feedback. For more information about tags, see Getting Started with Tags.
assessment: optional - string
An assessment of this feedback. Accepted values are pass and fail.
reasoning: optional - string
A text explanation of the feedback.

Example

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm

@llm(model_name="claude", name="invoke_llm", model_provider="anthropic")
def llm_call():
    completion = ... # user application logic to invoke LLM
    span_context = LLMObs.export_span(span=None)

    # submitting feedback for a trace
    LLMObs.submit_feedback(
        label="thumbs",
        metric_type="categorical",
        value="down",
        submitter={"id": "user-123", "type": "user"},
        trace_id=span_context["trace_id"],
        assessment="fail",
    )

    # connecting the span to a customer-defined entity
    LLMObs.annotate(tags={"feedback_join_key": "incident-123"})

    # submitting feedback for that entity
    LLMObs.submit_feedback(
        label="user_comment",
        metric_type="text",
        value="The investigation missed the customer impact.",
        submitter={"id": "user-123", "type": "user"},
        feedback_join_key="incident-123",
    )
    return completion

Use llmobs.submitFeedback() to submit end-user feedback associated with a span, trace, session, or customer-defined entity.

The llmobs.submitFeedback() method accepts an options object with the following properties:

Arguments

label: required - string
The name of the feedback metric. Must not contain a ..
metricType: required - string
The type of the feedback. Must be one of categorical, score, boolean, json, or text.
value: required - string, number, boolean, or object
The value of the feedback. Must be a string (for categorical and text metric types), number (for score), boolean (for boolean), or a JSON object (for json).
submitter: required - object
An object that identifies who submitted the feedback. Must contain a non-empty id (string), and can contain an optional type (string), such as user.
span: optional - object
The span context of the span to attach the feedback to. This should be the output of llmobs.exportSpan().
spanId: optional - string
The ID of the span to attach the feedback to.
traceId: optional - string
The ID of the trace to attach the feedback to.
sessionId: optional - string
The ID of the session to attach the feedback to.
feedbackJoinKey: optional - string
A customer-defined key to attach the feedback to, such as an incident ID or a ticket ID. Set the same key on your spans to connect the feedback to them.
Note: Exactly one of span, spanId, traceId, sessionId, or feedbackJoinKey is required. Supplying more than one, or none, throws an error.
mlApp: optional - string
The name of the ML application. If not provided, this defaults to the ML application configured for the SDK.
timestampMs: optional - number
The Unix timestamp in milliseconds when the feedback was generated. If not provided, this defaults to the current time.
tags: optional - object
An object of string key-value pairs that users can add as tags regarding the feedback. For more information about tags, see Getting Started with Tags.
assessment: optional - string
An assessment of this feedback. Accepted values are pass and fail.
reasoning: optional - string
A text explanation of the feedback.

Example

function llmCall () {
  const completion = ... // user application logic to invoke LLM
  const spanContext = llmobs.exportSpan()

  // submitting feedback for a trace
  llmobs.submitFeedback({
    label: 'thumbs',
    metricType: 'boolean',
    value: true,
    submitter: { id: 'user-123', type: 'user' },
    traceId: spanContext.traceId,
    assessment: 'pass'
  })

  // connecting the span to a customer-defined entity
  llmobs.annotate({
    tags: { feedback_join_key: 'incident-123' }
  })

  // submitting feedback for that entity
  llmobs.submitFeedback({
    label: 'user_comment',
    metricType: 'text',
    value: 'This answer was helpful.',
    submitter: { id: 'user-123', type: 'user' },
    feedbackJoinKey: 'incident-123'
  })
  return completion
}
llmCall = llmobs.wrap({ kind: 'llm', name: 'invokeLLM', modelName: 'claude', modelProvider: 'anthropic' }, llmCall)

Use LLMObs.submitFeedback() to submit end-user feedback associated with a span, trace, session, or customer-defined entity. Build the feedback with LLMObs.Feedback.builder().

The builder accepts the following methods:

Arguments

label(String label): required
The name of the feedback metric. Must not contain a ..
categoricalValue(String), scoreValue(double), booleanValue(boolean), jsonValue(Map<String, Object>), or textValue(String): required
The value of the feedback. Set exactly one of these methods, which also determines the metric type.
submitter(String id, String type) or submitter(Submitter submitter): required
Identifies who submitted the feedback. The id must be a non-empty string. The type is an optional qualifier, such as user.
span(LLMObsSpan span), spanId(String), traceId(String), sessionId(String), or feedbackJoinKey(String): required
The entity to attach the feedback to. Set exactly one of these methods. Use feedbackJoinKey for a customer-defined entity, such as an incident ID or a ticket ID, and set the same key on your spans to connect the feedback to them.
mlApp(String mlApp): optional
The name of the ML application. If not provided, this defaults to the ML application configured for the tracer.
timestampMs(long timestampMs): optional
The Unix timestamp in milliseconds when the feedback was generated. If not provided, this defaults to the current time.
tags(Map<String, Object> tags) or tag(String key, Object value): optional
Key-value pairs used to tag the feedback. For more information about tags, see Getting Started with Tags.
assessment(Assessment assessment): optional
An assessment of this feedback. Accepted values are LLMObs.Feedback.Assessment.PASS and LLMObs.Feedback.Assessment.FAIL.
reasoning(String reasoning): optional
A text explanation of the feedback.

Note: LLMObs.submitFeedback() validates the feedback and throws an IllegalArgumentException when Agent Observability is enabled and the feedback is invalid, such as when the target, value, or submitter is missing. When Agent Observability is disabled, or the Agent is not attached, the call is a no-op.

Example

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String invokeChat(String userInput) {
    LLMObsSpan llmSpan = LLMObs.startLLMSpan("my-llm-span-name", "my-llm-model", "my-company", "maybe-ml-app-override", "session-141");
    String chatResponse = "N/A";
    try {
      chatResponse = ... // user application logic to invoke LLM
    } catch (Exception e) {
      llmSpan.addThrowable(e);
      throw new RuntimeException(e);
    } finally {
      // connecting the span to a customer-defined entity
      llmSpan.setTag("feedback_join_key", "incident-123");
      llmSpan.finish();

      // submitting feedback for a trace
      LLMObs.submitFeedback(
          LLMObs.Feedback.builder()
              .traceId(llmSpan.getTraceId().toString())
              .label("thumbs")
              .booleanValue(true)
              .submitter("user-123", "end_user")
              .assessment(LLMObs.Feedback.Assessment.PASS)
              .reasoning("answered the question")
              .build());

      // submitting feedback for that entity
      LLMObs.submitFeedback(
          LLMObs.Feedback.builder()
              .feedbackJoinKey("incident-123")
              .label("user_comment")
              .textValue("The answer missed the customer impact.")
              .submitter("user-123", "end_user")
              .assessment(LLMObs.Feedback.Assessment.FAIL)
              .build());
    }
    return chatResponse;
  }
}

Span processing

To modify input and output data on spans, you can configure a processor function. The processor function has access to span tags to enable conditional input/output modification. Processor functions can either return the modified span to emit it, or return None/null to prevent the span from being emitted entirely. This is useful for filtering out spans that contain sensitive data or meet certain criteria.

Example

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs import LLMObsSpan

def redact_processor(span: LLMObsSpan) -> LLMObsSpan:
    if span.get_tag("no_output") == "true":
        for message in span.output:
            message["content"] = ""
    return span


# If using LLMObs.enable()
LLMObs.enable(
  ...
  span_processor=redact_processor,
)
# else when using `ddtrace-run`
LLMObs.register_processor(redact_processor)

with LLMObs.llm("invoke_llm_with_no_output"):
    LLMObs.annotate(tags={"no_output": "true"})

Example: conditional modification with auto-instrumentation

When using auto instrumentation, the span is not always contextually accessible. To conditionally modify the inputs and outputs on auto-instrumented spans, annotation_context() can be used in addition to a span processor.

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs import LLMObsSpan

def redact_processor(span: LLMObsSpan) -> LLMObsSpan:
    if span.get_tag("no_input") == "true":
        for message in span.input:
            message["content"] = ""
    return span

LLMObs.register_processor(redact_processor)


def call_openai():
    with LLMObs.annotation_context(tags={"no_input": "true"}):
        # make call to openai
        ...

Example: preventing spans from being emitted

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs import LLMObsSpan
from typing import Optional

def filter_processor(span: LLMObsSpan) -> Optional[LLMObsSpan]:
    # Skip spans that are marked as internal or contain sensitive data
    if span.get_tag("internal") == "true" or span.get_tag("sensitive") == "true":
        return None  # This span will not be emitted

    # Process and return the span normally
    return span

LLMObs.register_processor(filter_processor)

# This span will be filtered out and not sent to Datadog
with LLMObs.workflow("internal_workflow"):
    LLMObs.annotate(tags={"internal": "true"})
    # ... workflow logic

Example

const tracer = require('dd-trace').init({
  llmobs: {
    mlApp: "<YOUR_ML_APP_NAME>"
  }
})

const llmobs = tracer.llmobs

function redactProcessor(span) {
  if (span.getTag("no_output") === "true") {
    for (const message of span.output) {
      message.content = ""
    }
  }
  return span
}

llmobs.registerProcessor(redactProcessor)

Example: conditional modification with auto-instrumentation

When using auto instrumentation, the span is not always contextually accessible. To conditionally modify the inputs and outputs on auto-instrumented spans, llmobs.annotationContext() can be used in addition to a span processor.

const { llmobs } = require('dd-trace');

function redactProcessor(span) {
  if (span.getTag("no_input") == "true") {
    for (const message of span.input) {
      message.content = "";
    }
  }

  return span;
}

llmobs.registerProcessor(redactProcessor);

async function callOpenai() {
  await llmobs.annotationContext({ tags: { no_input: "true" } }, async () => {
    // make call to openai
  });
}

Example: preventing spans from being emitted

const tracer = require('dd-trace').init({
  llmobs: {
    mlApp: "<YOUR_ML_APP_NAME>"
  }
})

const llmobs = tracer.llmobs

function filterProcessor(span) {
  // Skip spans that are marked as internal or contain sensitive data
  if (span.getTag("internal") === "true" || span.getTag("sensitive") === "true") {
    return null  // This span will not be emitted
  }

  // Process and return the span normally
  return span
}

llmobs.registerProcessor(filterProcessor)

// This span will be filtered out and not sent to Datadog
function internalWorkflow() {
  return llmobs.trace({ kind: 'workflow', name: 'internalWorkflow' }, (span) => {
    llmobs.annotate({ tags: { internal: "true" } })
    // ... workflow logic
  })
}

Tracking user sessions

Session tracking allows you to associate multiple interactions with a given user.

When starting a root span for a new trace or span in a new process, specify the session_id argument with the string ID of the underlying user session, which is submitted as a tag on the span. Optionally, you can also specify the user_handle, user_name, and user_id tags.

from ddtrace.llmobs.decorators import workflow

@workflow(session_id="<SESSION_ID>")
def process_user_message():
    LLMObs.annotate(
        ...
        tags = {"user_handle": "poodle@dog.com", "user_id": "1234", "user_name": "poodle"}
    )
    return

Session tracking tags

Tag	Description
`session_id`	The ID representing a single user session, for example, a chat session.
`user_handle`	The handle for the user of the chat session.
`user_name`	The name for the user of the chat session.
`user_id`	The ID for the user of the chat session.

When starting a root span for a new trace or span in a new process, specify the sessionId argument with the string ID of the underlying user session:

function processMessage() {
    ... # user application logic
    return
}
processMessage = llmobs.wrap({ kind: 'workflow', sessionId: "<SESSION_ID>" }, processMessage)

When starting a root span for a new trace or span in a new process, specify the sessionId argument with the string ID of the underlying user session:

import datadog.trace.api.llmobs.LLMObs;

public class MyJavaClass {
  public String processChat(int userID) {
    LLMObsSpan workflowSpan = LLMObs.startWorkflowSpan("incoming-chat", null, "session-" + System.currentTimeMillis() + "-" + userID);
    String chatResponse = answerChat(); // user application logic
    workflowSpan.annotateIO(...); // record the input and output
    workflowSpan.finish();
    return chatResponse;
  }
}

When starting a root span for a new trace or span in a new process, pass llmobs.WithSessionID with the string ID of the underlying user session. Optionally, add the user_handle, user_name, and user_id tags.

package main

import (
	"context"

	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func processChat(ctx context.Context, sessionID, message string) string {
	span, ctx := llmobs.StartWorkflowSpan(ctx, "incoming-chat",
		llmobs.WithSessionID(sessionID),
	)
	defer span.Finish()

	span.Annotate(llmobs.WithAnnotatedTags(map[string]string{
		"user_handle": "poodle@dog.com",
		"user_id":     "1234",
		"user_name":   "poodle",
	}))

	response := answerChat(ctx, message) // user application logic
	span.AnnotateTextIO(message, response)
	return response
}

You can also set the session ID on an existing span with llmobs.WithAnnotatedSessionID.

Session tracking tags

Tag	Description
`session_id`	The ID representing a single user session, for example, a chat session.
`user_handle`	The handle for the user of the chat session.
`user_name`	The name for the user of the chat session.
`user_id`	The ID for the user of the chat session.

Distributed tracing

The SDK supports tracing across distributed services or hosts. Distributed tracing works by propagating span information across web requests.

The ddtrace library provides some out-of-the-box integrations that support distributed tracing for popular web framework and HTTP libraries. If your application makes requests using these supported libraries, you can enable distributed tracing by running:

from ddtrace import patch
patch(<INTEGRATION_NAME>=True)

If your application does not use any of these supported libraries, you can enable distributed tracing by manually propagating span information to and from HTTP headers. The SDK provides the helper methods LLMObs.inject_distributed_headers() and LLMObs.activate_distributed_headers() to inject and activate tracing contexts in request headers.

Injecting distributed headers

The LLMObs.inject_distributed_headers() method takes a span and injects its context into the HTTP headers to be included in the request. This method accepts the following arguments:

request_headers: required - dictionary
The HTTP headers to extend with tracing context attributes.
span: optional - Span - default: The current active span.
The span to inject its context into the provided request headers. Any spans (including those with function decorators), this defaults to the current active span.

Activating distributed headers

The LLMObs.activate_distributed_headers() method takes HTTP headers and extracts tracing context attributes to activate in the new service.

Note: You must call LLMObs.activate_distributed_headers() before starting any spans in your downstream service. Spans started prior (including function decorator spans) do not get captured in the distributed trace.

This method accepts the following argument:

request_headers: required - dictionary
The HTTP headers to extract tracing context attributes.

Example

client.py

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import workflow

@workflow
def client_send_request():
    request_headers = {}
    request_headers = LLMObs.inject_distributed_headers(request_headers)
    send_request("<method>", request_headers)  # arbitrary HTTP call

server.py

from ddtrace.llmobs import LLMObs

def server_process_request(request):
    LLMObs.activate_distributed_headers(request.headers)
    with LLMObs.task(name="process_request") as span:
        pass  # arbitrary server work

The dd-trace library provides out-of-the-box integrations that support distributed tracing for popular web frameworks. Requiring the tracer automatically enables these integrations, but you can disable them optionally with:

const tracer = require('dd-trace').init({
  llmobs: { ... },
})
tracer.use('http', false) // disable the http integration

Agent Observability span context travels on the standard APM distributed tracing headers, so there is no separate inject or extract API. Propagate the APM context with tracer.Inject and tracer.Extract, or use an instrumented HTTP integration. Agent Observability spans in the downstream service then attach to the upstream parent span automatically.

Example

client.go

package main

import (
	"context"
	"net/http"

	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func clientSendRequest(ctx context.Context, req *http.Request) error {
	span, ctx := llmobs.StartWorkflowSpan(ctx, "client-send-request")
	defer span.Finish()

	apmSpan, _ := tracer.StartSpanFromContext(ctx, "http.request")
	defer apmSpan.Finish()

	// inject the trace context into the outgoing request headers
	return tracer.Inject(apmSpan.Context(), tracer.HTTPHeadersCarrier(req.Header))
}

server.go

package main

import (
	"net/http"

	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
	"github.com/DataDog/dd-trace-go/v2/llmobs"
)

func serverHandler(w http.ResponseWriter, r *http.Request) {
	var opts []tracer.StartSpanOption
	if sctx, err := tracer.Extract(tracer.HTTPHeadersCarrier(r.Header)); err == nil {
		opts = append(opts, tracer.ChildOf(sctx))
	}
	// if extraction fails, the span starts a new trace

	apmSpan, ctx := tracer.StartSpanFromContext(r.Context(), "http.handler", opts...)
	defer apmSpan.Finish()

	// this span joins the distributed trace and the upstream parent span
	span, _ := llmobs.StartTaskSpan(ctx, "process-request")
	defer span.Finish()
}

Note: Extract the upstream context before starting any Agent Observability spans in the downstream service. Spans started beforehand are not captured in the distributed trace.

Advanced tracing

Tracing spans using inline methods

For each span kind, the ddtrace.llmobs.LLMObs class provides a corresponding inline method to automatically trace the operation a given code block entails. These methods have the same argument signature as their function decorator counterparts, with the addition that name defaults to the span kind (llm, workflow, etc.) if not provided. These methods can be used as context managers to automatically finish the span after the enclosed code block is completed.

Example

from ddtrace.llmobs import LLMObs

def process_message():
    with LLMObs.workflow(name="process_message", session_id="<SESSION_ID>", ml_app="<ML_APP>") as workflow_span:
        ... # user application logic
    return

Persisting a span across contexts

To manually start and stop a span across different contexts or scopes:

Start a span manually using the same methods (for example, the LLMObs.workflow method for a workflow span), but as a plain function call rather than as a context manager.
Pass the span object as an argument to other functions.
Stop the span manually with the span.finish() method. Note: the span must be manually finished, otherwise it is not submitted.

Example

from ddtrace.llmobs import LLMObs

def process_message():
    workflow_span = LLMObs.workflow(name="process_message")
    ... # user application logic
    separate_task(workflow_span)
    return

def separate_task(workflow_span):
    ... # user application logic
    workflow_span.finish()
    return

Force flushing in serverless environments

LLMObs.flush() is a blocking function that submits all buffered Agent Observability data to the Datadog backend. This can be useful in serverless environments to prevent an application from exiting until all Agent Observability traces are submitted.

Tracing multiple applications

The SDK supports tracing multiple LLM applications from the same service.

You can configure an environment variable DD_LLMOBS_ML_APP to the name of your LLM application, which all generated spans are grouped into by default.

To override this configuration and use a different LLM application name for a given root span, pass the ml_app argument with the string name of the underlying LLM application when starting a root span for a new trace or a span in a new process.

from ddtrace.llmobs.decorators import workflow

@workflow(name="process_message", ml_app="<NON_DEFAULT_ML_APP_NAME>")
def process_message():
    ... # user application logic
    return

Tracing spans using inline methods

The llmobs SDK provides a corresponding inline method to automatically trace the operation a given code block entails. These methods have the same argument signature as their function wrapper counterparts, with the addition that name is required, as the name cannot be inferred from an anonymous callback. This method will finish the span under the following conditions:

If the function returns a Promise, then the span finishes when the promise is resolved or rejected.
If the function takes a callback as its last parameter, then the span finishes when that callback is called.
If the function doesn’t accept a callback and doesn’t return a Promise, then the span finishes at the end of the function execution.

Example without a callback

function processMessage () {
  return llmobs.trace({ kind: 'workflow', name: 'processMessage', sessionId: '<SESSION_ID>', mlApp: '<ML_APP>' }, workflowSpan => {
    ... // user application logic
    return
  })
}

Example with a callback

function processMessage () {
  return llmobs.trace({ kind: 'workflow', name: 'processMessage', sessionId: '<SESSION_ID>', mlApp: '<ML_APP>' }, (workflowSpan, cb) => {
    ... // user application logic
    let maybeError = ...
    cb(maybeError) // the span will finish here, and tag the error if it is not null or undefined
    return
  })
}

The return type of this function matches the return type of the traced function:

function processMessage () {
  const result = llmobs.trace({ kind: 'workflow', name: 'processMessage', sessionId: '<SESSION_ID>', mlApp: '<ML_APP>' }, workflowSpan => {
    ... // user application logic
    return 'hello world'
  })

  console.log(result) // 'hello world'
  return result
}

Function decorators in TypeScript

The Node.js Agent Observability SDK offers an llmobs.decorate function which serves as a function decorator for TypeScript applications. This functions tracing behavior is the same as llmobs.wrap.

Example

// index.ts
import tracer from 'dd-trace';
tracer.init({
  llmobs: {
    mlApp: "<YOUR_ML_APP_NAME>",
  },
});

const { llmobs } = tracer;

class MyAgent {
  @llmobs.decorate({ kind: 'agent' })
  async runChain () {
    ... // user application logic
    return
  }
}

Force flushing in serverless environments

llmobs.flush() is a blocking function that submits all buffered Agent Observability data to the Datadog backend. This can be useful in serverless environments to prevent an application from exiting until all Agent Observability traces are submitted.

Tracing multiple applications

The SDK supports tracing multiple LLM applications from the same service.

You can configure an environment variable DD_LLMOBS_ML_APP to the name of your LLM application, which all generated spans are grouped into by default.

To override this configuration and use a different LLM application name for a given root span, pass the mlApp argument with the string name of the underlying LLM application when starting a root span for a new trace or a span in a new process.

function processMessage () {
  ... // user application logic
  return
}
processMessage = llmobs.wrap({ kind: 'workflow', name: 'processMessage', mlApp: '<NON_DEFAULT_ML_APP_NAME>' }, processMessage)

Tracing multiple applications

The SDK supports tracing multiple LLM applications from the same service.

You can configure an environment variable DD_LLMOBS_ML_APP to the name of your LLM application, which all generated spans are grouped into by default.

To override this configuration and use a different LLM application name for a given root span, pass llmobs.WithMLApp when starting a root span for a new trace or a span in a new process.

span, ctx := llmobs.StartWorkflowSpan(ctx, "process-message",
	llmobs.WithMLApp("<NON_DEFAULT_ML_APP_NAME>"),
)
defer span.Finish()

Force flushing in serverless environments

The tracer flushes spans in the background. Call tracer.Stop() before your process exits so buffered spans are submitted. In short-lived processes and serverless handlers, use defer tracer.Stop() in main to guarantee this.

func main() {
	if err := tracer.Start(
		tracer.WithLLMObsEnabled(true),
		tracer.WithLLMObsMLApp("<YOUR_ML_APP_NAME>"),
	); err != nil {
		log.Fatalf("failed to start tracer: %v", err)
	}
	defer tracer.Stop()

	// your application logic
}

Application naming guidelines

Your application name (the value of DD_LLMOBS_ML_APP) must follow these guidelines:

Must be a lowercase Unicode string
Can be up to 193 characters long
Cannot contain contiguous or trailing underscores
Can contain the following characters:
- Alphanumerics
- Underscores
- Minuses
- Colons
- Periods
- Slashes