LLM Observability is not available in the US1-FED site.
LLM Observability is in public beta, and this API is subject to change. If changes occur, Datadog will provide release notes with any applicable upgrade instructions.
Overview
The LLM Observability API provides an interface for developers to send LLM-related traces and spans to Datadog. If your application is written in Python, you can use the LLM Observability SDK for Python.
Spans API
Use this endpoint to send spans to Datadog. For details on the available kinds of spans, see Span Kinds.
{"data":{"type":"span","attributes":{"ml_app":"weather-bot","session_id":"1","tags":["service:weather-bot","env:staging","user_handle:example-user@example.com","user_id:1234"],"spans":[{"parent_id":"undefined","trace_id":"<TEST_TRACE_ID>","span_id":"<AGENT_SPAN_ID>","name":"health_coach_agent","meta":{"kind":"agent","input":{"value":"What is the weather like today and do i wear a jacket?"},"output":{"value":"It's very hot and sunny, there is no need for a jacket"}},"start_ns":1713889389104152000,"duration":10000000000},{"parent_id":"<AGENT_SPAN_ID>","trace_id":"<TEST_TRACE_ID>","span_id":"<WORKFLOW_ID>","name":"qa_workflow","meta":{"kind":"workflow","input":{"value":"What is the weather like today and do i wear a jacket?"},"output":{"value":"It's very hot and sunny, there is no need for a jacket"}},"start_ns":1713889389104152000,"duration":5000000000},{"parent_id":"<WORKFLOW_SPAN_ID>","trace_id":"<TEST_TRACE_ID>","span_id":"<LLM_SPAN_ID>","name":"generate_response","meta":{"kind":"llm","input":{"messages":[{"role":"system","content":"Your role is to ..."},{"role":"user","content":"What is the weather like today and do i wear a jacket?"}]},"output":{"messages":[{"content":"It's very hot and sunny, there is no need for a jacket","role":"assistant"}]}},"start_ns":1713889389104152000,"duration":2000000000}]}}}
Response
If the request is successful, the API responds with a 202 network code and an empty body.
API standards
Error
Field
Type
Description
message
string
The error message.
stack
string
The stack trace.
type
string
The error type.
IO
Field
Type
Description
value
string
Input or output value. This should be used for all spans except for LLM spans.
Dict[key (string), value] where the value is a float, bool, or string
Data about the span that is not input or output related. Use the following metadata keys for LLM spans: temperature, max_tokens, model_name, and model_provider.
Metrics
Field
Type
Description
prompt_tokens
float64
The number of prompt tokens. Only valid for LLM spans.
completion_tokens
float64
The number of completion tokens. Only valid for LLM spans.
total_tokens
float64
The total number of tokens associated with the span. Only valid for LLM spans.
time_to_first_token
float64
The time in seconds it takes for the first output token to be returned in streaming-based LLM applications. Set for root spans.
time_per_output_token
float64
The time in seconds it takes for the per output token to be returned in streaming-based LLM applications. Set for root spans.
Span
Field
Type
Description
name [required]
string
The name of the span.
span_id [required]
string
An ID unique to the span.
trace_id [required]
string
A unique ID shared by all spans in the same trace.
parent_id [required]
string
ID of the span’s direct parent. If the span is a root span, the parent_id must be undefined.
The session the list of spans belongs to. Can be overridden or set on individual spans as well.
Tag
Tags should be formatted as a list of strings (for example, ["user_handle:dog@gmail.com", "app_version:1.0.0"]). They are meant to store contextual information surrounding the span.
Your application name (the value of DD_LLMOBS_ML_APP) must be a lowercase Unicode string. It may contain the characters listed below:
Alphanumerics
Underscores
Minuses
Colons
Periods
Slashes
The name can be up to 193 characters long and may not contain contiguous or trailing underscores.
Evaluations API
Use this endpoint to send evaluations associated with a given span to Datadog.
Endpoint
https://api./api/unstable/llm-obs/v1/eval-metric
Method
POST
Evaluations require a span_id and trace_id.
If you are not using the LLM Observability SDK, send the span_id and trace_id that you used to create your target span.
If you are using the LLM Observability SDK, obtain the span_id and trace_id by finding your target span, and accessing the root_span.span_id and the root_span.trace_id attributes.
