--- title: Search LLM Observability spans description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > API Reference > LLM Observability --- # Search LLM Observability spans{% #search-llm-observability-spans %} Copy pageCopied {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------ | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/spans/events/search | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/spans/events/search | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/spans/events/search | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | ### Overview Search LLM Observability spans using structured filters in the request body. ### Request #### Body Data (required) Search spans payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | -------------------- | ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an LLM Observability spans search request. | | data | attributes [*required*] | object | Attributes of an LLM Observability spans search request. | | attributes | filter | object | Filter criteria for an LLM Observability span search. | | filter | from | string | Start of the time range. Accepts ISO 8601 or relative format (e.g., `now-15m`). Defaults to `now-15m`. | | filter | ml_app | string | Filter by ML application name. | | filter | query | string | Search query using LLM Observability query syntax. Supports attribute filters using the field:value syntax (e.g. session_id, trace_id, ml_app, meta.span.kind). When provided, structured field filters (`span_id`, `trace_id`, etc.) are ignored. | | filter | span_id | string | Filter by exact span ID. | | filter | span_kind | string | Filter by span kind (e.g., llm, agent, tool, task, workflow). | | filter | span_name | string | Filter by span name. | | filter | tags | object | Filter by tag key-value pairs. | | additionalProperties | | string | | filter | to | string | End of the time range. Accepts ISO 8601 or relative format (e.g., `now`). Defaults to `now`. | | filter | trace_id | string | Filter by exact trace ID. | | attributes | options | object | Additional options for a span search request. | | options | include_attachments | boolean | Whether to include attachment data in the response. Defaults to `true`. | | options | time_offset | int64 | Offset in seconds applied to both `from` and `to` timestamps. | | attributes | page | object | Pagination settings for a span search request. | | page | cursor | string | Cursor from the previous response to retrieve the next page. | | page | limit | int64 | Maximum number of spans to return. Defaults to `10`. | | attributes | sort | string | Sort order for the results. Use `-` prefix for descending order. | | data | type [*required*] | enum | Resource type for an LLM Observability spans search request. Allowed enum values: `spans` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "filter": { "from": "now-900s", "ml_app": "my-llm-app", "query": "@session_id:abc123def456", "span_id": "abc123def456", "span_kind": "llm", "span_name": "llm_call", "tags": { "": "string" }, "to": "now", "trace_id": "trace-9a8b7c6d5e4f" }, "options": { "include_attachments": true, "time_offset": 0 }, "page": { "cursor": "eyJzdGFydCI6MTAwfQ==", "limit": 10 }, "sort": "-start_ns" }, "type": "spans" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of LLM Observability spans. | Parent field | Field | Type | Description | | -------------------- | ---------------------------- | -------- | ----------------------------------------------------------------------------------- | | | data [*required*] | [object] | List of spans matching the query. | | data | attributes [*required*] | object | Attributes of an LLM Observability span. | | attributes | duration [*required*] | double | Duration of the span in nanoseconds. | | attributes | evaluation | object | Evaluation metrics keyed by evaluator name. | | additionalProperties | | object | An evaluation metric associated with an LLM Observability span. | | | assessment | string | Assessment result (e.g., pass or fail). | | | eval_metric_type | string | Type of the evaluation metric (e.g., score, categorical, boolean). | | | reasoning | string | Human-readable reasoning for the evaluation result. | | | status | string | Status of the evaluation execution. | | | tags | [string] | Tags associated with the evaluation metric. | | | value | | Value of the evaluation result. | | attributes | input | object | Input or output content of an LLM Observability span. | | input | messages | [object] | List of messages in the input or output. | | messages | content | string | Text content of the message. | | messages | id | string | Unique identifier of the message. | | messages | role | string | Role of the message sender (e.g., user, assistant, system). | | messages | tool_calls | [object] | Tool calls made in this message. | | tool_calls | arguments | object | Arguments passed to the tool. | | tool_calls | name | string | Name of the tool called. | | tool_calls | tool_id | string | Identifier of the tool call. | | tool_calls | type | string | Type of the tool call. | | messages | tool_results | [object] | Tool results returned in this message. | | tool_results | name | string | Name of the tool that produced this result. | | tool_results | result | string | Result value returned by the tool. | | tool_results | tool_id | string | Identifier of the corresponding tool call. | | tool_results | type | string | Type of the tool result. | | input | value | string | Plain-text value of the input or output. | | attributes | intent | string | Detected intent of the span. | | attributes | metadata | object | Arbitrary metadata associated with the span. | | attributes | metrics | object | Numeric metrics associated with the span (e.g., token counts). | | additionalProperties | | double | | attributes | ml_app [*required*] | string | Name of the ML application this span belongs to. | | attributes | model_name | string | Name of the model used in this span. | | attributes | model_provider | string | Provider of the model used in this span. | | attributes | name [*required*] | string | Name of the span. | | attributes | output | object | Input or output content of an LLM Observability span. | | output | messages | [object] | List of messages in the input or output. | | messages | content | string | Text content of the message. | | messages | id | string | Unique identifier of the message. | | messages | role | string | Role of the message sender (e.g., user, assistant, system). | | messages | tool_calls | [object] | Tool calls made in this message. | | tool_calls | arguments | object | Arguments passed to the tool. | | tool_calls | name | string | Name of the tool called. | | tool_calls | tool_id | string | Identifier of the tool call. | | tool_calls | type | string | Type of the tool call. | | messages | tool_results | [object] | Tool results returned in this message. | | tool_results | name | string | Name of the tool that produced this result. | | tool_results | result | string | Result value returned by the tool. | | tool_results | tool_id | string | Identifier of the corresponding tool call. | | tool_results | type | string | Type of the tool result. | | output | value | string | Plain-text value of the input or output. | | attributes | parent_id | string | Identifier of the parent span, if any. | | attributes | span_id [*required*] | string | Unique identifier of the span. | | attributes | span_kind [*required*] | string | Kind of span (e.g., llm, agent, tool, task, workflow). | | attributes | start_ns [*required*] | int64 | Start time of the span in nanoseconds since Unix epoch. | | attributes | status [*required*] | string | Status of the span (e.g., ok, error). | | attributes | tags | [string] | Tags associated with the span. | | attributes | tool_definitions | [object] | Tool definitions available to the span. | | tool_definitions | description | string | Description of what the tool does. | | tool_definitions | name | string | Name of the tool. | | tool_definitions | schema | object | JSON schema describing the tool's input parameters. | | tool_definitions | version | string | Version of the tool definition. | | attributes | trace_id [*required*] | string | Trace identifier this span belongs to. | | data | id [*required*] | string | Unique identifier of the span. | | data | type [*required*] | enum | Resource type for an LLM Observability span. Allowed enum values: `span` | | | links | object | Pagination links accompanying the spans response. | | links | next | string | URL to retrieve the next page of results. | | | meta [*required*] | object | Metadata accompanying the spans response. | | meta | elapsed [*required*] | int64 | Time elapsed for the query in milliseconds. | | meta | page [*required*] | object | Pagination cursor for the spans response. | | page | after | string | Cursor to retrieve the next page of results. Absent when there are no more results. | | meta | request_id [*required*] | string | Unique identifier for the request. | | meta | status [*required*] | string | Status of the query execution. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "duration": 1500000000, "evaluation": { "": { "assessment": "pass", "eval_metric_type": "score", "reasoning": "string", "status": "string", "tags": [], "value": "undefined" } }, "input": { "messages": [ { "content": "string", "id": "string", "role": "string", "tool_calls": [ { "arguments": {}, "name": "string", "tool_id": "string", "type": "string" } ], "tool_results": [ { "name": "string", "result": "string", "tool_id": "string", "type": "string" } ] } ], "value": "string" }, "intent": "string", "metadata": {}, "metrics": { "": "number" }, "ml_app": "my-llm-app", "model_name": "gpt-4o", "model_provider": "openai", "name": "llm_call", "output": { "messages": [ { "content": "string", "id": "string", "role": "string", "tool_calls": [ { "arguments": {}, "name": "string", "tool_id": "string", "type": "string" } ], "tool_results": [ { "name": "string", "result": "string", "tool_id": "string", "type": "string" } ] } ], "value": "string" }, "parent_id": "string", "span_id": "abc123def456", "span_kind": "llm", "start_ns": 1705314600000000000, "status": "ok", "tags": [], "tool_definitions": [ { "description": "string", "name": "string", "schema": {}, "version": "string" } ], "trace_id": "trace-9a8b7c6d5e4f" }, "id": "abc123def456", "type": "span" } ], "links": { "next": "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events?page[cursor]=eyJzdGFydCI6MTAwfQ==" }, "meta": { "elapsed": 132, "page": { "after": "eyJzdGFydCI6MTAwfQ==" }, "request_id": "req-abc123", "status": "done" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## Search spans for an ML app # \# Curl command curl -X POST "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "filter": { "from": "now-900s", "ml_app": "my-llm-app", "span_kind": "llm", "to": "now" }, "options": { "include_attachments": true }, "page": { "limit": 10 } }, "type": "spans" } } EOF \## Search all spans in a session # \# Curl command curl -X POST "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "filter": { "from": "now-900s", "query": "@session_id:abc123def456", "to": "now" }, "options": { "include_attachments": true }, "page": { "limit": 50 } }, "type": "spans" } } EOF {% /tab %}