--- title: LLM Observability description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > API Reference > LLM Observability --- # LLM Observability Manage LLM Observability spans, data, projects, datasets, dataset records, experiments, and annotations. ## List LLM Observability projects{% #list-llm-observability-projects %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------ | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/projects | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/projects | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/projects | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/projects | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/projects | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/projects | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/projects | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects | ### Overview List all LLM Observability projects sorted by creation date, newest first. ### Arguments #### Query Strings | Name | Type | Description | | ------------ | ------- | --------------------------------------------------------------- | | filter[id] | string | Filter projects by project ID. | | filter[name] | string | Filter projects by name. | | page[cursor] | string | Use the Pagination cursor to retrieve the next page of results. | | page[limit] | integer | Maximum number of results to return per page. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of LLM Observability projects. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | [object] | List of projects. | | data | attributes [*required*] | object | Attributes of an LLM Observability project. | | attributes | created_at [*required*] | date-time | Timestamp when the project was created. | | attributes | description [*required*] | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | attributes | updated_at [*required*] | date-time | Timestamp when the project was last updated. | | data | id [*required*] | string | Unique identifier of the project. | | data | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | | | meta | object | Pagination cursor metadata. | | meta | after | string | Cursor for the next page of results. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "description": "", "name": "My LLM Project", "updated_at": "2024-01-15T10:30:00Z" }, "id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "type": "projects" } ], "meta": { "after": "string" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Create an LLM Observability project{% #create-an-llm-observability-project %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/projects | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/projects | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/projects | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/projects | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/projects | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/projects | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/projects | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects | ### Overview Create a new LLM Observability project. Returns the existing project if a name conflict occurs. ### Request #### Body Data (required) Create project payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | ------ | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for creating an LLM Observability project. | | data | attributes [*required*] | object | Attributes for creating an LLM Observability project. | | attributes | description | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | data | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "description": "string", "name": "My LLM Project" }, "type": "projects" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a single LLM Observability project. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability project. | | data | attributes [*required*] | object | Attributes of an LLM Observability project. | | attributes | created_at [*required*] | date-time | Timestamp when the project was created. | | attributes | description [*required*] | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | attributes | updated_at [*required*] | date-time | Timestamp when the project was last updated. | | data | id [*required*] | string | Unique identifier of the project. | | data | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "created_at": "2024-01-15T10:30:00Z", "description": "", "name": "My LLM Project", "updated_at": "2024-01-15T10:30:00Z" }, "id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "type": "projects" } } ``` {% /tab %} {% /tab %} {% tab title="201" %} Created {% tab title="Model" %} Response containing a single LLM Observability project. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability project. | | data | attributes [*required*] | object | Attributes of an LLM Observability project. | | attributes | created_at [*required*] | date-time | Timestamp when the project was created. | | attributes | description [*required*] | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | attributes | updated_at [*required*] | date-time | Timestamp when the project was last updated. | | data | id [*required*] | string | Unique identifier of the project. | | data | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "created_at": "2024-01-15T10:30:00Z", "description": "", "name": "My LLM Project", "updated_at": "2024-01-15T10:30:00Z" }, "id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "type": "projects" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "name": "My LLM Project" }, "type": "projects" } } EOF {% /tab %} ## Update an LLM Observability project{% #update-an-llm-observability-project %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------- | | ap1.datadoghq.com | PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/projects/{project_id} | | ap2.datadoghq.com | PATCH https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/projects/{project_id} | | app.datadoghq.eu | PATCH https://api.datadoghq.eu/api/v2/llm-obs/v1/projects/{project_id} | | app.ddog-gov.com | PATCH https://api.ddog-gov.com/api/v2/llm-obs/v1/projects/{project_id} | | us2.ddog-gov.com | PATCH https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/projects/{project_id} | | app.datadoghq.com | PATCH https://api.datadoghq.com/api/v2/llm-obs/v1/projects/{project_id} | | us3.datadoghq.com | PATCH https://api.us3.datadoghq.com/api/v2/llm-obs/v1/projects/{project_id} | | us5.datadoghq.com | PATCH https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects/{project_id} | ### Overview Partially update an existing LLM Observability project. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | ### Request #### Body Data (required) Update project payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | ------ | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for updating an LLM Observability project. | | data | attributes [*required*] | object | Attributes for updating an LLM Observability project. | | attributes | description | string | Updated description of the project. | | attributes | name | string | Updated name of the project. | | data | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "description": "string", "name": "string" }, "type": "projects" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a single LLM Observability project. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability project. | | data | attributes [*required*] | object | Attributes of an LLM Observability project. | | attributes | created_at [*required*] | date-time | Timestamp when the project was created. | | attributes | description [*required*] | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | attributes | updated_at [*required*] | date-time | Timestamp when the project was last updated. | | data | id [*required*] | string | Unique identifier of the project. | | data | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "created_at": "2024-01-15T10:30:00Z", "description": "", "name": "My LLM Project", "updated_at": "2024-01-15T10:30:00Z" }, "id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "type": "projects" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"\# Curl commandcurl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects/${project_id}" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "type": "projects" } } EOF {% /tab %} ## Delete LLM Observability projects{% #delete-llm-observability-projects %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | -------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/projects/delete | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/projects/delete | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/projects/delete | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/projects/delete | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/projects/delete | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/projects/delete | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/projects/delete | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects/delete | ### Overview Delete one or more LLM Observability projects. ### Request #### Body Data (required) Delete projects payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ----------------------------- | -------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for deleting LLM Observability projects. | | data | attributes [*required*] | object | Attributes for deleting LLM Observability projects. | | attributes | project_ids [*required*] | [string] | List of project IDs to delete. | | data | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "project_ids": [ "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ] }, "type": "projects" } } ``` {% /tab %} ### Response {% tab title="204" %} No Content {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/projects/delete" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "project_ids": [ "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ] }, "type": "projects" } } EOF {% /tab %} ## List LLM Observability datasets{% #list-llm-observability-datasets %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | ### Overview List all LLM Observability datasets for a project, sorted by creation date, newest first. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | #### Query Strings | Name | Type | Description | | ------------ | ------- | --------------------------------------------------------------- | | filter[name] | string | Filter datasets by name. | | filter[id] | string | Filter datasets by dataset ID. | | page[cursor] | string | Use the Pagination cursor to retrieve the next page of results. | | page[limit] | integer | Maximum number of results to return per page. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of LLM Observability datasets. | Parent field | Field | Type | Description | | ------------ | --------------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | [object] | List of datasets. | | data | attributes [*required*] | object | Attributes of an LLM Observability dataset. | | attributes | created_at [*required*] | date-time | Timestamp when the dataset was created. | | attributes | current_version [*required*] | int64 | Current version number of the dataset. | | attributes | description [*required*] | string | Description of the dataset. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | attributes | updated_at [*required*] | date-time | Timestamp when the dataset was last updated. | | data | id [*required*] | string | Unique identifier of the dataset. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | | | meta | object | Pagination cursor metadata. | | meta | after | string | Cursor for the next page of results. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "current_version": 1, "description": "", "metadata": {}, "name": "My LLM Dataset", "updated_at": "2024-01-15T10:30:00Z" }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "datasets" } ], "meta": { "after": "string" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Create an LLM Observability dataset{% #create-an-llm-observability-dataset %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | -------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets | ### Overview Create a new LLM Observability dataset within the specified project. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | ### Request #### Body Data (required) Create dataset payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | ------ | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for creating an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes for creating an LLM Observability dataset. | | attributes | description | string | Description of the dataset. | | attributes | metadata | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "description": "string", "metadata": {}, "name": "My LLM Dataset" }, "type": "datasets" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a single LLM Observability dataset. | Parent field | Field | Type | Description | | ------------ | --------------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes of an LLM Observability dataset. | | attributes | created_at [*required*] | date-time | Timestamp when the dataset was created. | | attributes | current_version [*required*] | int64 | Current version number of the dataset. | | attributes | description [*required*] | string | Description of the dataset. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | attributes | updated_at [*required*] | date-time | Timestamp when the dataset was last updated. | | data | id [*required*] | string | Unique identifier of the dataset. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "created_at": "2024-01-15T10:30:00Z", "current_version": 1, "description": "", "metadata": {}, "name": "My LLM Dataset", "updated_at": "2024-01-15T10:30:00Z" }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "datasets" } } ``` {% /tab %} {% /tab %} {% tab title="201" %} Created {% tab title="Model" %} Response containing a single LLM Observability dataset. | Parent field | Field | Type | Description | | ------------ | --------------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes of an LLM Observability dataset. | | attributes | created_at [*required*] | date-time | Timestamp when the dataset was created. | | attributes | current_version [*required*] | int64 | Current version number of the dataset. | | attributes | description [*required*] | string | Description of the dataset. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | attributes | updated_at [*required*] | date-time | Timestamp when the dataset was last updated. | | data | id [*required*] | string | Unique identifier of the dataset. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "created_at": "2024-01-15T10:30:00Z", "current_version": 1, "description": "", "metadata": {}, "name": "My LLM Dataset", "updated_at": "2024-01-15T10:30:00Z" }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "datasets" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "name": "My LLM Dataset" }, "type": "datasets" } } EOF {% /tab %} ## List LLM Observability dataset records{% #list-llm-observability-dataset-records %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | ### Overview List all records in an LLM Observability dataset, sorted by creation date, newest first. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | #### Query Strings | Name | Type | Description | | --------------- | ------- | ---------------------------------------------------------------------------------- | | filter[version] | integer | Retrieve records from a specific dataset version. Defaults to the current version. | | page[cursor] | string | Use the Pagination cursor to retrieve the next page of results. | | page[limit] | integer | Maximum number of results to return per page. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a paginated list of LLM Observability dataset records. | Parent field | Field | Type | Description | | --------------- | --------------------------------- | ------------------- | ----------------------------------------------------- | | | data [*required*] | [object] | List of dataset records. | | data | created_at [*required*] | date-time | Timestamp when the record was created. | | data | dataset_id [*required*] | string | Identifier of the dataset this record belongs to. | | data | expected_output [*required*] | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | data | id [*required*] | string | Unique identifier of the record. | | data | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | data | metadata [*required*] | object | Arbitrary metadata associated with the record. | | data | updated_at [*required*] | date-time | Timestamp when the record was last updated. | | | meta | object | Pagination cursor metadata. | | meta | after | string | Cursor for the next page of results. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {}, "updated_at": "2024-01-15T10:30:00Z" } ], "meta": { "after": "string" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/records" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Append records to an LLM Observability dataset{% #append-records-to-an-llm-observability-dataset %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ----------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | ### Overview Append one or more records to an LLM Observability dataset. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Request #### Body Data (required) Append records payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | --------------- | ---------------------------- | ------------------- | ---------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for appending records to an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes for appending records to an LLM Observability dataset. | | attributes | deduplicate | boolean | Whether to deduplicate records before appending. Defaults to `true`. | | attributes | records [*required*] | [object] | List of records to append to the dataset. | | records | expected_output | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | records | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | records | metadata | object | Arbitrary metadata associated with the record. | | data | type [*required*] | enum | Resource type of LLM Observability dataset records. Allowed enum values: `records` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "deduplicate": false, "records": [ { "expected_output": { "description": "undefined", "type": "undefined" }, "input": { "description": "undefined", "type": "undefined" }, "metadata": {} } ] }, "type": "records" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing records after a create or update operation. | Parent field | Field | Type | Description | | --------------- | --------------------------------- | ------------------- | ----------------------------------------------------- | | | data [*required*] | [object] | List of affected dataset records. | | data | records [*required*] | [object] | List of affected dataset records. | | records | created_at [*required*] | date-time | Timestamp when the record was created. | | records | dataset_id [*required*] | string | Identifier of the dataset this record belongs to. | | records | expected_output [*required*] | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | records | id [*required*] | string | Unique identifier of the record. | | records | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | records | metadata [*required*] | object | Arbitrary metadata associated with the record. | | records | updated_at [*required*] | date-time | Timestamp when the record was last updated. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "records": [ { "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {}, "updated_at": "2024-01-15T10:30:00Z" } ] } ] } ``` {% /tab %} {% /tab %} {% tab title="201" %} Created {% tab title="Model" %} Response containing records after a create or update operation. | Parent field | Field | Type | Description | | --------------- | --------------------------------- | ------------------- | ----------------------------------------------------- | | | data [*required*] | [object] | List of affected dataset records. | | data | records [*required*] | [object] | List of affected dataset records. | | records | created_at [*required*] | date-time | Timestamp when the record was created. | | records | dataset_id [*required*] | string | Identifier of the dataset this record belongs to. | | records | expected_output [*required*] | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | records | id [*required*] | string | Unique identifier of the record. | | records | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | records | metadata [*required*] | object | Arbitrary metadata associated with the record. | | records | updated_at [*required*] | date-time | Timestamp when the record was last updated. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "records": [ { "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {}, "updated_at": "2024-01-15T10:30:00Z" } ] } ] } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/records" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "type": "records" } } EOF {% /tab %} ## Update an LLM Observability dataset{% #update-an-llm-observability-dataset %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------------------------- | | ap1.datadoghq.com | PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | | ap2.datadoghq.com | PATCH https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | | app.datadoghq.eu | PATCH https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | | app.ddog-gov.com | PATCH https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | | us2.ddog-gov.com | PATCH https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | | app.datadoghq.com | PATCH https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | | us3.datadoghq.com | PATCH https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | | us5.datadoghq.com | PATCH https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id} | ### Overview Partially update an existing LLM Observability dataset within the specified project. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Request #### Body Data (required) Update dataset payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | ------ | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for updating an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes for updating an LLM Observability dataset. | | attributes | description | string | Updated description of the dataset. | | attributes | metadata | object | Updated metadata associated with the dataset. | | attributes | name | string | Updated name of the dataset. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "description": "string", "metadata": {}, "name": "string" }, "type": "datasets" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a single LLM Observability dataset. | Parent field | Field | Type | Description | | ------------ | --------------------------------- | --------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes of an LLM Observability dataset. | | attributes | created_at [*required*] | date-time | Timestamp when the dataset was created. | | attributes | current_version [*required*] | int64 | Current version number of the dataset. | | attributes | description [*required*] | string | Description of the dataset. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | attributes | updated_at [*required*] | date-time | Timestamp when the dataset was last updated. | | data | id [*required*] | string | Unique identifier of the dataset. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "created_at": "2024-01-15T10:30:00Z", "current_version": 1, "description": "", "metadata": {}, "name": "My LLM Dataset", "updated_at": "2024-01-15T10:30:00Z" }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "datasets" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "type": "datasets" } } EOF {% /tab %} ## Update LLM Observability dataset records{% #update-llm-observability-dataset-records %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------------------------ | | ap1.datadoghq.com | PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | ap2.datadoghq.com | PATCH https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.datadoghq.eu | PATCH https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.ddog-gov.com | PATCH https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us2.ddog-gov.com | PATCH https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | app.datadoghq.com | PATCH https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us3.datadoghq.com | PATCH https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | | us5.datadoghq.com | PATCH https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records | ### Overview Update one or more existing records in an LLM Observability dataset. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Request #### Body Data (required) Update records payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | --------------- | ---------------------------- | ------------------- | ---------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for updating records in an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes for updating records in an LLM Observability dataset. | | attributes | records [*required*] | [object] | List of records to update. | | records | expected_output | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | records | id [*required*] | string | Unique identifier of the record to update. | | records | input | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | records | metadata | object | Updated metadata associated with the record. | | data | type [*required*] | enum | Resource type of LLM Observability dataset records. Allowed enum values: `records` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "records": [ { "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {} } ] }, "type": "records" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing records after a create or update operation. | Parent field | Field | Type | Description | | --------------- | --------------------------------- | ------------------- | ----------------------------------------------------- | | | data [*required*] | [object] | List of affected dataset records. | | data | records [*required*] | [object] | List of affected dataset records. | | records | created_at [*required*] | date-time | Timestamp when the record was created. | | records | dataset_id [*required*] | string | Identifier of the dataset this record belongs to. | | records | expected_output [*required*] | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | records | id [*required*] | string | Unique identifier of the record. | | records | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | records | metadata [*required*] | object | Arbitrary metadata associated with the record. | | records | updated_at [*required*] | date-time | Timestamp when the record was last updated. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "records": [ { "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {}, "updated_at": "2024-01-15T10:30:00Z" } ] } ] } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/records" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "records": [ { "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c" } ] }, "type": "records" } } EOF {% /tab %} ## Delete LLM Observability datasets{% #delete-llm-observability-datasets %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/delete | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/delete | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/delete | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/delete | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/delete | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/delete | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/delete | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/delete | ### Overview Delete one or more LLM Observability datasets within the specified project. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | ### Request #### Body Data (required) Delete datasets payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ----------------------------- | -------- | ------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for deleting LLM Observability datasets. | | data | attributes [*required*] | object | Attributes for deleting LLM Observability datasets. | | attributes | dataset_ids [*required*] | [string] | List of dataset IDs to delete. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "dataset_ids": [ "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ] }, "type": "datasets" } } ``` {% /tab %} ### Response {% tab title="204" %} No Content {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/delete" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "dataset_ids": [ "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ] }, "type": "datasets" } } EOF {% /tab %} ## Delete LLM Observability dataset records{% #delete-llm-observability-dataset-records %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------------------------------ | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/records/delete | ### Overview Delete one or more records from an LLM Observability dataset. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Request #### Body Data (required) Delete records payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | -------- | ---------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for deleting records from an LLM Observability dataset. | | data | attributes [*required*] | object | Attributes for deleting records from an LLM Observability dataset. | | attributes | record_ids [*required*] | [string] | List of record IDs to delete. | | data | type [*required*] | enum | Resource type of LLM Observability dataset records. Allowed enum values: `records` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "record_ids": [ "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c" ] }, "type": "records" } } ``` {% /tab %} ### Response {% tab title="204" %} No Content {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/records/delete" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "record_ids": [ "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c" ] }, "type": "records" } } EOF {% /tab %} ## List LLM Observability experiments{% #list-llm-observability-experiments %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/experiments | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/experiments | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/experiments | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments | ### Overview List all LLM Observability experiments sorted by creation date, newest first. ### Arguments #### Query Strings | Name | Type | Description | | ------------------ | ------- | ----------------------------------------------------------------------------------- | | filter[project_id] | string | Filter experiments by project ID. Required if `filter[dataset_id]` is not provided. | | filter[dataset_id] | string | Filter experiments by dataset ID. | | filter[id] | string | Filter experiments by experiment ID. Can be specified multiple times. | | page[cursor] | string | Use the Pagination cursor to retrieve the next page of results. | | page[limit] | integer | Maximum number of results to return per page. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of LLM Observability experiments. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------------ | | | data [*required*] | [object] | List of experiments. | | data | attributes [*required*] | object | Attributes of an LLM Observability experiment. | | attributes | config [*required*] | object | Configuration parameters for the experiment. | | attributes | created_at [*required*] | date-time | Timestamp when the experiment was created. | | attributes | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | attributes | description [*required*] | string | Description of the experiment. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the experiment. | | attributes | name [*required*] | string | Name of the experiment. | | attributes | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | attributes | updated_at [*required*] | date-time | Timestamp when the experiment was last updated. | | data | id [*required*] | string | Unique identifier of the experiment. | | data | type [*required*] | enum | Resource type of an LLM Observability experiment. Allowed enum values: `experiments` | | | meta | object | Pagination cursor metadata. | | meta | after | string | Cursor for the next page of results. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "config": {}, "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "description": "", "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "updated_at": "2024-01-15T10:30:00Z" }, "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "type": "experiments" } ], "meta": { "after": "string" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Create an LLM Observability experiment{% #create-an-llm-observability-experiment %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/experiments | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/experiments | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/experiments | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments | ### Overview Create a new LLM Observability experiment. ### Request #### Body Data (required) Create experiment payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | ------- | ------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for creating an LLM Observability experiment. | | data | attributes [*required*] | object | Attributes for creating an LLM Observability experiment. | | attributes | config | object | Configuration parameters for the experiment. | | attributes | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | attributes | dataset_version | int64 | Version of the dataset to use. Defaults to the current version if not specified. | | attributes | description | string | Description of the experiment. | | attributes | ensure_unique | boolean | Whether to ensure the experiment name is unique. Defaults to `true`. | | attributes | metadata | object | Arbitrary metadata associated with the experiment. | | attributes | name [*required*] | string | Name of the experiment. | | attributes | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | data | type [*required*] | enum | Resource type of an LLM Observability experiment. Allowed enum values: `experiments` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": {}, "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "dataset_version": "integer", "description": "string", "ensure_unique": false, "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" }, "type": "experiments" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a single LLM Observability experiment. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability experiment. | | data | attributes [*required*] | object | Attributes of an LLM Observability experiment. | | attributes | config [*required*] | object | Configuration parameters for the experiment. | | attributes | created_at [*required*] | date-time | Timestamp when the experiment was created. | | attributes | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | attributes | description [*required*] | string | Description of the experiment. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the experiment. | | attributes | name [*required*] | string | Name of the experiment. | | attributes | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | attributes | updated_at [*required*] | date-time | Timestamp when the experiment was last updated. | | data | id [*required*] | string | Unique identifier of the experiment. | | data | type [*required*] | enum | Resource type of an LLM Observability experiment. Allowed enum values: `experiments` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": {}, "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "description": "", "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "updated_at": "2024-01-15T10:30:00Z" }, "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "type": "experiments" } } ``` {% /tab %} {% /tab %} {% tab title="201" %} Created {% tab title="Model" %} Response containing a single LLM Observability experiment. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability experiment. | | data | attributes [*required*] | object | Attributes of an LLM Observability experiment. | | attributes | config [*required*] | object | Configuration parameters for the experiment. | | attributes | created_at [*required*] | date-time | Timestamp when the experiment was created. | | attributes | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | attributes | description [*required*] | string | Description of the experiment. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the experiment. | | attributes | name [*required*] | string | Name of the experiment. | | attributes | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | attributes | updated_at [*required*] | date-time | Timestamp when the experiment was last updated. | | data | id [*required*] | string | Unique identifier of the experiment. | | data | type [*required*] | enum | Resource type of an LLM Observability experiment. Allowed enum values: `experiments` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": {}, "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "description": "", "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "updated_at": "2024-01-15T10:30:00Z" }, "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "type": "experiments" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" }, "type": "experiments" } } EOF {% /tab %} ## Update an LLM Observability experiment{% #update-an-llm-observability-experiment %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------- | | ap1.datadoghq.com | PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id} | | ap2.datadoghq.com | PATCH https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id} | | app.datadoghq.eu | PATCH https://api.datadoghq.eu/api/v2/llm-obs/v1/experiments/{experiment_id} | | app.ddog-gov.com | PATCH https://api.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id} | | us2.ddog-gov.com | PATCH https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id} | | app.datadoghq.com | PATCH https://api.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id} | | us3.datadoghq.com | PATCH https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id} | | us5.datadoghq.com | PATCH https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id} | ### Overview Partially update an existing LLM Observability experiment. ### Arguments #### Path Parameters | Name | Type | Description | | ------------------------------- | ------ | ------------------------------------------- | | experiment_id [*required*] | string | The ID of the LLM Observability experiment. | ### Request #### Body Data (required) Update experiment payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | ------ | ------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for updating an LLM Observability experiment. | | data | attributes [*required*] | object | Attributes for updating an LLM Observability experiment. | | attributes | description | string | Updated description of the experiment. | | attributes | name | string | Updated name of the experiment. | | data | type [*required*] | enum | Resource type of an LLM Observability experiment. Allowed enum values: `experiments` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "description": "string", "name": "string" }, "type": "experiments" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a single LLM Observability experiment. | Parent field | Field | Type | Description | | ------------ | ----------------------------- | --------- | ------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability experiment. | | data | attributes [*required*] | object | Attributes of an LLM Observability experiment. | | attributes | config [*required*] | object | Configuration parameters for the experiment. | | attributes | created_at [*required*] | date-time | Timestamp when the experiment was created. | | attributes | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | attributes | description [*required*] | string | Description of the experiment. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the experiment. | | attributes | name [*required*] | string | Name of the experiment. | | attributes | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | attributes | updated_at [*required*] | date-time | Timestamp when the experiment was last updated. | | data | id [*required*] | string | Unique identifier of the experiment. | | data | type [*required*] | enum | Resource type of an LLM Observability experiment. Allowed enum values: `experiments` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": {}, "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "description": "", "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "updated_at": "2024-01-15T10:30:00Z" }, "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "type": "experiments" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"\# Curl commandcurl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/${experiment_id}" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "type": "experiments" } } EOF {% /tab %} ## Delete LLM Observability experiments{% #delete-llm-observability-experiments %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ----------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments/delete | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments/delete | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/experiments/delete | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/experiments/delete | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments/delete | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/experiments/delete | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments/delete | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/delete | ### Overview Delete one or more LLM Observability experiments. ### Request #### Body Data (required) Delete experiments payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | -------------------------------- | -------- | ------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for deleting LLM Observability experiments. | | data | attributes [*required*] | object | Attributes for deleting LLM Observability experiments. | | attributes | experiment_ids [*required*] | [string] | List of experiment IDs to delete. | | data | type [*required*] | enum | Resource type of an LLM Observability experiment. Allowed enum values: `experiments` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "experiment_ids": [ "3fd6b5e0-8910-4b1c-a7d0-5b84de329012" ] }, "type": "experiments" } } ``` {% /tab %} ### Response {% tab title="204" %} No Content {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/delete" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "experiment_ids": [ "3fd6b5e0-8910-4b1c-a7d0-5b84de329012" ] }, "type": "experiments" } } EOF {% /tab %} ## Push events for an LLM Observability experiment{% #push-events-for-an-llm-observability-experiment %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/experiments/{experiment_id}/events | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events | ### Overview Push spans and metrics for an LLM Observability experiment. ### Arguments #### Path Parameters | Name | Type | Description | | ------------------------------- | ------ | ------------------------------------------- | | experiment_id [*required*] | string | The ID of the LLM Observability experiment. | ### Request #### Body Data (required) Experiment events payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ------------------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for pushing experiment events. | | data | attributes [*required*] | object | Attributes for pushing experiment events including spans and metrics. | | attributes | metrics | [object] | List of metrics to push for the experiment. | | metrics | assessment | enum | Assessment result for an LLM Observability experiment metric. Allowed enum values: `pass,fail` | | metrics | boolean_value | boolean | Boolean value. Used when `metric_type` is `boolean`. | | metrics | categorical_value | string | Categorical value. Used when `metric_type` is `categorical`. | | metrics | error | object | Error details for an experiment metric evaluation. | | error | message | string | Error message associated with the metric evaluation. | | metrics | json_value | object | JSON value. Used when `metric_type` is `json`. | | metrics | label [*required*] | string | Label or name for the metric. | | metrics | metadata | object | Arbitrary metadata associated with the metric. | | metrics | metric_type [*required*] | enum | Type of metric recorded for an LLM Observability experiment. Allowed enum values: `score,categorical,boolean,json` | | metrics | reasoning | string | Human-readable reasoning for the metric value. | | metrics | score_value | double | Numeric score value. Used when `metric_type` is `score`. | | metrics | span_id [*required*] | string | The ID of the span this metric measures. | | metrics | tags | [string] | List of tags associated with the metric. | | metrics | timestamp_ms [*required*] | int64 | Timestamp when the metric was recorded, in milliseconds since Unix epoch. | | attributes | spans | [object] | List of spans to push for the experiment. | | spans | dataset_id [*required*] | string | Dataset ID associated with this span. | | spans | duration [*required*] | int64 | Duration of the span in nanoseconds. | | spans | meta | object | Metadata associated with an experiment span. | | meta | error | object | Error details for an experiment span. | | error | message | string | Error message. | | error | stack | string | Stack trace of the error. | | error | type | string | The error type or exception class name. | | meta | expected_output | object | Expected output for the span, used for evaluation. | | meta | input | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | meta | output | object | Represents any valid JSON value. | | output | Option 1 | string | A scalar string value. | | output | Option 2 | double | A scalar numeric value. | | output | Option 3 | object | An arbitrary object value with additional properties. | | output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | output | Option 5 | boolean | A scalar boolean value. | | spans | name [*required*] | string | Name of the span. | | spans | project_id [*required*] | string | Project ID associated with this span. | | spans | span_id [*required*] | string | Unique identifier of the span. | | spans | start_ns [*required*] | int64 | Start time of the span in nanoseconds since Unix epoch. | | spans | status [*required*] | enum | Status of the span. Allowed enum values: `ok,error` | | spans | tags | [string] | List of tags associated with the span. | | spans | trace_id [*required*] | string | Trace ID for the span. | | data | type [*required*] | enum | Resource type for LLM Observability experiment events. Allowed enum values: `events` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "metrics": [ { "assessment": "pass", "boolean_value": false, "categorical_value": "string", "error": { "message": "string" }, "json_value": {}, "label": "faithfulness", "metadata": {}, "metric_type": "score", "reasoning": "string", "score_value": "number", "span_id": "span-7a1b2c3d", "tags": [], "timestamp_ms": 1705314600000 } ], "spans": [ { "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "duration": 1500000000, "meta": { "error": { "message": "Model response timed out", "stack": "Traceback (most recent call last):\n File \"main.py\", line 10, in \n response = model.generate(input)\n File \"model.py\", line 45, in generate\n raise TimeoutError(\"Model response timed out\")\nTimeoutError: Model response timed out", "type": "TimeoutError" }, "expected_output": {}, "input": { "description": "undefined", "type": "undefined" }, "output": { "description": "undefined", "type": "undefined" } }, "name": "llm_call", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "span_id": "span-7a1b2c3d", "start_ns": 1705314600000000000, "status": "ok", "tags": [], "trace_id": "abc123def456" } ] }, "type": "events" } } ``` {% /tab %} ### Response {% tab title="202" %} Accepted {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "metrics": [ { "assessment": "pass", "label": "faithfulness", "metric_type": "score", "span_id": "span-7a1b2c3d", "timestamp_ms": 1705314600000 } ], "spans": [ { "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "duration": 1500000000, "name": "llm_call", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "span_id": "span-7a1b2c3d", "start_ns": 1705314600000000000, "status": "ok", "trace_id": "abc123def456" } ] }, "type": "events" } } EOF {% /tab %} ## List LLM Observability annotation queues{% #list-llm-observability-annotation-queues %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | ### Overview List annotation queues. Optionally filter by project ID or queue IDs. These parameters are mutually exclusive. If neither is provided, all queues in the organization are returned. ### Arguments #### Query Strings | Name | Type | Description | | --------- | ------ | -------------------------------------------------------------------------------------------------- | | projectId | string | Filter annotation queues by project ID. Cannot be used together with `queueIds`. | | queueIds | array | Filter annotation queues by queue IDs (comma-separated). Cannot be used together with `projectId`. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of LLM Observability annotation queues. | Parent field | Field | Type | Description | | ----------------- | ------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | [object] | List of annotation queues. | | data | attributes [*required*] | object | Attributes of an LLM Observability annotation queue. | | attributes | annotation_schema | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | attributes | created_at [*required*] | date-time | Timestamp when the queue was created. | | attributes | created_by [*required*] | string | Identifier of the user who created the queue. | | attributes | description [*required*] | string | Description of the annotation queue. | | attributes | modified_at [*required*] | date-time | Timestamp when the queue was last modified. | | attributes | modified_by [*required*] | string | Identifier of the user who last modified the queue. | | attributes | name [*required*] | string | Name of the annotation queue. | | attributes | owned_by [*required*] | string | Identifier of the user who owns the queue. | | attributes | project_id [*required*] | string | Identifier of the project this queue belongs to. | | data | id [*required*] | string | Unique identifier of the annotation queue. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] }, "created_at": "2024-01-15T10:30:00Z", "created_by": "00000000-0000-0000-0000-000000000002", "description": "Queue for annotating customer support traces", "modified_at": "2024-01-15T10:30:00Z", "modified_by": "00000000-0000-0000-0000-000000000002", "name": "My annotation queue", "owned_by": "00000000-0000-0000-0000-000000000002", "project_id": "00000000-0000-0000-0000-000000000002" }, "id": "00000000-0000-0000-0000-000000000001", "type": "queues" } ] } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Create an LLM Observability annotation queue{% #create-an-llm-observability-annotation-queue %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues | ### Overview Create an annotation queue. The `name` and `project_id` fields are required. An optional `annotation_schema` can be provided to define the labels for the queue. Fields such as `created_by`, `owned_by`, `created_at`, `modified_by`, and `modified_at` are inferred by the backend. ### Request #### Body Data (required) Create annotation queue payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ----------------- | ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for creating an LLM Observability annotation queue. | | data | attributes [*required*] | object | Attributes for creating an LLM Observability annotation queue. | | attributes | annotation_schema | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | attributes | description | string | Description of the annotation queue. | | attributes | name [*required*] | string | Name of the annotation queue. | | attributes | project_id [*required*] | string | Identifier of the project this queue belongs to. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] }, "description": "Queue for annotating customer support traces", "name": "My annotation queue", "project_id": "00000000-0000-0000-0000-000000000002" }, "type": "queues" } } ``` {% /tab %} ### Response {% tab title="201" %} Created {% tab title="Model" %} Response containing a single LLM Observability annotation queue. | Parent field | Field | Type | Description | | ----------------- | ------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an LLM Observability annotation queue. | | data | attributes [*required*] | object | Attributes of an LLM Observability annotation queue. | | attributes | annotation_schema | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | attributes | created_at [*required*] | date-time | Timestamp when the queue was created. | | attributes | created_by [*required*] | string | Identifier of the user who created the queue. | | attributes | description [*required*] | string | Description of the annotation queue. | | attributes | modified_at [*required*] | date-time | Timestamp when the queue was last modified. | | attributes | modified_by [*required*] | string | Identifier of the user who last modified the queue. | | attributes | name [*required*] | string | Name of the annotation queue. | | attributes | owned_by [*required*] | string | Identifier of the user who owns the queue. | | attributes | project_id [*required*] | string | Identifier of the project this queue belongs to. | | data | id [*required*] | string | Unique identifier of the annotation queue. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] }, "created_at": "2024-01-15T10:30:00Z", "created_by": "00000000-0000-0000-0000-000000000002", "description": "Queue for annotating customer support traces", "modified_at": "2024-01-15T10:30:00Z", "modified_by": "00000000-0000-0000-0000-000000000002", "name": "My annotation queue", "owned_by": "00000000-0000-0000-0000-000000000002", "project_id": "00000000-0000-0000-0000-000000000002" }, "id": "00000000-0000-0000-0000-000000000001", "type": "queues" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "description": "Queue for annotating customer support traces", "name": "My annotation queue", "project_id": "00000000-0000-0000-0000-000000000002" }, "type": "queues" } } EOF\## Create queue with annotation schema # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score" }, { "name": "sentiment", "type": "categorical", "values": [ "positive", "negative", "neutral" ] } ] }, "description": "Queue for annotating customer support traces", "name": "My annotation queue", "project_id": "00000000-0000-0000-0000-000000000002" }, "type": "queues" } } EOF {% /tab %} ## Add annotation queue interactions{% #add-annotation-queue-interactions %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions | ### Overview Add one or more interactions to an annotation queue. At least one interaction must be provided. Each interaction has a `type`: - `trace`, `experiment_trace`, `session`: `content_id` references the upstream entity; the server fetches the actual content. - `display_block`: omit `content_id` and provide the rendered content in `display_block`. The server generates `content_id` as a deterministic hash of the block list. Items of different types can be mixed in a single request. ### Arguments #### Path Parameters | Name | Type | Description | | -------------------------- | ------ | ------------------------------------------------- | | queue_id [*required*] | string | The ID of the LLM Observability annotation queue. | ### Request #### Body Data (required) Add interactions payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------- | ------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for adding interactions to an annotation queue. | | data | attributes [*required*] | object | Attributes for adding interactions to an annotation queue. | | attributes | interactions [*required*] | [ ] | List of interactions to add to the queue. Must contain at least one item. | | interactions | Option 1 | object | An interaction that references an upstream trace, experiment trace, or session. | | Option 1 | content_id [*required*] | string | Upstream entity identifier (trace, experiment trace, or session ID). | | Option 1 | type [*required*] | enum | Type of an upstream-entity interaction. Allowed enum values: `trace,experiment_trace,session` | | interactions | Option 2 | object | An interaction whose rendered content is supplied directly as a list of display blocks. The server generates `content_id` deterministically from the block list. | | Option 2 | display_block [*required*] | [object] | List of content blocks that make up a `display_block` interaction. Must contain at least one block. | | display_block | alt | string | Alternative text for an `image` block. | | display_block | content | | Block payload. A string for `markdown`, `header`, and `text`; an arbitrary JSON value (object, array, or scalar) for `json`. Omitted for `image`, `widget`, and `llmobs_trace`. | | display_block | height | int64 | Optional rendered height. Must be positive when set. | | display_block | interactionType | enum | Upstream interaction type referenced by an `llmobs_trace` block. Restricted to `trace` or `experiment_trace`. Allowed enum values: `trace,experiment_trace` | | display_block | label | string | Optional label rendered alongside the block. | | display_block | level | enum | Visual size for a `header` block. Allowed enum values: `sm,md,lg,xl` | | display_block | tileDef | | Tile definition for a `widget` block. Required for `widget`. The schema is owned by the frontend renderer. | | display_block | timeFrame | object | Unix-millis time range used by chart blocks. | | timeFrame | end [*required*] | int64 | End of the range, in Unix milliseconds. | | timeFrame | start [*required*] | int64 | Start of the range, in Unix milliseconds. | | display_block | traceId | string | Trace identifier. Required for `llmobs_trace` blocks. | | display_block | type [*required*] | enum | Discriminator for a single `display_block` content block. Adding a variant requires coordinated changes in the frontend renderer. Allowed enum values: `markdown,header,text,json,image,widget,llmobs_trace` | | display_block | url | string | URL of the image. Required for `image` blocks. | | Option 2 | type [*required*] | enum | Type discriminator for a `display_block` interaction. Allowed enum values: `display_block` | | data | type [*required*] | enum | Resource type for annotation queue interactions. Allowed enum values: `interactions` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "interactions": [ { "content_id": "trace-abc-123", "type": "trace" } ] }, "type": "interactions" } } ``` {% /tab %} ### Response {% tab title="201" %} Created {% tab title="Model" %} Response containing the result of adding interactions to an annotation queue. | Parent field | Field | Type | Description | | ------------- | --------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for the interaction addition response. | | data | attributes [*required*] | object | Attributes of the interaction addition response. | | attributes | interactions [*required*] | [ ] | List of interactions that were processed. | | interactions | Option 1 | object | A trace, experiment trace, or session interaction result. | | Option 1 | already_existed [*required*] | boolean | Whether this interaction already existed in the queue. | | Option 1 | content_id [*required*] | string | Upstream entity identifier supplied by the caller. | | Option 1 | created_at [*required*] | date-time | Timestamp when the interaction was added to the queue. | | Option 1 | id [*required*] | string | Unique identifier of the interaction. | | Option 1 | modified_at [*required*] | date-time | Timestamp when the interaction was last updated. | | Option 1 | type [*required*] | enum | Type of an upstream-entity interaction. Allowed enum values: `trace,experiment_trace,session` | | interactions | Option 2 | object | A display_block interaction result. | | Option 2 | already_existed [*required*] | boolean | Whether this interaction already existed in the queue. | | Option 2 | content_id [*required*] | string | Server-generated deterministic identifier derived from the block list. | | Option 2 | display_block [*required*] | [object] | List of content blocks that make up a `display_block` interaction. Must contain at least one block. | | display_block | alt | string | Alternative text for an `image` block. | | display_block | content | | Block payload. A string for `markdown`, `header`, and `text`; an arbitrary JSON value (object, array, or scalar) for `json`. Omitted for `image`, `widget`, and `llmobs_trace`. | | display_block | height | int64 | Optional rendered height. Must be positive when set. | | display_block | interactionType | enum | Upstream interaction type referenced by an `llmobs_trace` block. Restricted to `trace` or `experiment_trace`. Allowed enum values: `trace,experiment_trace` | | display_block | label | string | Optional label rendered alongside the block. | | display_block | level | enum | Visual size for a `header` block. Allowed enum values: `sm,md,lg,xl` | | display_block | tileDef | | Tile definition for a `widget` block. Required for `widget`. The schema is owned by the frontend renderer. | | display_block | timeFrame | object | Unix-millis time range used by chart blocks. | | timeFrame | end [*required*] | int64 | End of the range, in Unix milliseconds. | | timeFrame | start [*required*] | int64 | Start of the range, in Unix milliseconds. | | display_block | traceId | string | Trace identifier. Required for `llmobs_trace` blocks. | | display_block | type [*required*] | enum | Discriminator for a single `display_block` content block. Adding a variant requires coordinated changes in the frontend renderer. Allowed enum values: `markdown,header,text,json,image,widget,llmobs_trace` | | display_block | url | string | URL of the image. Required for `image` blocks. | | Option 2 | id [*required*] | string | Unique identifier of the interaction. | | Option 2 | type [*required*] | enum | Type discriminator for a `display_block` interaction. Allowed enum values: `display_block` | | data | id [*required*] | string | The queue ID the interactions were added to. | | data | type [*required*] | enum | Resource type for annotation queue interactions. Allowed enum values: `interactions` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "interactions": [ { "already_existed": false, "content_id": "trace-abc-123", "created_at": "2024-01-15T10:30:00Z", "id": "00000000-0000-0000-0000-000000000000", "modified_at": "2024-01-15T10:30:00Z", "type": "trace" } ] }, "id": "00000000-0000-0000-0000-000000000001", "type": "interactions" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/interactions" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "interactions": [ { "content_id": "trace-abc-123", "type": "trace" } ] }, "type": "interactions" } } EOF\## Add a display_block interaction # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/interactions" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "interactions": [ { "display_block": [ { "content": "## Triage Instructions", "type": "markdown" }, { "content": "Inputs", "level": "md", "type": "header" }, { "content": { "experiment_id": "abc-123" }, "label": "Experiments", "type": "json" } ], "type": "display_block" } ] }, "type": "interactions" } } EOF {% /tab %} ## Get annotated queue interactions{% #get-annotated-queue-interactions %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotated-interactions | ### Overview Retrieve all interactions (traces and sessions) and their annotations for a given annotation queue. ### Arguments #### Path Parameters | Name | Type | Description | | -------------------------- | ------ | ------------------------------------------------- | | queue_id [*required*] | string | The ID of the LLM Observability annotation queue. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing the annotated interactions for an annotation queue. | Parent field | Field | Type | Description | | ---------------------- | ---------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for annotated interactions. | | data | attributes [*required*] | object | Attributes containing the list of annotated interactions. | | attributes | annotated_interactions [*required*] | [ ] | List of interactions with their annotations. | | annotated_interactions | Option 1 | object | A trace, experiment trace, or session interaction with its associated annotations. | | Option 1 | annotations [*required*] | [object] | List of annotations for this interaction. | | annotations | created_at [*required*] | date-time | Timestamp when the annotation was created. | | annotations | created_by [*required*] | string | Identifier of the user who created the annotation. | | annotations | id [*required*] | string | Unique identifier of the annotation. | | annotations | interaction_id [*required*] | string | Identifier of the interaction this annotation belongs to. | | annotations | label_values [*required*] | object | The label values for this annotation. | | annotations | modified_at [*required*] | date-time | Timestamp when the annotation was last modified. | | annotations | modified_by [*required*] | string | Identifier of the user who last modified the annotation. | | Option 1 | content_id [*required*] | string | Upstream entity identifier supplied by the caller. | | Option 1 | created_at [*required*] | date-time | Timestamp when the interaction was added to the queue. | | Option 1 | id [*required*] | string | Unique identifier of the interaction. | | Option 1 | modified_at [*required*] | date-time | Timestamp when the interaction was last updated. | | Option 1 | type [*required*] | enum | Type of an upstream-entity interaction. Allowed enum values: `trace,experiment_trace,session` | | annotated_interactions | Option 2 | object | A display_block interaction with its associated annotations. | | Option 2 | annotations [*required*] | [object] | List of annotations for this interaction. | | annotations | created_at [*required*] | date-time | Timestamp when the annotation was created. | | annotations | created_by [*required*] | string | Identifier of the user who created the annotation. | | annotations | id [*required*] | string | Unique identifier of the annotation. | | annotations | interaction_id [*required*] | string | Identifier of the interaction this annotation belongs to. | | annotations | label_values [*required*] | object | The label values for this annotation. | | annotations | modified_at [*required*] | date-time | Timestamp when the annotation was last modified. | | annotations | modified_by [*required*] | string | Identifier of the user who last modified the annotation. | | Option 2 | content_id [*required*] | string | Server-generated deterministic identifier derived from the block list. | | Option 2 | display_block [*required*] | [object] | List of content blocks that make up a `display_block` interaction. Must contain at least one block. | | display_block | alt | string | Alternative text for an `image` block. | | display_block | content | | Block payload. A string for `markdown`, `header`, and `text`; an arbitrary JSON value (object, array, or scalar) for `json`. Omitted for `image`, `widget`, and `llmobs_trace`. | | display_block | height | int64 | Optional rendered height. Must be positive when set. | | display_block | interactionType | enum | Upstream interaction type referenced by an `llmobs_trace` block. Restricted to `trace` or `experiment_trace`. Allowed enum values: `trace,experiment_trace` | | display_block | label | string | Optional label rendered alongside the block. | | display_block | level | enum | Visual size for a `header` block. Allowed enum values: `sm,md,lg,xl` | | display_block | tileDef | | Tile definition for a `widget` block. Required for `widget`. The schema is owned by the frontend renderer. | | display_block | timeFrame | object | Unix-millis time range used by chart blocks. | | timeFrame | end [*required*] | int64 | End of the range, in Unix milliseconds. | | timeFrame | start [*required*] | int64 | Start of the range, in Unix milliseconds. | | display_block | traceId | string | Trace identifier. Required for `llmobs_trace` blocks. | | display_block | type [*required*] | enum | Discriminator for a single `display_block` content block. Adding a variant requires coordinated changes in the frontend renderer. Allowed enum values: `markdown,header,text,json,image,widget,llmobs_trace` | | display_block | url | string | URL of the image. Required for `image` blocks. | | Option 2 | id [*required*] | string | Unique identifier of the interaction. | | Option 2 | type [*required*] | enum | Type discriminator for a `display_block` interaction. Allowed enum values: `display_block` | | data | id [*required*] | string | The queue ID. | | data | type [*required*] | enum | Resource type for annotated interactions. Allowed enum values: `annotated_interactions` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotated_interactions": [ { "annotations": [ { "created_at": "2024-01-15T10:30:00Z", "created_by": "00000000-0000-0000-0000-000000000002", "id": "annotation-789", "interaction_id": "interaction-456", "label_values": { "quality": "good" }, "modified_at": "2024-01-15T10:30:00Z", "modified_by": "00000000-0000-0000-0000-000000000002" } ], "content_id": "trace-abc-123", "created_at": "2024-01-15T10:30:00Z", "id": "interaction-456", "modified_at": "2024-01-15T10:30:00Z", "type": "trace" } ] }, "id": "00000000-0000-0000-0000-000000000001", "type": "annotated_interactions" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/annotated-interactions" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Update an LLM Observability annotation queue{% #update-an-llm-observability-annotation-queue %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------------------- | | ap1.datadoghq.com | PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | ap2.datadoghq.com | PATCH https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | app.datadoghq.eu | PATCH https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | app.ddog-gov.com | PATCH https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | us2.ddog-gov.com | PATCH https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | app.datadoghq.com | PATCH https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | us3.datadoghq.com | PATCH https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | us5.datadoghq.com | PATCH https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | ### Overview Partially update an annotation queue. The `name`, `description`, and `annotation_schema` fields can be updated. ### Arguments #### Path Parameters | Name | Type | Description | | -------------------------- | ------ | ------------------------------------------------- | | queue_id [*required*] | string | The ID of the LLM Observability annotation queue. | ### Request #### Body Data (required) Update annotation queue payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ----------------- | ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for updating an LLM Observability annotation queue. | | data | attributes [*required*] | object | Attributes for updating an LLM Observability annotation queue. All fields are optional. | | attributes | annotation_schema | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | attributes | description | string | Updated description of the annotation queue. | | attributes | name | string | Updated name of the annotation queue. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] }, "description": "Updated description", "name": "Updated queue name" }, "type": "queues" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a single LLM Observability annotation queue. | Parent field | Field | Type | Description | | ----------------- | ------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an LLM Observability annotation queue. | | data | attributes [*required*] | object | Attributes of an LLM Observability annotation queue. | | attributes | annotation_schema | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | attributes | created_at [*required*] | date-time | Timestamp when the queue was created. | | attributes | created_by [*required*] | string | Identifier of the user who created the queue. | | attributes | description [*required*] | string | Description of the annotation queue. | | attributes | modified_at [*required*] | date-time | Timestamp when the queue was last modified. | | attributes | modified_by [*required*] | string | Identifier of the user who last modified the queue. | | attributes | name [*required*] | string | Name of the annotation queue. | | attributes | owned_by [*required*] | string | Identifier of the user who owns the queue. | | attributes | project_id [*required*] | string | Identifier of the project this queue belongs to. | | data | id [*required*] | string | Unique identifier of the annotation queue. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] }, "created_at": "2024-01-15T10:30:00Z", "created_by": "00000000-0000-0000-0000-000000000002", "description": "Queue for annotating customer support traces", "modified_at": "2024-01-15T10:30:00Z", "modified_by": "00000000-0000-0000-0000-000000000002", "name": "My annotation queue", "owned_by": "00000000-0000-0000-0000-000000000002", "project_id": "00000000-0000-0000-0000-000000000002" }, "id": "00000000-0000-0000-0000-000000000001", "type": "queues" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "description": "Updated description", "name": "Updated queue name" }, "type": "queues" } } EOF {% /tab %} ## Delete an LLM Observability annotation queue{% #delete-an-llm-observability-annotation-queue %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ----------------------------------------------------------------------------------- | | ap1.datadoghq.com | DELETE https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | ap2.datadoghq.com | DELETE https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | app.datadoghq.eu | DELETE https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | app.ddog-gov.com | DELETE https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | us2.ddog-gov.com | DELETE https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | app.datadoghq.com | DELETE https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | us3.datadoghq.com | DELETE https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | | us5.datadoghq.com | DELETE https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id} | ### Overview Delete an annotation queue by its ID. ### Arguments #### Path Parameters | Name | Type | Description | | -------------------------- | ------ | ------------------------------------------------- | | queue_id [*required*] | string | The ID of the LLM Observability annotation queue. | ### Response {% tab title="204" %} No Content {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X DELETE "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Delete annotation queue interactions{% #delete-annotation-queue-interactions %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ----------------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/interactions/delete | ### Overview Delete one or more interactions from an annotation queue. ### Arguments #### Path Parameters | Name | Type | Description | | -------------------------- | ------ | ------------------------------------------------- | | queue_id [*required*] | string | The ID of the LLM Observability annotation queue. | ### Request #### Body Data (required) Delete interactions payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | --------------------------------- | -------- | ------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for deleting interactions from an annotation queue. | | data | attributes [*required*] | object | Attributes for deleting interactions from an annotation queue. | | attributes | interaction_ids [*required*] | [string] | List of interaction IDs to delete. Must contain at least one item. | | data | type [*required*] | enum | Resource type for annotation queue interactions. Allowed enum values: `interactions` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "interaction_ids": [ "00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001" ] }, "type": "interactions" } } ``` {% /tab %} ### Response {% tab title="204" %} No Content {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/interactions/delete" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "interaction_ids": [ "00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001" ] }, "type": "interactions" } } EOF {% /tab %} ## Get a custom evaluator configuration{% #get-a-custom-evaluator-configuration %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.datadoghq.com | GET https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | ### Overview Retrieve a custom LLM Observability evaluator configuration by its name. ### Arguments #### Path Parameters | Name | Type | Description | | --------------------------- | ------ | ----------------------------------------------------------------- | | eval_name [*required*] | string | The name of the custom LLM Observability evaluator configuration. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a custom LLM Observability evaluator configuration. | Parent field | Field | Type | Description | | ------------------- | -------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for a custom LLM Observability evaluator configuration. | | data | attributes [*required*] | object | Attributes of a custom LLM Observability evaluator configuration. | | attributes | category | string | Category of the evaluator. | | attributes | created_at [*required*] | date-time | Timestamp when the evaluator configuration was created. | | attributes | created_by | object | A Datadog user associated with a custom evaluator configuration. | | created_by | email | string | Email address of the user. | | attributes | eval_name [*required*] | string | Name of the custom evaluator. | | attributes | last_updated_by | object | A Datadog user associated with a custom evaluator configuration. | | last_updated_by | email | string | Email address of the user. | | attributes | llm_judge_config | object | LLM judge configuration for a custom evaluator. | | llm_judge_config | assessment_criteria | object | Criteria used to assess the pass/fail result of a custom evaluator. | | assessment_criteria | max_threshold | double | Maximum numeric threshold for a passing result. | | assessment_criteria | min_threshold | double | Minimum numeric threshold for a passing result. | | assessment_criteria | pass_values | [string] | Specific output values considered as a passing result. | | assessment_criteria | pass_when | boolean | When true, a boolean output of true is treated as passing. | | llm_judge_config | inference_params [*required*] | object | LLM inference parameters for a custom evaluator. | | inference_params | frequency_penalty | double | Frequency penalty to reduce repetition. | | inference_params | max_tokens | int64 | Maximum number of tokens to generate. | | inference_params | presence_penalty | double | Presence penalty to reduce repetition. | | inference_params | temperature | double | Sampling temperature for the LLM. | | inference_params | top_k | int64 | Top-k sampling parameter. | | inference_params | top_p | double | Top-p (nucleus) sampling parameter. | | llm_judge_config | last_used_library_prompt_template_name | string | Name of the last library prompt template used. | | llm_judge_config | modified_library_prompt_template | boolean | Whether the library prompt template was modified. | | llm_judge_config | output_schema | object | JSON schema describing the expected output format of the LLM judge. | | llm_judge_config | parsing_type | enum | Output parsing type for a custom LLM judge evaluator. Allowed enum values: `structured_output,json` | | llm_judge_config | prompt_template | [object] | List of messages forming the LLM judge prompt template. | | prompt_template | content | string | Text content of the message. | | prompt_template | contents | [object] | Multi-part content blocks for the message. | | contents | type [*required*] | string | Content block type. | | contents | value [*required*] | object | Value of a prompt message content block. | | value | text | string | Text content of the message block. | | value | tool_call | object | A tool call within a prompt message. | | tool_call | arguments | string | JSON-encoded arguments for the tool call. | | tool_call | id | string | Unique identifier of the tool call. | | tool_call | name | string | Name of the tool being called. | | tool_call | type | string | Type of the tool call. | | value | tool_call_result | object | A tool call result within a prompt message. | | tool_call_result | name | string | Name of the tool that produced this result. | | tool_call_result | result | string | The result returned by the tool. | | tool_call_result | tool_id | string | Identifier of the tool call this result corresponds to. | | tool_call_result | type | string | Type of the tool result. | | prompt_template | role [*required*] | string | Role of the message author. | | attributes | llm_provider | object | LLM provider configuration for a custom evaluator. | | llm_provider | bedrock | object | AWS Bedrock-specific options for LLM provider configuration. | | bedrock | region | string | AWS region for Bedrock. | | llm_provider | integration_account_id | string | Integration account identifier. | | llm_provider | integration_provider | enum | Name of the LLM integration provider. Allowed enum values: `openai,amazon-bedrock,anthropic,azure-openai,vertex-ai,llm-proxy` | | llm_provider | model_name | string | Name of the LLM model. | | llm_provider | vertex_ai | object | Google Vertex AI-specific options for LLM provider configuration. | | vertex_ai | location | string | Google Cloud region. | | vertex_ai | project | string | Google Cloud project ID. | | attributes | target | object | Target application configuration for a custom evaluator. | | target | application_name [*required*] | string | Name of the ML application this evaluator targets. | | target | enabled [*required*] | boolean | Whether the evaluator is active for the target application. | | target | eval_scope | enum | Scope at which to evaluate spans. Allowed enum values: `span,trace,session` | | target | filter | string | Filter expression to select which spans to evaluate. | | target | root_spans_only | boolean | When true, only root spans are evaluated. | | target | sampling_percentage | double | Percentage of traces to evaluate. Must be greater than 0 and at most 100. | | attributes | updated_at [*required*] | date-time | Timestamp when the evaluator configuration was last updated. | | data | id [*required*] | string | Unique name identifier of the evaluator configuration. | | data | type [*required*] | enum | Type of the custom LLM Observability evaluator configuration resource. Allowed enum values: `evaluator_config` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "category": "Custom", "created_at": "2024-01-15T10:30:00Z", "created_by": { "email": "user@example.com" }, "eval_name": "my-custom-evaluator", "last_updated_by": { "email": "user@example.com" }, "llm_judge_config": { "assessment_criteria": { "max_threshold": 1, "min_threshold": 0.7, "pass_values": [ "pass", "yes" ], "pass_when": true }, "inference_params": { "frequency_penalty": 0, "max_tokens": 1024, "presence_penalty": 0, "temperature": 0.7, "top_k": 50, "top_p": 1 }, "last_used_library_prompt_template_name": "sentiment-analysis-v1", "modified_library_prompt_template": false, "output_schema": {}, "parsing_type": "structured_output", "prompt_template": [ { "content": "Rate the quality of the following response:", "contents": [ { "type": "text", "value": { "text": "What is the sentiment of this review?", "tool_call": { "arguments": "{\"location\": \"San Francisco\"}", "id": "call_abc123", "name": "get_weather", "type": "function" }, "tool_call_result": { "name": "get_weather", "result": "sunny, 72F", "tool_id": "call_abc123", "type": "function" } } } ], "role": "user" } ] }, "llm_provider": { "bedrock": { "region": "us-east-1" }, "integration_account_id": "my-account-id", "integration_provider": "openai", "model_name": "gpt-4o", "vertex_ai": { "location": "us-central1", "project": "my-gcp-project" } }, "target": { "application_name": "my-llm-app", "enabled": true, "eval_scope": "span", "filter": "@service:my-service", "root_spans_only": true, "sampling_percentage": 50 }, "updated_at": "2024-01-15T10:30:00Z" }, "id": "my-custom-evaluator", "type": "evaluator_config" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport eval_name="my-custom-evaluator"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Create or update a custom evaluator configuration{% #create-or-update-a-custom-evaluator-configuration %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | PUT https://api.ap1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | ap2.datadoghq.com | PUT https://api.ap2.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.datadoghq.eu | PUT https://api.datadoghq.eu/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.ddog-gov.com | PUT https://api.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us2.ddog-gov.com | PUT https://api.us2.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.datadoghq.com | PUT https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us3.datadoghq.com | PUT https://api.us3.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us5.datadoghq.com | PUT https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | ### Overview Create or update a custom LLM Observability evaluator configuration by its name. ### Arguments #### Path Parameters | Name | Type | Description | | --------------------------- | ------ | ----------------------------------------------------------------- | | eval_name [*required*] | string | The name of the custom LLM Observability evaluator configuration. | ### Request #### Body Data (required) Custom evaluator configuration payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------------- | -------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for creating or updating a custom LLM Observability evaluator configuration. | | data | attributes [*required*] | object | Attributes for creating or updating a custom LLM Observability evaluator configuration. | | attributes | category | string | Category of the evaluator. | | attributes | eval_name | string | Name of the custom evaluator. If provided, must match the eval_name path parameter. | | attributes | llm_judge_config | object | LLM judge configuration for a custom evaluator. | | llm_judge_config | assessment_criteria | object | Criteria used to assess the pass/fail result of a custom evaluator. | | assessment_criteria | max_threshold | double | Maximum numeric threshold for a passing result. | | assessment_criteria | min_threshold | double | Minimum numeric threshold for a passing result. | | assessment_criteria | pass_values | [string] | Specific output values considered as a passing result. | | assessment_criteria | pass_when | boolean | When true, a boolean output of true is treated as passing. | | llm_judge_config | inference_params [*required*] | object | LLM inference parameters for a custom evaluator. | | inference_params | frequency_penalty | double | Frequency penalty to reduce repetition. | | inference_params | max_tokens | int64 | Maximum number of tokens to generate. | | inference_params | presence_penalty | double | Presence penalty to reduce repetition. | | inference_params | temperature | double | Sampling temperature for the LLM. | | inference_params | top_k | int64 | Top-k sampling parameter. | | inference_params | top_p | double | Top-p (nucleus) sampling parameter. | | llm_judge_config | last_used_library_prompt_template_name | string | Name of the last library prompt template used. | | llm_judge_config | modified_library_prompt_template | boolean | Whether the library prompt template was modified. | | llm_judge_config | output_schema | object | JSON schema describing the expected output format of the LLM judge. | | llm_judge_config | parsing_type | enum | Output parsing type for a custom LLM judge evaluator. Allowed enum values: `structured_output,json` | | llm_judge_config | prompt_template | [object] | List of messages forming the LLM judge prompt template. | | prompt_template | content | string | Text content of the message. | | prompt_template | contents | [object] | Multi-part content blocks for the message. | | contents | type [*required*] | string | Content block type. | | contents | value [*required*] | object | Value of a prompt message content block. | | value | text | string | Text content of the message block. | | value | tool_call | object | A tool call within a prompt message. | | tool_call | arguments | string | JSON-encoded arguments for the tool call. | | tool_call | id | string | Unique identifier of the tool call. | | tool_call | name | string | Name of the tool being called. | | tool_call | type | string | Type of the tool call. | | value | tool_call_result | object | A tool call result within a prompt message. | | tool_call_result | name | string | Name of the tool that produced this result. | | tool_call_result | result | string | The result returned by the tool. | | tool_call_result | tool_id | string | Identifier of the tool call this result corresponds to. | | tool_call_result | type | string | Type of the tool result. | | prompt_template | role [*required*] | string | Role of the message author. | | attributes | llm_provider | object | LLM provider configuration for a custom evaluator. | | llm_provider | bedrock | object | AWS Bedrock-specific options for LLM provider configuration. | | bedrock | region | string | AWS region for Bedrock. | | llm_provider | integration_account_id | string | Integration account identifier. | | llm_provider | integration_provider | enum | Name of the LLM integration provider. Allowed enum values: `openai,amazon-bedrock,anthropic,azure-openai,vertex-ai,llm-proxy` | | llm_provider | model_name | string | Name of the LLM model. | | llm_provider | vertex_ai | object | Google Vertex AI-specific options for LLM provider configuration. | | vertex_ai | location | string | Google Cloud region. | | vertex_ai | project | string | Google Cloud project ID. | | attributes | target [*required*] | object | Target application configuration for a custom evaluator. | | target | application_name [*required*] | string | Name of the ML application this evaluator targets. | | target | enabled [*required*] | boolean | Whether the evaluator is active for the target application. | | target | eval_scope | enum | Scope at which to evaluate spans. Allowed enum values: `span,trace,session` | | target | filter | string | Filter expression to select which spans to evaluate. | | target | root_spans_only | boolean | When true, only root spans are evaluated. | | target | sampling_percentage | double | Percentage of traces to evaluate. Must be greater than 0 and at most 100. | | data | id | string | Name of the evaluator. If provided, must match the eval_name path parameter. | | data | type [*required*] | enum | Type of the custom LLM Observability evaluator configuration resource. Allowed enum values: `evaluator_config` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "category": "Custom", "eval_name": "my-custom-evaluator", "llm_judge_config": { "assessment_criteria": { "max_threshold": 1, "min_threshold": 0.7, "pass_values": [ "pass", "yes" ], "pass_when": true }, "inference_params": { "frequency_penalty": 0, "max_tokens": 1024, "presence_penalty": 0, "temperature": 0.7, "top_k": 50, "top_p": 1 }, "last_used_library_prompt_template_name": "sentiment-analysis-v1", "modified_library_prompt_template": false, "output_schema": {}, "parsing_type": "structured_output", "prompt_template": [ { "content": "Rate the quality of the following response:", "contents": [ { "type": "text", "value": { "text": "What is the sentiment of this review?", "tool_call": { "arguments": "{\"location\": \"San Francisco\"}", "id": "call_abc123", "name": "get_weather", "type": "function" }, "tool_call_result": { "name": "get_weather", "result": "sunny, 72F", "tool_id": "call_abc123", "type": "function" } } } ], "role": "user" } ] }, "llm_provider": { "bedrock": { "region": "us-east-1" }, "integration_account_id": "my-account-id", "integration_provider": "openai", "model_name": "gpt-4o", "vertex_ai": { "location": "us-central1", "project": "my-gcp-project" } }, "target": { "application_name": "my-llm-app", "enabled": true, "eval_scope": "span", "filter": "@service:my-service", "root_spans_only": true, "sampling_percentage": 50 } }, "id": "my-custom-evaluator", "type": "evaluator_config" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="422" %} Unprocessable Entity {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport eval_name="my-custom-evaluator"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "llm_judge_config": { "inference_params": { "max_tokens": 1024, "temperature": 0.7 }, "parsing_type": "structured_output" }, "llm_provider": { "integration_provider": "openai", "model_name": "gpt-4o" }, "target": { "application_name": "my-llm-app", "enabled": true, "sampling_percentage": 50 } }, "id": "my-custom-evaluator", "type": "evaluator_config" } } EOF\## Full example with prompt template, output schema, and assessment criteria # \# Path parametersexport eval_name="my-custom-evaluator"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "category": "Custom", "eval_name": "my-custom-evaluator", "llm_judge_config": { "assessment_criteria": { "pass_when": false }, "inference_params": { "frequency_penalty": 0, "max_tokens": 4096, "presence_penalty": 0, "temperature": 1, "top_p": 1 }, "output_schema": { "name": "boolean_eval", "strict": true }, "parsing_type": "structured_output", "prompt_template": [ { "content": "You are a judge LLM.", "role": "system" }, { "content": "{{span_output}}", "role": "user" } ] }, "llm_provider": { "integration_account_id": "your-account-uuid", "integration_provider": "openai", "model_name": "gpt-4o" }, "target": { "application_name": "my-llm-app", "enabled": true, "eval_scope": "span", "filter": "@meta.span.kind:llm", "root_spans_only": false, "sampling_percentage": 100 } }, "id": "my-custom-evaluator", "type": "evaluator_config" } } EOF {% /tab %} ## Delete a custom evaluator configuration{% #delete-a-custom-evaluator-configuration %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | DELETE https://api.ap1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | ap2.datadoghq.com | DELETE https://api.ap2.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.datadoghq.eu | DELETE https://api.datadoghq.eu/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.ddog-gov.com | DELETE https://api.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us2.ddog-gov.com | DELETE https://api.us2.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | app.datadoghq.com | DELETE https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us3.datadoghq.com | DELETE https://api.us3.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | | us5.datadoghq.com | DELETE https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name} | ### Overview Delete a custom LLM Observability evaluator configuration by its name. ### Arguments #### Path Parameters | Name | Type | Description | | --------------------------- | ------ | ----------------------------------------------------------------- | | eval_name [*required*] | string | The name of the custom LLM Observability evaluator configuration. | ### Response {% tab title="204" %} No Content {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport eval_name="my-custom-evaluator"\# Curl commandcurl -X DELETE "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Get annotation queue label schema{% #get-annotation-queue-label-schema %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | ### Overview Retrieve the label schema for a given annotation queue. ### Arguments #### Path Parameters | Name | Type | Description | | -------------------------- | ------ | ------------------------------------------------- | | queue_id [*required*] | string | The ID of the LLM Observability annotation queue. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing the label schema of an annotation queue. | Parent field | Field | Type | Description | | ----------------- | ----------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an annotation queue label schema. | | data | attributes [*required*] | object | Attributes of an annotation queue label schema. | | attributes | annotation_schema [*required*] | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | data | id [*required*] | string | Unique identifier of the annotation queue. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] } }, "id": "00000000-0000-0000-0000-000000000001", "type": "queues" } } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/label-schema" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Update annotation queue label schema{% #update-annotation-queue-label-schema %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | PUT https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | ap2.datadoghq.com | PUT https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | app.datadoghq.eu | PUT https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | app.ddog-gov.com | PUT https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | us2.ddog-gov.com | PUT https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | app.datadoghq.com | PUT https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | us3.datadoghq.com | PUT https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | | us5.datadoghq.com | PUT https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/label-schema | ### Overview Create or replace the label schema for a given annotation queue. The label schema defines the labels annotators can apply to interactions in the queue. Label names must be unique within the queue and match the pattern `^[a-zA-Z0-9_-]+$`. Each label must have a valid type: score, categorical, boolean, or text. ### Arguments #### Path Parameters | Name | Type | Description | | -------------------------- | ------ | ------------------------------------------------- | | queue_id [*required*] | string | The ID of the LLM Observability annotation queue. | ### Request #### Body Data (required) Update label schema payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ----------------- | ----------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for updating an annotation queue label schema. | | data | attributes [*required*] | object | Attributes for updating an annotation queue label schema. | | attributes | annotation_schema [*required*] | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] } }, "type": "queues" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing the label schema of an annotation queue. | Parent field | Field | Type | Description | | ----------------- | ----------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an annotation queue label schema. | | data | attributes [*required*] | object | Attributes of an annotation queue label schema. | | attributes | annotation_schema [*required*] | object | Schema defining the labels for an annotation queue. | | annotation_schema | label_schemas [*required*] | [object] | List of label schema definitions. | | label_schemas | description | string | Description of the label. | | label_schemas | has_assessment | boolean | Whether this label includes an assessment field. | | label_schemas | has_reasoning | boolean | Whether this label includes a reasoning field. | | label_schemas | id | string | Unique identifier of the label schema. Assigned by the server if not provided. | | label_schemas | is_assessment | boolean | Whether the boolean label represents an assessment. Requires `has_assessment` to be true. | | label_schemas | is_integer | boolean | Whether score values must be integers. Applicable to score-type labels. | | label_schemas | is_required | boolean | Whether this label is required for an annotation. | | label_schemas | max | double | Maximum value for score-type labels. | | label_schemas | min | double | Minimum value for score-type labels. | | label_schemas | name [*required*] | string | Name of the label. Must match the pattern `^[a-zA-Z0-9_-]+$` and be unique within the queue. | | label_schemas | type [*required*] | enum | Type of a label in an annotation queue label schema. Allowed enum values: `score,categorical,boolean,text` | | label_schemas | values | [string] | Allowed values for categorical-type labels. Must contain at least one non-empty, unique value. | | data | id [*required*] | string | Unique identifier of the annotation queue. | | data | type [*required*] | enum | Resource type of an LLM Observability annotation queue. Allowed enum values: `queues` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "description": "Rating of the response quality.", "has_assessment": false, "has_reasoning": false, "id": "abc-123", "is_assessment": false, "is_integer": false, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score", "values": [ "good", "bad", "neutral" ] } ] } }, "id": "00000000-0000-0000-0000-000000000001", "type": "queues" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## Boolean label schema # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/label-schema" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "has_assessment": true, "is_assessment": true, "is_required": true, "name": "is_correct", "type": "boolean" } ] } }, "type": "queues" } } EOF\## Categorical label schema # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/label-schema" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "is_required": true, "name": "sentiment", "type": "categorical", "values": [ "positive", "negative", "neutral" ] } ] } }, "type": "queues" } } EOF\## default # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/label-schema" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score" } ] } }, "type": "queues" } } EOF\## Score label schema # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/label-schema" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "has_reasoning": true, "is_required": true, "max": 5, "min": 0, "name": "quality", "type": "score" } ] } }, "type": "queues" } } EOF\## Text label schema # \# Path parametersexport queue_id="00000000-0000-0000-0000-000000000001"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/label-schema" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotation_schema": { "label_schemas": [ { "is_required": false, "name": "feedback", "type": "text" } ] } }, "type": "queues" } } EOF {% /tab %} ## List events for an LLM Observability experiment{% #list-events-for-an-llm-observability-experiment %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | -------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v3/experiments/{experiment_id}/events | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v3/experiments/{experiment_id}/events | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v3/experiments/{experiment_id}/events | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v3/experiments/{experiment_id}/events | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v3/experiments/{experiment_id}/events | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v3/experiments/{experiment_id}/events | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v3/experiments/{experiment_id}/events | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v3/experiments/{experiment_id}/events | ### Overview Retrieve spans and experiment-level summary metrics for a given experiment with cursor-based pagination. ### Arguments #### Path Parameters | Name | Type | Description | | ------------------------------- | ------ | ------------------------------------------- | | experiment_id [*required*] | string | The ID of the LLM Observability experiment. | #### Query Strings | Name | Type | Description | | ------------ | ------- | ------------------------------------------------------------------------- | | page[limit] | integer | Maximum number of spans to return per page. Defaults to 5000. | | page[cursor] | string | Opaque cursor from a previous response to fetch the next page of results. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response for listing experiment events (v2/v3). Returns spans and summary metrics in a single resource. | Parent field | Field | Type | Description | | -------------------- | --------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ | | | data [*required*] | object | JSON:API data object for an experiment events response. | | data | attributes [*required*] | object | Attributes of an experiment events response. | | attributes | spans [*required*] | [object] | Experiment spans, each enriched with their associated evaluation metrics. | | spans | dataset_record_id | string | ID of the dataset record this span evaluated. | | spans | duration | double | Duration of the span in nanoseconds. | | spans | eval_metrics | [object] | Evaluation metrics associated with this span. | | eval_metrics | assessment | enum | Assessment result for an LLM Observability experiment metric. Allowed enum values: `pass,fail` | | eval_metrics | boolean_value | boolean | Boolean value. Present when `metric_type` is `boolean`. | | eval_metrics | categorical_value | string | Categorical value. Present when `metric_type` is `categorical`. | | eval_metrics | eval_source_type | string | Source type of the evaluation. | | eval_metrics | id | string | Unique identifier of the evaluation metric event. | | eval_metrics | json_value | object | JSON value. Present when `metric_type` is `json`. | | eval_metrics | label | string | Label or name for the metric. | | eval_metrics | metadata | object | Arbitrary metadata associated with the metric. | | eval_metrics | metric_source | string | Source of the metric. Either `custom` (user-submitted) or `summary` (experiment-level aggregate). | | eval_metrics | metric_type | enum | Type of metric recorded for an LLM Observability experiment. Allowed enum values: `score,categorical,boolean,json` | | eval_metrics | reasoning | string | Human-readable reasoning for the metric value. | | eval_metrics | score_value | double | Numeric score. Present when `metric_type` is `score`. | | eval_metrics | span_id | string | Span ID this metric is associated with. | | eval_metrics | tags | [string] | Tags associated with the metric. | | eval_metrics | timestamp_ms | int64 | Timestamp when the metric was recorded, in milliseconds since Unix epoch. | | eval_metrics | trace_id | string | Trace ID linking this metric to a span. | | spans | id | string | Unique identifier of the span. | | spans | meta | object | Metadata associated with an experiment span. | | meta | error | object | Error details for an experiment span. | | error | message | string | Error message. | | error | stack | string | Stack trace of the error. | | error | type | string | The error type or exception class name. | | meta | expected_output | object | Expected output for the span, used for evaluation. | | meta | input | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | meta | output | object | Represents any valid JSON value. | | output | Option 1 | string | A scalar string value. | | output | Option 2 | double | A scalar numeric value. | | output | Option 3 | object | An arbitrary object value with additional properties. | | output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | output | Option 5 | boolean | A scalar boolean value. | | spans | metrics | object | Numeric metrics attached to the span. | | additionalProperties | | double | | spans | name | string | Name of the span. | | spans | parent_id | string | Parent span ID, if any. | | spans | span_id | string | Span ID. | | spans | start_ns | int64 | Start time in nanoseconds since Unix epoch. | | spans | status | enum | Status of the span. Allowed enum values: `ok,error` | | spans | tags | [string] | Tags associated with the span. | | spans | trace_id | string | Trace ID. | | attributes | summary_metrics [*required*] | [object] | Experiment-level summary evaluation metrics (not tied to individual spans). | | summary_metrics | assessment | enum | Assessment result for an LLM Observability experiment metric. Allowed enum values: `pass,fail` | | summary_metrics | boolean_value | boolean | Boolean value. Present when `metric_type` is `boolean`. | | summary_metrics | categorical_value | string | Categorical value. Present when `metric_type` is `categorical`. | | summary_metrics | eval_source_type | string | Source type of the evaluation. | | summary_metrics | id | string | Unique identifier of the evaluation metric event. | | summary_metrics | json_value | object | JSON value. Present when `metric_type` is `json`. | | summary_metrics | label | string | Label or name for the metric. | | summary_metrics | metadata | object | Arbitrary metadata associated with the metric. | | summary_metrics | metric_source | string | Source of the metric. Either `custom` (user-submitted) or `summary` (experiment-level aggregate). | | summary_metrics | metric_type | enum | Type of metric recorded for an LLM Observability experiment. Allowed enum values: `score,categorical,boolean,json` | | summary_metrics | reasoning | string | Human-readable reasoning for the metric value. | | summary_metrics | score_value | double | Numeric score. Present when `metric_type` is `score`. | | summary_metrics | span_id | string | Span ID this metric is associated with. | | summary_metrics | tags | [string] | Tags associated with the metric. | | summary_metrics | timestamp_ms | int64 | Timestamp when the metric was recorded, in milliseconds since Unix epoch. | | summary_metrics | trace_id | string | Trace ID linking this metric to a span. | | data | id [*required*] | string | Identifier for this events resource. | | data | type [*required*] | enum | Resource type for an experiment events collection. Allowed enum values: `experiment_events` | | | meta | object | Pagination cursor metadata. | | meta | after | string | Cursor for the next page of results. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "spans": [ { "dataset_record_id": "string", "duration": 1500000000, "eval_metrics": [ { "assessment": "pass", "boolean_value": false, "categorical_value": "string", "eval_source_type": "managed", "id": "00000000-0000-0000-0000-000000000001", "json_value": {}, "label": "faithfulness", "metadata": {}, "metric_source": "custom", "metric_type": "score", "reasoning": "string", "score_value": "number", "span_id": "span-7a1b2c3d", "tags": [], "timestamp_ms": 1705314600000, "trace_id": "abc123def456" } ], "id": "00000000-0000-0000-0000-000000000001", "meta": { "error": { "message": "Model response timed out", "stack": "Traceback (most recent call last):\n File \"main.py\", line 10, in \n response = model.generate(input)\n File \"model.py\", line 45, in generate\n raise TimeoutError(\"Model response timed out\")\nTimeoutError: Model response timed out", "type": "TimeoutError" }, "expected_output": {}, "input": { "description": "undefined", "type": "undefined" }, "output": { "description": "undefined", "type": "undefined" } }, "metrics": { "": "number" }, "name": "llm_call", "parent_id": "string", "span_id": "span-7a1b2c3d", "start_ns": 1705314600000000000, "status": "ok", "tags": [], "trace_id": "abc123def456" } ], "summary_metrics": [ { "assessment": "pass", "boolean_value": false, "categorical_value": "string", "eval_source_type": "managed", "id": "00000000-0000-0000-0000-000000000001", "json_value": {}, "label": "faithfulness", "metadata": {}, "metric_source": "custom", "metric_type": "score", "reasoning": "string", "score_value": "number", "span_id": "span-7a1b2c3d", "tags": [], "timestamp_ms": 1705314600000, "trace_id": "abc123def456" } ] }, "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "type": "experiment_events" }, "meta": { "after": "string" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v3/experiments/${experiment_id}/events" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Search LLM Observability experimentation entities{% #search-llm-observability-experimentation-entities %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experimentation/search | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experimentation/search | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/experimentation/search | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/experimentation/search | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experimentation/search | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/experimentation/search | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experimentation/search | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experimentation/search | ### Overview Search across LLM Observability experimentation entities — projects, datasets, dataset records, experiments, and experiment runs — using cursor-based pagination. The `filter.scope` field controls which entity types are returned. At least one valid scope must be provided. Returns `200 OK` when all results fit in a single page. Returns `206 Partial Content` with a cursor in `meta.after` when additional pages are available. ### Request #### Body Data (required) Experimentation search payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | --------------- | ---------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an experimentation search request. | | data | attributes [*required*] | object | Attributes for an experimentation search request. | | attributes | content_preview | object | Options to control content preview truncation. | | content_preview | limit | int64 | Maximum number of characters to include in content previews. | | attributes | filter [*required*] | object | Filter criteria for an experimentation search request. | | filter | include_deleted | boolean | When `true`, include soft-deleted entities alongside active ones. | | filter | is_deleted | boolean | When `true`, return only soft-deleted entities. | | filter | query | string | Free-text search query. | | filter | scope [*required*] | [string] | Entity types to search. Valid values are `projects`, `datasets`, `dataset_records`, `experiments`, and `experiment_runs`. | | filter | version | int64 | Filter dataset records by a specific dataset version. | | attributes | include | object | Additional data to include in the response. | | include | user_data | boolean | When `true`, enrich results with author user data (name and email). | | attributes | page | object | Cursor-based pagination parameters. | | page | cursor | string | Opaque cursor returned from a previous response to fetch the next page. | | page | limit | int64 | Maximum number of results per page. | | data | type [*required*] | enum | Resource type for experimentation search and analytics operations. Allowed enum values: `experimentation` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "content_preview": { "limit": 500 }, "filter": { "include_deleted": false, "is_deleted": false, "query": "my experiment", "scope": [ "experiments" ], "version": "integer" }, "include": { "user_data": false }, "page": { "cursor": "string", "limit": 100 } }, "type": "experimentation" } } ``` {% /tab %} ### Response {% tab title="200" %} OK — all results returned in a single page. {% tab title="Model" %} Response to a cursor-based experimentation search. Returns `200 OK` when all results fit in one page; `206 Partial Content` when a next-page cursor is available. | Parent field | Field | Type | Description | | --------------- | --------------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | JSON:API data object for an experimentation search response. | | data | attributes [*required*] | object | The matching experimentation entities grouped by type. | | attributes | dataset_records | [object] | Matching dataset records. Present when `dataset_records` is included in `filter.scope`. | | dataset_records | created_at [*required*] | date-time | Timestamp when the record was created. | | dataset_records | dataset_id [*required*] | string | Identifier of the dataset this record belongs to. | | dataset_records | expected_output [*required*] | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | dataset_records | id [*required*] | string | Unique identifier of the record. | | dataset_records | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | dataset_records | metadata [*required*] | object | Arbitrary metadata associated with the record. | | dataset_records | updated_at [*required*] | date-time | Timestamp when the record was last updated. | | attributes | datasets | [object] | Matching datasets. Present when `datasets` is included in `filter.scope`. | | datasets | attributes [*required*] | object | Attributes of an LLM Observability dataset. | | attributes | created_at [*required*] | date-time | Timestamp when the dataset was created. | | attributes | current_version [*required*] | int64 | Current version number of the dataset. | | attributes | description [*required*] | string | Description of the dataset. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | attributes | updated_at [*required*] | date-time | Timestamp when the dataset was last updated. | | datasets | id [*required*] | string | Unique identifier of the dataset. | | datasets | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | | attributes | experiment_runs | [object] | Matching experiment runs. Present when `experiment_runs` is included in `filter.scope`. | | experiment_runs | aggregate_data | object | Aggregated metric data for this run. | | experiment_runs | created_at | date-time | Timestamp when the run was created. | | experiment_runs | experiment_id | string | Identifier of the experiment this run belongs to. | | experiment_runs | id | string | Unique identifier of the experiment run. | | experiment_runs | run_number | int32 | Sequential number of this run within the experiment. | | attributes | experiments | [object] | Matching experiments. Present when `experiments` is included in `filter.scope`. | | experiments | config [*required*] | object | Configuration parameters for the experiment. | | experiments | created_at [*required*] | date-time | Timestamp when the experiment was created. | | experiments | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | experiments | description [*required*] | string | Description of the experiment. | | experiments | metadata [*required*] | object | Arbitrary metadata associated with the experiment. | | experiments | name [*required*] | string | Name of the experiment. | | experiments | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | experiments | updated_at [*required*] | date-time | Timestamp when the experiment was last updated. | | attributes | projects | [object] | Matching projects. Present when `projects` is included in `filter.scope`. | | projects | attributes [*required*] | object | Attributes of an LLM Observability project. | | attributes | created_at [*required*] | date-time | Timestamp when the project was created. | | attributes | description [*required*] | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | attributes | updated_at [*required*] | date-time | Timestamp when the project was last updated. | | projects | id [*required*] | string | Unique identifier of the project. | | projects | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | | data | id [*required*] | string | Server-generated identifier for this search result. | | data | type [*required*] | enum | Resource type for experimentation search and analytics operations. Allowed enum values: `experimentation` | | | meta | object | Pagination cursor metadata. | | meta | after | string | Cursor for the next page of results. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "dataset_records": [ { "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {}, "updated_at": "2024-01-15T10:30:00Z" } ], "datasets": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "current_version": 1, "description": "", "metadata": {}, "name": "My LLM Dataset", "updated_at": "2024-01-15T10:30:00Z" }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "datasets" } ], "experiment_runs": [ { "aggregate_data": {}, "created_at": "2024-01-15T10:30:00Z", "experiment_id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "id": "7a1b2c3d-4e5f-6789-abcd-ef0123456789", "run_number": 1 } ], "experiments": [ { "config": {}, "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "description": "", "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "updated_at": "2024-01-15T10:30:00Z" } ], "projects": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "description": "", "name": "My LLM Project", "updated_at": "2024-01-15T10:30:00Z" }, "id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "type": "projects" } ] }, "id": "00000000-0000-0000-0000-000000000001", "type": "experimentation" }, "meta": { "after": "string" } } ``` {% /tab %} {% /tab %} {% tab title="206" %} Partial Content — more results are available. Use `meta.after` as the next `page.cursor`. {% tab title="Model" %} Response to a cursor-based experimentation search. Returns `200 OK` when all results fit in one page; `206 Partial Content` when a next-page cursor is available. | Parent field | Field | Type | Description | | --------------- | --------------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | JSON:API data object for an experimentation search response. | | data | attributes [*required*] | object | The matching experimentation entities grouped by type. | | attributes | dataset_records | [object] | Matching dataset records. Present when `dataset_records` is included in `filter.scope`. | | dataset_records | created_at [*required*] | date-time | Timestamp when the record was created. | | dataset_records | dataset_id [*required*] | string | Identifier of the dataset this record belongs to. | | dataset_records | expected_output [*required*] | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | dataset_records | id [*required*] | string | Unique identifier of the record. | | dataset_records | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | dataset_records | metadata [*required*] | object | Arbitrary metadata associated with the record. | | dataset_records | updated_at [*required*] | date-time | Timestamp when the record was last updated. | | attributes | datasets | [object] | Matching datasets. Present when `datasets` is included in `filter.scope`. | | datasets | attributes [*required*] | object | Attributes of an LLM Observability dataset. | | attributes | created_at [*required*] | date-time | Timestamp when the dataset was created. | | attributes | current_version [*required*] | int64 | Current version number of the dataset. | | attributes | description [*required*] | string | Description of the dataset. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | attributes | updated_at [*required*] | date-time | Timestamp when the dataset was last updated. | | datasets | id [*required*] | string | Unique identifier of the dataset. | | datasets | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | | attributes | experiment_runs | [object] | Matching experiment runs. Present when `experiment_runs` is included in `filter.scope`. | | experiment_runs | aggregate_data | object | Aggregated metric data for this run. | | experiment_runs | created_at | date-time | Timestamp when the run was created. | | experiment_runs | experiment_id | string | Identifier of the experiment this run belongs to. | | experiment_runs | id | string | Unique identifier of the experiment run. | | experiment_runs | run_number | int32 | Sequential number of this run within the experiment. | | attributes | experiments | [object] | Matching experiments. Present when `experiments` is included in `filter.scope`. | | experiments | config [*required*] | object | Configuration parameters for the experiment. | | experiments | created_at [*required*] | date-time | Timestamp when the experiment was created. | | experiments | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | experiments | description [*required*] | string | Description of the experiment. | | experiments | metadata [*required*] | object | Arbitrary metadata associated with the experiment. | | experiments | name [*required*] | string | Name of the experiment. | | experiments | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | experiments | updated_at [*required*] | date-time | Timestamp when the experiment was last updated. | | attributes | projects | [object] | Matching projects. Present when `projects` is included in `filter.scope`. | | projects | attributes [*required*] | object | Attributes of an LLM Observability project. | | attributes | created_at [*required*] | date-time | Timestamp when the project was created. | | attributes | description [*required*] | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | attributes | updated_at [*required*] | date-time | Timestamp when the project was last updated. | | projects | id [*required*] | string | Unique identifier of the project. | | projects | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | | data | id [*required*] | string | Server-generated identifier for this search result. | | data | type [*required*] | enum | Resource type for experimentation search and analytics operations. Allowed enum values: `experimentation` | | | meta | object | Pagination cursor metadata. | | meta | after | string | Cursor for the next page of results. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "dataset_records": [ { "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {}, "updated_at": "2024-01-15T10:30:00Z" } ], "datasets": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "current_version": 1, "description": "", "metadata": {}, "name": "My LLM Dataset", "updated_at": "2024-01-15T10:30:00Z" }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "datasets" } ], "experiment_runs": [ { "aggregate_data": {}, "created_at": "2024-01-15T10:30:00Z", "experiment_id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "id": "7a1b2c3d-4e5f-6789-abcd-ef0123456789", "run_number": 1 } ], "experiments": [ { "config": {}, "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "description": "", "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "updated_at": "2024-01-15T10:30:00Z" } ], "projects": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "description": "", "name": "My LLM Project", "updated_at": "2024-01-15T10:30:00Z" }, "id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "type": "projects" } ] }, "id": "00000000-0000-0000-0000-000000000001", "type": "experimentation" }, "meta": { "after": "string" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experimentation/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "filter": { "query": "@project_id:a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "scope": [ "experiments" ] }, "page": { "limit": 50 } }, "type": "experimentation" } } EOF {% /tab %} ## Simple search experimentation entities{% #simple-search-experimentation-entities %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experimentation/simple-search | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experimentation/simple-search | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/experimentation/simple-search | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/experimentation/simple-search | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experimentation/simple-search | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/experimentation/simple-search | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experimentation/simple-search | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experimentation/simple-search | ### Overview Search across LLM Observability experimentation entities using offset-based (page-number) pagination. Use this endpoint when you need total page count or want to navigate to a specific page number. The `filter.scope` field controls which entity types are returned. At least one valid scope must be provided. ### Request #### Body Data (required) Simple search payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | --------------- | ---------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an experimentation simple search request. | | data | attributes [*required*] | object | Attributes for an experimentation simple search request. | | attributes | content_preview | object | Options to control content preview truncation. | | content_preview | limit | int64 | Maximum number of characters to include in content previews. | | attributes | filter [*required*] | object | Filter criteria for an experimentation search request. | | filter | include_deleted | boolean | When `true`, include soft-deleted entities alongside active ones. | | filter | is_deleted | boolean | When `true`, return only soft-deleted entities. | | filter | query | string | Free-text search query. | | filter | scope [*required*] | [string] | Entity types to search. Valid values are `projects`, `datasets`, `dataset_records`, `experiments`, and `experiment_runs`. | | filter | version | int64 | Filter dataset records by a specific dataset version. | | attributes | include | object | Additional data to include in the response. | | include | user_data | boolean | When `true`, enrich results with author user data (name and email). | | attributes | page | object | Offset-based pagination parameters for simple search. | | page | limit | int32 | Maximum number of results per page. | | page | number | int32 | Page number to retrieve (1-indexed). | | attributes | sort | [object] | Sort order for results. | | sort | direction | enum | Sort direction. Allowed enum values: `asc,desc` | | sort | field [*required*] | string | The field name to sort on. | | data | type [*required*] | enum | Resource type for experimentation search and analytics operations. Allowed enum values: `experimentation` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "content_preview": { "limit": 500 }, "filter": { "include_deleted": false, "is_deleted": false, "query": "my experiment", "scope": [ "experiments" ], "version": "integer" }, "include": { "user_data": false }, "page": { "limit": 50, "number": 1 }, "sort": [ { "direction": "desc", "field": "created_at" } ] }, "type": "experimentation" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response to an offset-based experimentation simple search. | Parent field | Field | Type | Description | | --------------- | --------------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | JSON:API data object for a simple search response. | | data | attributes [*required*] | object | The matching experimentation entities grouped by type. | | attributes | dataset_records | [object] | Matching dataset records. Present when `dataset_records` is included in `filter.scope`. | | dataset_records | created_at [*required*] | date-time | Timestamp when the record was created. | | dataset_records | dataset_id [*required*] | string | Identifier of the dataset this record belongs to. | | dataset_records | expected_output [*required*] | object | Represents any valid JSON value. | | expected_output | Option 1 | string | A scalar string value. | | expected_output | Option 2 | double | A scalar numeric value. | | expected_output | Option 3 | object | An arbitrary object value with additional properties. | | expected_output | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | expected_output | Option 5 | boolean | A scalar boolean value. | | dataset_records | id [*required*] | string | Unique identifier of the record. | | dataset_records | input [*required*] | object | Represents any valid JSON value. | | input | Option 1 | string | A scalar string value. | | input | Option 2 | double | A scalar numeric value. | | input | Option 3 | object | An arbitrary object value with additional properties. | | input | Option 4 | [ ] | An array of arbitrary values. | | Option 4 | Option 1 | string | A scalar string value. | | Option 4 | Option 2 | double | A scalar numeric value. | | Option 4 | Option 3 | object | An arbitrary object value with additional properties. | | Option 4 | Option 4 | boolean | A scalar boolean value. | | input | Option 5 | boolean | A scalar boolean value. | | dataset_records | metadata [*required*] | object | Arbitrary metadata associated with the record. | | dataset_records | updated_at [*required*] | date-time | Timestamp when the record was last updated. | | attributes | datasets | [object] | Matching datasets. Present when `datasets` is included in `filter.scope`. | | datasets | attributes [*required*] | object | Attributes of an LLM Observability dataset. | | attributes | created_at [*required*] | date-time | Timestamp when the dataset was created. | | attributes | current_version [*required*] | int64 | Current version number of the dataset. | | attributes | description [*required*] | string | Description of the dataset. | | attributes | metadata [*required*] | object | Arbitrary metadata associated with the dataset. | | attributes | name [*required*] | string | Name of the dataset. | | attributes | updated_at [*required*] | date-time | Timestamp when the dataset was last updated. | | datasets | id [*required*] | string | Unique identifier of the dataset. | | datasets | type [*required*] | enum | Resource type of an LLM Observability dataset. Allowed enum values: `datasets` | | attributes | experiment_runs | [object] | Matching experiment runs. Present when `experiment_runs` is included in `filter.scope`. | | experiment_runs | aggregate_data | object | Aggregated metric data for this run. | | experiment_runs | created_at | date-time | Timestamp when the run was created. | | experiment_runs | experiment_id | string | Identifier of the experiment this run belongs to. | | experiment_runs | id | string | Unique identifier of the experiment run. | | experiment_runs | run_number | int32 | Sequential number of this run within the experiment. | | attributes | experiments | [object] | Matching experiments. Present when `experiments` is included in `filter.scope`. | | experiments | config [*required*] | object | Configuration parameters for the experiment. | | experiments | created_at [*required*] | date-time | Timestamp when the experiment was created. | | experiments | dataset_id [*required*] | string | Identifier of the dataset used in this experiment. | | experiments | description [*required*] | string | Description of the experiment. | | experiments | metadata [*required*] | object | Arbitrary metadata associated with the experiment. | | experiments | name [*required*] | string | Name of the experiment. | | experiments | project_id [*required*] | string | Identifier of the project this experiment belongs to. | | experiments | updated_at [*required*] | date-time | Timestamp when the experiment was last updated. | | attributes | projects | [object] | Matching projects. Present when `projects` is included in `filter.scope`. | | projects | attributes [*required*] | object | Attributes of an LLM Observability project. | | attributes | created_at [*required*] | date-time | Timestamp when the project was created. | | attributes | description [*required*] | string | Description of the project. | | attributes | name [*required*] | string | Name of the project. | | attributes | updated_at [*required*] | date-time | Timestamp when the project was last updated. | | projects | id [*required*] | string | Unique identifier of the project. | | projects | type [*required*] | enum | Resource type of an LLM Observability project. Allowed enum values: `projects` | | data | id [*required*] | string | Server-generated identifier for this search result. | | data | type [*required*] | enum | Resource type for experimentation search and analytics operations. Allowed enum values: `experimentation` | | | meta | object | Pagination metadata for a simple search response. | | meta | page | object | Page metadata. | | page | current | int32 | Current page number. | | page | limit | int32 | Page size used for this response. | | page | total_count | int32 | Total number of matching results (capped at the maximum search limit). | | page | total_pages | int32 | Total number of pages available. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "dataset_records": [ { "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "expected_output": { "description": "undefined", "type": "undefined" }, "id": "rec-7c3f5a1b-9e2d-4f8a-b1c6-3d7e9f0a2b4c", "input": { "description": "undefined", "type": "undefined" }, "metadata": {}, "updated_at": "2024-01-15T10:30:00Z" } ], "datasets": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "current_version": 1, "description": "", "metadata": {}, "name": "My LLM Dataset", "updated_at": "2024-01-15T10:30:00Z" }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "datasets" } ], "experiment_runs": [ { "aggregate_data": {}, "created_at": "2024-01-15T10:30:00Z", "experiment_id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "id": "7a1b2c3d-4e5f-6789-abcd-ef0123456789", "run_number": 1 } ], "experiments": [ { "config": {}, "created_at": "2024-01-15T10:30:00Z", "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "description": "", "metadata": {}, "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "updated_at": "2024-01-15T10:30:00Z" } ], "projects": [ { "attributes": { "created_at": "2024-01-15T10:30:00Z", "description": "", "name": "My LLM Project", "updated_at": "2024-01-15T10:30:00Z" }, "id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "type": "projects" } ] }, "id": "00000000-0000-0000-0000-000000000001", "type": "experimentation" }, "meta": { "page": { "current": 1, "limit": 50, "total_count": 193, "total_pages": 4 } } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experimentation/simple-search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "filter": { "query": "@project_id:a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "scope": [ "experiments" ] }, "page": { "limit": 50, "number": 1 }, "sort": [ { "direction": "desc", "field": "created_at" } ] }, "type": "experimentation" } } EOF {% /tab %} ## Aggregate LLM Observability experimentation{% #aggregate-llm-observability-experimentation %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------ | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/experimentation/analytics | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/experimentation/analytics | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experimentation/analytics | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics | ### Overview Execute an analytics aggregation over LLM Observability experimentation data. Use this endpoint to compute metrics (for example average eval scores) grouped by fields such as `span_id` or `experiment_id`. At least one `compute` definition and one `index` must be provided. ### Request #### Body Data (required) Analytics payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | ------------ | ---------------------------- | -------- | --------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an analytics request. | | data | attributes [*required*] | object | Attributes for an analytics request. | | attributes | aggregate [*required*] | object | Analytics aggregation parameters. | | aggregate | compute [*required*] | [object] | List of metric computations to perform. | | compute | metric [*required*] | string | Name of the metric to compute. | | compute | name | string | Optional alias for this computation in the response. | | aggregate | dataset_version | int64 | Filter to a specific dataset version. | | aggregate | group_by | [object] | Fields to group results by. | | group_by | field [*required*] | string | Field name to group by. | | aggregate | indexes [*required*] | [string] | Data indexes to query. At least one is required. | | aggregate | limit | int32 | Maximum number of results to return. | | aggregate | search [*required*] | object | Search query for filtering analytics data. | | search | query [*required*] | string | Filter expression. | | aggregate | time | object | Unix-millisecond time range for filtering analytics data. | | time | from [*required*] | int64 | Start of the time range in milliseconds since Unix epoch. | | time | to [*required*] | int64 | End of the time range in milliseconds since Unix epoch. | | data | type [*required*] | enum | Resource type for experimentation search and analytics operations. Allowed enum values: `experimentation` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "aggregate": { "compute": [ { "metric": "score_value", "name": "avg_faithfulness" } ], "dataset_version": "integer", "group_by": [ { "field": "span_id" } ], "indexes": [ "experiment-evals" ], "limit": 1000, "search": { "query": "@experiment_id:3fd6b5e0-8910-4b1c-a7d0-5b84de329012" }, "time": { "from": 1705312200000, "to": 1705315800000 } } }, "type": "experimentation" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response to an analytics query. | Parent field | Field | Type | Description | | ------------ | ---------------------------- | -------- | --------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | JSON:API data object for an analytics response. | | data | attributes [*required*] | object | Attributes of an analytics response. | | attributes | hit_count [*required*] | int64 | Total number of events matched by the query before grouping. | | attributes | result [*required*] | object | Analytics query result containing all buckets. | | result | values [*required*] | [object] | List of result buckets. | | values | by | object | The group-by field values for this bucket. | | values | metrics [*required*] | object | Computed metric values for this bucket. | | data | id [*required*] | string | Server-generated identifier for this analytics result. | | data | type [*required*] | enum | Resource type for experimentation search and analytics operations. Allowed enum values: `experimentation` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "hit_count": 1500, "result": { "values": [ { "by": { "span_id": "span-7a1b2c3d" }, "metrics": { "score_value": 0.85 } } ] } }, "id": "00000000-0000-0000-0000-000000000001", "type": "experimentation" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "aggregate": { "compute": [ { "metric": "score_value", "name": "avg_faithfulness" } ], "group_by": [ { "field": "span_id" } ], "indexes": [ "experiment-evals" ], "search": { "query": "@experiment_id:3fd6b5e0-8910-4b1c-a7d0-5b84de329012 @label:faithfulness" } } }, "type": "experimentation" } } EOF {% /tab %} ## List LLM Observability spans{% #list-llm-observability-spans %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/spans/events | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/spans/events | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/spans/events | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/spans/events | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/spans/events | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/spans/events | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/spans/events | ### Overview List LLM Observability spans matching the specified filters. ### Arguments #### Query Strings | Name | Type | Description | | ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | filter[from] | string | Start of the time range. Accepts ISO 8601 or relative format (e.g., `now-15m`). Defaults to `now-15m`. | | filter[to] | string | End of the time range. Accepts ISO 8601 or relative format. Defaults to `now`. | | filter[query] | string | Search query using LLM Observability query syntax. Supports attribute filters using the field:value syntax (e.g. session_id, trace_id, ml_app, meta.span.kind). When provided, structured field filters (`filter[span_id]`, `filter[trace_id]`, etc.) are ignored. | | filter[span_id] | string | Filter by exact span ID. | | filter[trace_id] | string | Filter by exact trace ID. | | filter[span_kind] | string | Filter by span kind (e.g., llm, agent, tool, task, workflow). | | filter[span_name] | string | Filter by span name. | | filter[ml_app] | string | Filter by ML application name. | | page[limit] | integer | Maximum number of spans to return. Defaults to `10`. | | page[cursor] | string | Cursor from the previous response to retrieve the next page. | | sort | string | Sort order for the results. | | include_attachments | boolean | Whether to include attachment data in the response. Defaults to `true`. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of LLM Observability spans. | Parent field | Field | Type | Description | | -------------------- | ---------------------------- | -------- | ----------------------------------------------------------------------------------- | | | data [*required*] | [object] | List of spans matching the query. | | data | attributes [*required*] | object | Attributes of an LLM Observability span. | | attributes | duration [*required*] | double | Duration of the span in nanoseconds. | | attributes | evaluation | object | Evaluation metrics keyed by evaluator name. | | additionalProperties | | object | An evaluation metric associated with an LLM Observability span. | | | assessment | string | Assessment result (e.g., pass or fail). | | | eval_metric_type | string | Type of the evaluation metric (e.g., score, categorical, boolean). | | | reasoning | string | Human-readable reasoning for the evaluation result. | | | status | string | Status of the evaluation execution. | | | tags | [string] | Tags associated with the evaluation metric. | | | value | | Value of the evaluation result. | | attributes | input | object | Input or output content of an LLM Observability span. | | input | messages | [object] | List of messages in the input or output. | | messages | content | string | Text content of the message. | | messages | id | string | Unique identifier of the message. | | messages | role | string | Role of the message sender (e.g., user, assistant, system). | | messages | tool_calls | [object] | Tool calls made in this message. | | tool_calls | arguments | object | Arguments passed to the tool. | | tool_calls | name | string | Name of the tool called. | | tool_calls | tool_id | string | Identifier of the tool call. | | tool_calls | type | string | Type of the tool call. | | messages | tool_results | [object] | Tool results returned in this message. | | tool_results | name | string | Name of the tool that produced this result. | | tool_results | result | string | Result value returned by the tool. | | tool_results | tool_id | string | Identifier of the corresponding tool call. | | tool_results | type | string | Type of the tool result. | | input | value | string | Plain-text value of the input or output. | | attributes | intent | string | Detected intent of the span. | | attributes | metadata | object | Arbitrary metadata associated with the span. | | attributes | metrics | object | Numeric metrics associated with the span (e.g., token counts). | | additionalProperties | | double | | attributes | ml_app [*required*] | string | Name of the ML application this span belongs to. | | attributes | model_name | string | Name of the model used in this span. | | attributes | model_provider | string | Provider of the model used in this span. | | attributes | name [*required*] | string | Name of the span. | | attributes | output | object | Input or output content of an LLM Observability span. | | output | messages | [object] | List of messages in the input or output. | | messages | content | string | Text content of the message. | | messages | id | string | Unique identifier of the message. | | messages | role | string | Role of the message sender (e.g., user, assistant, system). | | messages | tool_calls | [object] | Tool calls made in this message. | | tool_calls | arguments | object | Arguments passed to the tool. | | tool_calls | name | string | Name of the tool called. | | tool_calls | tool_id | string | Identifier of the tool call. | | tool_calls | type | string | Type of the tool call. | | messages | tool_results | [object] | Tool results returned in this message. | | tool_results | name | string | Name of the tool that produced this result. | | tool_results | result | string | Result value returned by the tool. | | tool_results | tool_id | string | Identifier of the corresponding tool call. | | tool_results | type | string | Type of the tool result. | | output | value | string | Plain-text value of the input or output. | | attributes | parent_id | string | Identifier of the parent span, if any. | | attributes | span_id [*required*] | string | Unique identifier of the span. | | attributes | span_kind [*required*] | string | Kind of span (e.g., llm, agent, tool, task, workflow). | | attributes | start_ns [*required*] | int64 | Start time of the span in nanoseconds since Unix epoch. | | attributes | status [*required*] | string | Status of the span (e.g., ok, error). | | attributes | tags | [string] | Tags associated with the span. | | attributes | tool_definitions | [object] | Tool definitions available to the span. | | tool_definitions | description | string | Description of what the tool does. | | tool_definitions | name | string | Name of the tool. | | tool_definitions | schema | object | JSON schema describing the tool's input parameters. | | tool_definitions | version | string | Version of the tool definition. | | attributes | trace_id [*required*] | string | Trace identifier this span belongs to. | | data | id [*required*] | string | Unique identifier of the span. | | data | type [*required*] | enum | Resource type for an LLM Observability span. Allowed enum values: `span` | | | links | object | Pagination links accompanying the spans response. | | links | next | string | URL to retrieve the next page of results. | | | meta [*required*] | object | Metadata accompanying the spans response. | | meta | elapsed [*required*] | int64 | Time elapsed for the query in milliseconds. | | meta | page [*required*] | object | Pagination cursor for the spans response. | | page | after | string | Cursor to retrieve the next page of results. Absent when there are no more results. | | meta | request_id [*required*] | string | Unique identifier for the request. | | meta | status [*required*] | string | Status of the query execution. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "duration": 1500000000, "evaluation": { "": { "assessment": "pass", "eval_metric_type": "score", "reasoning": "string", "status": "string", "tags": [], "value": "undefined" } }, "input": { "messages": [ { "content": "string", "id": "string", "role": "string", "tool_calls": [ { "arguments": {}, "name": "string", "tool_id": "string", "type": "string" } ], "tool_results": [ { "name": "string", "result": "string", "tool_id": "string", "type": "string" } ] } ], "value": "string" }, "intent": "string", "metadata": {}, "metrics": { "": "number" }, "ml_app": "my-llm-app", "model_name": "gpt-4o", "model_provider": "openai", "name": "llm_call", "output": { "messages": [ { "content": "string", "id": "string", "role": "string", "tool_calls": [ { "arguments": {}, "name": "string", "tool_id": "string", "type": "string" } ], "tool_results": [ { "name": "string", "result": "string", "tool_id": "string", "type": "string" } ] } ], "value": "string" }, "parent_id": "string", "span_id": "abc123def456", "span_kind": "llm", "start_ns": 1705314600000000000, "status": "ok", "tags": [], "tool_definitions": [ { "description": "string", "name": "string", "schema": {}, "version": "string" } ], "trace_id": "trace-9a8b7c6d5e4f" }, "id": "abc123def456", "type": "span" } ], "links": { "next": "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events?page[cursor]=eyJzdGFydCI6MTAwfQ==" }, "meta": { "elapsed": 132, "page": { "after": "eyJzdGFydCI6MTAwfQ==" }, "request_id": "req-abc123", "status": "done" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/spans/events" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Search LLM Observability spans{% #search-llm-observability-spans %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------ | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/spans/events/search | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/spans/events/search | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/spans/events/search | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/spans/events/search | ### Overview Search LLM Observability spans using structured filters in the request body. ### Request #### Body Data (required) Search spans payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | -------------------- | ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an LLM Observability spans search request. | | data | attributes [*required*] | object | Attributes of an LLM Observability spans search request. | | attributes | filter | object | Filter criteria for an LLM Observability span search. | | filter | from | string | Start of the time range. Accepts ISO 8601 or relative format (e.g., `now-15m`). Defaults to `now-15m`. | | filter | ml_app | string | Filter by ML application name. | | filter | query | string | Search query using LLM Observability query syntax. Supports attribute filters using the field:value syntax (e.g. session_id, trace_id, ml_app, meta.span.kind). When provided, structured field filters (`span_id`, `trace_id`, etc.) are ignored. | | filter | span_id | string | Filter by exact span ID. | | filter | span_kind | string | Filter by span kind (e.g., llm, agent, tool, task, workflow). | | filter | span_name | string | Filter by span name. | | filter | tags | object | Filter by tag key-value pairs. | | additionalProperties | | string | | filter | to | string | End of the time range. Accepts ISO 8601 or relative format (e.g., `now`). Defaults to `now`. | | filter | trace_id | string | Filter by exact trace ID. | | attributes | options | object | Additional options for a span search request. | | options | include_attachments | boolean | Whether to include attachment data in the response. Defaults to `true`. | | options | time_offset | int64 | Offset in seconds applied to both `from` and `to` timestamps. | | attributes | page | object | Pagination settings for a span search request. | | page | cursor | string | Cursor from the previous response to retrieve the next page. | | page | limit | int64 | Maximum number of spans to return. Defaults to `10`. | | attributes | sort | string | Sort order for the results. Use `-` prefix for descending order. | | data | type [*required*] | enum | Resource type for an LLM Observability spans search request. Allowed enum values: `spans` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "filter": { "from": "now-900s", "ml_app": "my-llm-app", "query": "@session_id:abc123def456", "span_id": "abc123def456", "span_kind": "llm", "span_name": "llm_call", "tags": { "": "string" }, "to": "now", "trace_id": "trace-9a8b7c6d5e4f" }, "options": { "include_attachments": true, "time_offset": 0 }, "page": { "cursor": "eyJzdGFydCI6MTAwfQ==", "limit": 10 }, "sort": "-start_ns" }, "type": "spans" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of LLM Observability spans. | Parent field | Field | Type | Description | | -------------------- | ---------------------------- | -------- | ----------------------------------------------------------------------------------- | | | data [*required*] | [object] | List of spans matching the query. | | data | attributes [*required*] | object | Attributes of an LLM Observability span. | | attributes | duration [*required*] | double | Duration of the span in nanoseconds. | | attributes | evaluation | object | Evaluation metrics keyed by evaluator name. | | additionalProperties | | object | An evaluation metric associated with an LLM Observability span. | | | assessment | string | Assessment result (e.g., pass or fail). | | | eval_metric_type | string | Type of the evaluation metric (e.g., score, categorical, boolean). | | | reasoning | string | Human-readable reasoning for the evaluation result. | | | status | string | Status of the evaluation execution. | | | tags | [string] | Tags associated with the evaluation metric. | | | value | | Value of the evaluation result. | | attributes | input | object | Input or output content of an LLM Observability span. | | input | messages | [object] | List of messages in the input or output. | | messages | content | string | Text content of the message. | | messages | id | string | Unique identifier of the message. | | messages | role | string | Role of the message sender (e.g., user, assistant, system). | | messages | tool_calls | [object] | Tool calls made in this message. | | tool_calls | arguments | object | Arguments passed to the tool. | | tool_calls | name | string | Name of the tool called. | | tool_calls | tool_id | string | Identifier of the tool call. | | tool_calls | type | string | Type of the tool call. | | messages | tool_results | [object] | Tool results returned in this message. | | tool_results | name | string | Name of the tool that produced this result. | | tool_results | result | string | Result value returned by the tool. | | tool_results | tool_id | string | Identifier of the corresponding tool call. | | tool_results | type | string | Type of the tool result. | | input | value | string | Plain-text value of the input or output. | | attributes | intent | string | Detected intent of the span. | | attributes | metadata | object | Arbitrary metadata associated with the span. | | attributes | metrics | object | Numeric metrics associated with the span (e.g., token counts). | | additionalProperties | | double | | attributes | ml_app [*required*] | string | Name of the ML application this span belongs to. | | attributes | model_name | string | Name of the model used in this span. | | attributes | model_provider | string | Provider of the model used in this span. | | attributes | name [*required*] | string | Name of the span. | | attributes | output | object | Input or output content of an LLM Observability span. | | output | messages | [object] | List of messages in the input or output. | | messages | content | string | Text content of the message. | | messages | id | string | Unique identifier of the message. | | messages | role | string | Role of the message sender (e.g., user, assistant, system). | | messages | tool_calls | [object] | Tool calls made in this message. | | tool_calls | arguments | object | Arguments passed to the tool. | | tool_calls | name | string | Name of the tool called. | | tool_calls | tool_id | string | Identifier of the tool call. | | tool_calls | type | string | Type of the tool call. | | messages | tool_results | [object] | Tool results returned in this message. | | tool_results | name | string | Name of the tool that produced this result. | | tool_results | result | string | Result value returned by the tool. | | tool_results | tool_id | string | Identifier of the corresponding tool call. | | tool_results | type | string | Type of the tool result. | | output | value | string | Plain-text value of the input or output. | | attributes | parent_id | string | Identifier of the parent span, if any. | | attributes | span_id [*required*] | string | Unique identifier of the span. | | attributes | span_kind [*required*] | string | Kind of span (e.g., llm, agent, tool, task, workflow). | | attributes | start_ns [*required*] | int64 | Start time of the span in nanoseconds since Unix epoch. | | attributes | status [*required*] | string | Status of the span (e.g., ok, error). | | attributes | tags | [string] | Tags associated with the span. | | attributes | tool_definitions | [object] | Tool definitions available to the span. | | tool_definitions | description | string | Description of what the tool does. | | tool_definitions | name | string | Name of the tool. | | tool_definitions | schema | object | JSON schema describing the tool's input parameters. | | tool_definitions | version | string | Version of the tool definition. | | attributes | trace_id [*required*] | string | Trace identifier this span belongs to. | | data | id [*required*] | string | Unique identifier of the span. | | data | type [*required*] | enum | Resource type for an LLM Observability span. Allowed enum values: `span` | | | links | object | Pagination links accompanying the spans response. | | links | next | string | URL to retrieve the next page of results. | | | meta [*required*] | object | Metadata accompanying the spans response. | | meta | elapsed [*required*] | int64 | Time elapsed for the query in milliseconds. | | meta | page [*required*] | object | Pagination cursor for the spans response. | | page | after | string | Cursor to retrieve the next page of results. Absent when there are no more results. | | meta | request_id [*required*] | string | Unique identifier for the request. | | meta | status [*required*] | string | Status of the query execution. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "duration": 1500000000, "evaluation": { "": { "assessment": "pass", "eval_metric_type": "score", "reasoning": "string", "status": "string", "tags": [], "value": "undefined" } }, "input": { "messages": [ { "content": "string", "id": "string", "role": "string", "tool_calls": [ { "arguments": {}, "name": "string", "tool_id": "string", "type": "string" } ], "tool_results": [ { "name": "string", "result": "string", "tool_id": "string", "type": "string" } ] } ], "value": "string" }, "intent": "string", "metadata": {}, "metrics": { "": "number" }, "ml_app": "my-llm-app", "model_name": "gpt-4o", "model_provider": "openai", "name": "llm_call", "output": { "messages": [ { "content": "string", "id": "string", "role": "string", "tool_calls": [ { "arguments": {}, "name": "string", "tool_id": "string", "type": "string" } ], "tool_results": [ { "name": "string", "result": "string", "tool_id": "string", "type": "string" } ] } ], "value": "string" }, "parent_id": "string", "span_id": "abc123def456", "span_kind": "llm", "start_ns": 1705314600000000000, "status": "ok", "tags": [], "tool_definitions": [ { "description": "string", "name": "string", "schema": {}, "version": "string" } ], "trace_id": "trace-9a8b7c6d5e4f" }, "id": "abc123def456", "type": "span" } ], "links": { "next": "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events?page[cursor]=eyJzdGFydCI6MTAwfQ==" }, "meta": { "elapsed": 132, "page": { "after": "eyJzdGFydCI6MTAwfQ==" }, "request_id": "req-abc123", "status": "done" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## Search spans for an ML app # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/spans/events/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "filter": { "from": "now-900s", "ml_app": "my-llm-app", "span_kind": "llm", "to": "now" }, "options": { "include_attachments": true }, "page": { "limit": 10 } }, "type": "spans" } } EOF\## Search all spans in a session # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/spans/events/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "filter": { "from": "now-900s", "query": "@session_id:abc123def456", "to": "now" }, "options": { "include_attachments": true }, "page": { "limit": 50 } }, "type": "spans" } } EOF {% /tab %} ## Delete LLM Observability data{% #delete-llm-observability-data %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/deletion/data/llmobs | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/deletion/data/llmobs | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/deletion/data/llmobs | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/deletion/data/llmobs | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/deletion/data/llmobs | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/deletion/data/llmobs | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/deletion/data/llmobs | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/deletion/data/llmobs | ### Overview Submit a request to delete LLM Observability span data matching a trace ID filter within a specified time range. ### Request #### Body Data (required) Data deletion request payload. {% tab title="Model" %} | Parent field | Field | Type | Description | | -------------------- | ---------------------------- | ------ | -------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an LLM Observability data deletion request. | | data | attributes [*required*] | object | Attributes for an LLM Observability data deletion request. | | attributes | delay | int64 | Optional delay in seconds before the deletion is executed. | | attributes | from [*required*] | int64 | Start of the deletion time range in milliseconds since Unix epoch. | | attributes | query [*required*] | object | Query filters selecting the data to delete. Must include a `query` key with an `@trace_id` filter. | | additionalProperties | | string | | attributes | to [*required*] | int64 | End of the deletion time range in milliseconds since Unix epoch. | | data | type [*required*] | enum | Resource type for an LLM Observability data deletion request. Allowed enum values: `create_deletion_req` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "delay": 0, "from": 1705314600000, "query": { "": "string" }, "to": 1705315200000 }, "type": "create_deletion_req" } } ``` {% /tab %} ### Response {% tab title="202" %} Accepted {% tab title="Model" %} Response containing details of a submitted LLM Observability data deletion request. | Parent field | Field | Type | Description | | ------------ | ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for an LLM Observability data deletion response. | | data | attributes [*required*] | object | Attributes of a submitted LLM Observability data deletion request. | | attributes | created_at [*required*] | date-time | Timestamp when the deletion request was created. | | attributes | created_by [*required*] | string | UUID of the user who created the deletion request. | | attributes | from_time [*required*] | int64 | Start of the deletion time range in milliseconds since Unix epoch. | | attributes | org_id [*required*] | int64 | ID of the organization that submitted the deletion request. | | attributes | product [*required*] | string | Product name for the deletion request. | | attributes | query [*required*] | string | The query string used to select data for deletion. | | attributes | to_time [*required*] | int64 | End of the deletion time range in milliseconds since Unix epoch. | | data | id [*required*] | string | Unique identifier of the deletion request. | | data | type [*required*] | enum | Resource type for an LLM Observability data deletion response. Allowed enum values: `deletion_request` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "created_at": "2024-01-15T10:30:00Z", "created_by": "user-uuid-1234", "from_time": 1705314600000, "org_id": 12345, "product": "llmobs", "query": "@trace_id:abc123def456", "to_time": 1705315200000 }, "id": "msg-abc123", "type": "deletion_request" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/deletion/data/llmobs" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "from": 1705314600000, "query": { "query": "@trace_id:abc123def456" }, "to": 1705315200000 }, "type": "create_deletion_req" } } EOF {% /tab %} ## List LLM integration accounts{% #list-llm-integration-accounts %} {% tab title="v2" %} **Note**: This endpoint is in Preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/accounts | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/accounts | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/integrations/{integration}/accounts | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/accounts | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/accounts | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/accounts | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/accounts | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/accounts | ### Overview Retrieve the list of configured accounts for the specified LLM provider integration. ### Arguments #### Path Parameters | Name | Type | Description | | ----------------------------- | ------ | -------------------------------- | | integration [*required*] | string | The name of the LLM integration. | ### Response {% tab title="200" %} OK {% tab title="Model" %} | Parent field | Field | Type | Description | | --------------------- | --------------------- | -------- | ------------------------------------------------------------------------------- | | | account_id | string | Provider-specific account identifier. | | | account_name | string | Human-readable name for the integration account. | | | account_region | string | Provider region associated with the account, if applicable. | | | azure_openai_metadata | object | Azure OpenAI-specific metadata for an integration account or inference request. | | azure_openai_metadata | deployment_id | string | The Azure OpenAI deployment ID. | | azure_openai_metadata | model_version | string | The model version deployed in Azure. | | azure_openai_metadata | resource_name | string | The Azure OpenAI resource name. | | | id | string | Unique identifier for the integration account. | | | integration | string | The name of the LLM provider integration. | | | vertex_ai_metadata | object | Vertex AI-specific metadata for an integration account or inference request. | | vertex_ai_metadata | location | string | The Vertex AI region. | | vertex_ai_metadata | project | string | The Google Cloud project ID. | | vertex_ai_metadata | project_ids | [string] | List of Google Cloud project IDs available to the service account. | {% /tab %} {% tab title="Example" %} ```json { "account_id": "org-XYZ123", "account_name": "Production OpenAI", "account_region": "us-east-1", "azure_openai_metadata": { "deployment_id": "my-gpt4-deployment", "model_version": "0613", "resource_name": "my-azure-resource" }, "id": "account-abc123", "integration": "openai", "vertex_ai_metadata": { "location": "us-central1", "project": "my-gcp-project", "project_ids": [ "my-gcp-project" ] } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport integration="openai"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/integrations/${integration}/accounts" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## List LLM integration models{% #list-llm-integration-models %} {% tab title="v2" %} **Note**: This endpoint is in Preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | -------------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/models | ### Overview Retrieve the list of models available for the specified LLM provider integration and account. ### Arguments #### Path Parameters | Name | Type | Description | | ----------------------------- | ------ | ---------------------------------- | | integration [*required*] | string | The name of the LLM integration. | | account_id [*required*] | string | The ID of the integration account. | ### Response {% tab title="200" %} OK {% tab title="Model" %} | Parent field | Field | Type | Description | | -------------------- | ------------------------ | ------- | ------------------------------------------------------------- | | | has_access | boolean | Whether the account has access to this model. | | | id | string | Unique identifier for the model entry. | | | integration | string | The name of the LLM provider integration. | | | integration_display_name | string | Human-readable name of the LLM provider integration. | | | json_schema | boolean | Whether the model supports structured output via JSON schema. | | | model_display_name | string | Human-readable model name. | | | model_id | string | Provider-specific model identifier used in inference calls. | | | provider | string | The underlying model provider. | | | provider_display_name | string | Human-readable name of the underlying model provider. | | | region_prefix_overrides | object | Map of region-specific model ID prefix overrides. | | additionalProperties | | string | {% /tab %} {% tab title="Example" %} ```json { "has_access": true, "id": "gpt-4o", "integration": "openai", "integration_display_name": "OpenAI", "json_schema": true, "model_display_name": "GPT-4o", "model_id": "gpt-4o", "provider": "openai", "provider_display_name": "OpenAI", "region_prefix_overrides": { "": "string" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport integration="openai"export account_id="account-abc123"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/integrations/${integration}/${account_id}/models" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Run an LLM inference{% #run-an-llm-inference %} {% tab title="v2" %} **Note**: This endpoint is in Preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------------------------------ | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | ### Overview Run an LLM inference request through the specified integration and account, returning the model response and token usage. ### Arguments #### Path Parameters | Name | Type | Description | | ----------------------------- | ------ | ---------------------------------- | | integration [*required*] | string | The name of the LLM integration. | | account_id [*required*] | string | The ID of the integration account. | ### Request #### Body Data (required) Inference request parameters. {% tab title="Model" %} | Parent field | Field | Type | Description | | --------------------- | ---------------------------- | -------- | --------------------------------------------------------------------------------------------------------------- | | | anthropic_metadata | object | Anthropic-specific metadata for an inference request. | | anthropic_metadata | effort | enum | The effort level for Anthropic inference. Allowed enum values: `low,medium,high,max` | | anthropic_metadata | thinking | object | Configuration for Anthropic extended thinking feature. | | thinking | budget_tokens | int64 | Maximum token budget for extended thinking. Required when type is `enabled`. | | thinking | type [*required*] | enum | The thinking mode for Anthropic extended thinking. Allowed enum values: `enabled,disabled,adaptive` | | | azure_openai_metadata | object | Azure OpenAI-specific metadata for an integration account or inference request. | | azure_openai_metadata | deployment_id | string | The Azure OpenAI deployment ID. | | azure_openai_metadata | model_version | string | The model version deployed in Azure. | | azure_openai_metadata | resource_name | string | The Azure OpenAI resource name. | | | bedrock_metadata | object | Amazon Bedrock-specific metadata for an inference request. | | bedrock_metadata | region | string | The AWS region for the Bedrock request. | | | frequency_penalty | double | Penalty for token frequency to reduce repetition. | | | json_schema | string | JSON schema for structured output, if supported by the model. | | | max_completion_tokens | int64 | Maximum number of completion tokens to generate (alternative to max_tokens for some providers). | | | max_tokens | int64 | Maximum number of tokens to generate. | | | messages [*required*] | [object] | List of messages in an inference conversation. | | messages | content | string | Plain text content of the message. | | messages | contents | [object] | List of structured content blocks in a message. | | contents | type [*required*] | string | The content block type. | | contents | value [*required*] | object | The typed value of a message content block. | | value | text | string | Plain text content. | | value | tool_call | object | A tool call made during LLM inference. | | tool_call | arguments | object | The arguments passed to the tool. | | tool_call | name | string | The name of the tool being called. | | tool_call | tool_id | string | Unique identifier for the tool call. | | tool_call | type | string | The type of tool call. | | value | tool_call_result | object | The result returned by a tool call during LLM inference. | | tool_call_result | name | string | The name of the tool that produced this result. | | tool_call_result | result | string | The result content returned by the tool. | | tool_call_result | tool_id | string | Identifier matching the corresponding tool call. | | tool_call_result | type | string | The type of tool result. | | messages | id | string | Unique identifier for the message. | | messages | role | string | The role of the message author. | | messages | tool_calls | [object] | List of tool calls in a message. | | tool_calls | arguments | object | The arguments passed to the tool. | | tool_calls | name | string | The name of the tool being called. | | tool_calls | tool_id | string | Unique identifier for the tool call. | | tool_calls | type | string | The type of tool call. | | messages | tool_results | [object] | List of tool results in a message. | | tool_results | name | string | The name of the tool that produced this result. | | tool_results | result | string | The result content returned by the tool. | | tool_results | tool_id | string | Identifier matching the corresponding tool call. | | tool_results | type | string | The type of tool result. | | | model_id [*required*] | string | The model identifier to use for inference. | | | openai_metadata | object | OpenAI-specific metadata for an inference request. | | openai_metadata | reasoning_effort | enum | The reasoning effort level for OpenAI models that support it. Allowed enum values: `none,low,medium,high,xhigh` | | openai_metadata | reasoning_summary | enum | The verbosity of the reasoning summary. Allowed enum values: `auto,concise,detailed` | | | presence_penalty | double | Penalty for token presence to encourage topic diversity. | | | temperature | double | Sampling temperature between 0 and 2. Higher values produce more random output. | | | tools | [object] | List of tools available to the model. | | tools | function [*required*] | object | A function definition for a tool available to the model. | | function | description | string | A description of what the function does. | | function | name [*required*] | string | The name of the function. | | function | parameters [*required*] | object | JSON schema describing the function parameters. | | tools | type [*required*] | string | The type of tool. | | | top_k | int64 | Top-K sampling parameter. | | | top_p | double | Nucleus sampling probability mass. | | | vertex_ai_metadata | object | Vertex AI-specific metadata for an integration account or inference request. | | vertex_ai_metadata | location | string | The Vertex AI region. | | vertex_ai_metadata | project | string | The Google Cloud project ID. | | vertex_ai_metadata | project_ids | [string] | List of Google Cloud project IDs available to the service account. | {% /tab %} {% tab title="Example" %} ```json { "anthropic_metadata": { "effort": "medium", "thinking": { "budget_tokens": 1024, "type": "enabled" } }, "azure_openai_metadata": { "deployment_id": "my-gpt4-deployment", "model_version": "0613", "resource_name": "my-azure-resource" }, "bedrock_metadata": { "region": "us-east-1" }, "frequency_penalty": 0, "json_schema": "{\"type\":\"object\",\"properties\":{\"answer\":{\"type\":\"string\"}}}", "max_completion_tokens": 1024, "max_tokens": 1024, "messages": [ { "content": "What is the capital of France?", "contents": [ { "type": "text", "value": { "text": "Hello, how can I help you?", "tool_call": { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" }, "tool_call_result": { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } } } ], "id": "msg_001", "role": "user", "tool_calls": [ { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" } ], "tool_results": [ { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } ] } ], "model_id": "gpt-4o", "openai_metadata": { "reasoning_effort": "medium", "reasoning_summary": "auto" }, "presence_penalty": 0, "temperature": 0.7, "tools": [ { "function": { "description": "Get the current weather for a location.", "name": "get_weather", "parameters": { "properties": { "location": { "type": "string" } }, "type": "object" } }, "type": "function" } ], "top_k": 50, "top_p": 1, "vertex_ai_metadata": { "location": "us-central1", "project": "my-gcp-project", "project_ids": [ "my-gcp-project" ] } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} The result of an LLM inference request, including input parameters and the model response. | Parent field | Field | Type | Description | | --------------------- | --------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------- | | | anthropic_metadata | object | Anthropic-specific metadata for an inference request. | | anthropic_metadata | effort | enum | The effort level for Anthropic inference. Allowed enum values: `low,medium,high,max` | | anthropic_metadata | thinking | object | Configuration for Anthropic extended thinking feature. | | thinking | budget_tokens | int64 | Maximum token budget for extended thinking. Required when type is `enabled`. | | thinking | type [*required*] | enum | The thinking mode for Anthropic extended thinking. Allowed enum values: `enabled,disabled,adaptive` | | | azure_openai_metadata | object | Azure OpenAI-specific metadata for an integration account or inference request. | | azure_openai_metadata | deployment_id | string | The Azure OpenAI deployment ID. | | azure_openai_metadata | model_version | string | The model version deployed in Azure. | | azure_openai_metadata | resource_name | string | The Azure OpenAI resource name. | | | bedrock_metadata | object | Amazon Bedrock-specific metadata for an inference request. | | bedrock_metadata | region | string | The AWS region for the Bedrock request. | | | error_response | object | Error details returned when an inference provider returns an error. | | error_response | message [*required*] | string | A human-readable description of the error. | | error_response | type [*required*] | string | The provider-specific error type. | | | frequency_penalty | double | Frequency penalty that was applied. | | | json_schema | string | JSON schema that was applied for structured output. | | | max_completion_tokens | int64 | Maximum number of completion tokens that were configured. | | | max_tokens | int64 | Maximum number of tokens that were configured. | | | messages [*required*] | [object] | List of messages in an inference conversation. | | messages | content | string | Plain text content of the message. | | messages | contents | [object] | List of structured content blocks in a message. | | contents | type [*required*] | string | The content block type. | | contents | value [*required*] | object | The typed value of a message content block. | | value | text | string | Plain text content. | | value | tool_call | object | A tool call made during LLM inference. | | tool_call | arguments | object | The arguments passed to the tool. | | tool_call | name | string | The name of the tool being called. | | tool_call | tool_id | string | Unique identifier for the tool call. | | tool_call | type | string | The type of tool call. | | value | tool_call_result | object | The result returned by a tool call during LLM inference. | | tool_call_result | name | string | The name of the tool that produced this result. | | tool_call_result | result | string | The result content returned by the tool. | | tool_call_result | tool_id | string | Identifier matching the corresponding tool call. | | tool_call_result | type | string | The type of tool result. | | messages | id | string | Unique identifier for the message. | | messages | role | string | The role of the message author. | | messages | tool_calls | [object] | List of tool calls in a message. | | tool_calls | arguments | object | The arguments passed to the tool. | | tool_calls | name | string | The name of the tool being called. | | tool_calls | tool_id | string | Unique identifier for the tool call. | | tool_calls | type | string | The type of tool call. | | messages | tool_results | [object] | List of tool results in a message. | | tool_results | name | string | The name of the tool that produced this result. | | tool_results | result | string | The result content returned by the tool. | | tool_results | tool_id | string | Identifier matching the corresponding tool call. | | tool_results | type | string | The type of tool result. | | | model_id [*required*] | string | The model identifier used for inference. | | | openai_metadata | object | OpenAI-specific metadata for an inference request. | | openai_metadata | reasoning_effort | enum | The reasoning effort level for OpenAI models that support it. Allowed enum values: `none,low,medium,high,xhigh` | | openai_metadata | reasoning_summary | enum | The verbosity of the reasoning summary. Allowed enum values: `auto,concise,detailed` | | | presence_penalty | double | Presence penalty that was applied. | | | response [*required*] | object | The output of a completed LLM inference call. | | response | assessment [*required*] | string | An optional assessment of the inference output quality. | | response | content [*required*] | string | The text content of the model response. | | response | finish_reason [*required*] | string | The reason the model stopped generating tokens. | | response | inference_codes [*required*] | [object] | List of generated code snippets for the inference configuration. | | inference_codes | code [*required*] | string | The generated code content. | | inference_codes | id [*required*] | string | Unique identifier for the code snippet. | | inference_codes | type [*required*] | string | The programming language or SDK type of the code snippet. | | response | input_tokens [*required*] | int64 | Number of input tokens consumed. | | response | internal_reasoning | object | The model's internal reasoning or thinking output, if available. | | internal_reasoning | reasoning_tokens | int64 | Number of tokens used for internal reasoning. | | internal_reasoning | text [*required*] | string | The reasoning text produced by the model. | | response | latency [*required*] | int64 | Request latency in milliseconds. | | response | output_tokens [*required*] | int64 | Number of output tokens generated. | | response | tools [*required*] | [object] | List of tools available to the model. | | tools | function [*required*] | object | A function definition for a tool available to the model. | | function | description | string | A description of what the function does. | | function | name [*required*] | string | The name of the function. | | function | parameters [*required*] | object | JSON schema describing the function parameters. | | tools | type [*required*] | string | The type of tool. | | response | total_tokens [*required*] | int64 | Total tokens used (input plus output). | | | temperature | double | Sampling temperature that was used. | | | tools | [object] | List of tools available to the model. | | tools | function [*required*] | object | A function definition for a tool available to the model. | | function | description | string | A description of what the function does. | | function | name [*required*] | string | The name of the function. | | function | parameters [*required*] | object | JSON schema describing the function parameters. | | tools | type [*required*] | string | The type of tool. | | | top_k | int64 | Top-K sampling parameter that was used. | | | top_p | double | Nucleus sampling parameter that was used. | | | vertex_ai_metadata | object | Vertex AI-specific metadata for an integration account or inference request. | | vertex_ai_metadata | location | string | The Vertex AI region. | | vertex_ai_metadata | project | string | The Google Cloud project ID. | | vertex_ai_metadata | project_ids | [string] | List of Google Cloud project IDs available to the service account. | {% /tab %} {% tab title="Example" %} ```json { "anthropic_metadata": { "effort": "medium", "thinking": { "budget_tokens": 1024, "type": "enabled" } }, "azure_openai_metadata": { "deployment_id": "my-gpt4-deployment", "model_version": "0613", "resource_name": "my-azure-resource" }, "bedrock_metadata": { "region": "us-east-1" }, "error_response": { "message": "The model does not exist.", "type": "invalid_request_error" }, "frequency_penalty": 0, "json_schema": "{\"type\":\"object\",\"properties\":{\"answer\":{\"type\":\"string\"}}}", "max_completion_tokens": 1024, "max_tokens": 1024, "messages": [ { "content": "What is the capital of France?", "contents": [ { "type": "text", "value": { "text": "Hello, how can I help you?", "tool_call": { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" }, "tool_call_result": { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } } } ], "id": "msg_001", "role": "user", "tool_calls": [ { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" } ], "tool_results": [ { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } ] } ], "model_id": "gpt-4o", "openai_metadata": { "reasoning_effort": "medium", "reasoning_summary": "auto" }, "presence_penalty": 0, "response": { "assessment": "pass", "content": "The capital of France is Paris.", "finish_reason": "stop", "inference_codes": [ { "code": "import openai\nclient = openai.OpenAI()\n...", "id": "code-python-001", "type": "python" } ], "input_tokens": 15, "internal_reasoning": { "reasoning_tokens": 256, "text": "Let me think about this step by step..." }, "latency": 843, "output_tokens": 10, "tools": [ { "function": { "description": "Get the current weather for a location.", "name": "get_weather", "parameters": { "properties": { "location": { "type": "string" } }, "type": "object" } }, "type": "function" } ], "total_tokens": 25 }, "temperature": 0.7, "tools": [ { "function": { "description": "Get the current weather for a location.", "name": "get_weather", "parameters": { "properties": { "location": { "type": "string" } }, "type": "object" } }, "type": "function" } ], "top_k": 50, "top_p": 1, "vertex_ai_metadata": { "location": "us-central1", "project": "my-gcp-project", "project_ids": [ "my-gcp-project" ] } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parametersexport integration="openai"export account_id="account-abc123"\# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/integrations/${integration}/${account_id}/inference" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "max_tokens": 256, "messages": [ { "content": "What is the capital of France?", "role": "user" } ], "model_id": "gpt-4o", "temperature": 0.7 } EOF {% /tab %} ## Get annotated interactions by content IDs{% #get-annotated-interactions-by-content-ids %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | -------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotated-interactions | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotated-interactions | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/annotated-interactions | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/annotated-interactions | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotated-interactions | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/annotated-interactions | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotated-interactions | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotated-interactions | ### Overview Returns annotated interactions across all annotation queues for the given content IDs. Results include queue metadata (ID and name) for each interaction. ### Arguments #### Query Strings | Name | Type | Description | | ---------------------------- | ------- | ----------------------------------------------------------------------------------------- | | contentIds [*required*] | array | One or more content IDs to retrieve annotated interactions for. At least one is required. | | offset | integer | Pagination offset. Must be >= 0. Defaults to 0. | | limit | integer | Maximum number of results to return. Must be > 0. Defaults to 100. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing annotated interactions across all queues for the requested content IDs. | Parent field | Field | Type | Description | | ---------------------- | ---------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | data [*required*] | object | Data object for the cross-queue annotated interactions response. | | data | attributes [*required*] | object | Attributes of the cross-queue annotated interactions response. | | attributes | annotated_interactions [*required*] | [object] | List of annotated interactions across all queues for the requested content IDs. | | annotated_interactions | annotations [*required*] | [object] | List of annotations for this interaction. | | annotations | created_at [*required*] | date-time | Timestamp when the annotation was created. | | annotations | created_by [*required*] | string | Identifier of the user who created the annotation. | | annotations | id [*required*] | string | Unique identifier of the annotation. | | annotations | interaction_id [*required*] | string | Identifier of the interaction this annotation belongs to. | | annotations | label_values [*required*] | object | The label values for this annotation. | | annotations | modified_at [*required*] | date-time | Timestamp when the annotation was last modified. | | annotations | modified_by [*required*] | string | Identifier of the user who last modified the annotation. | | annotated_interactions | content_id [*required*] | string | Upstream entity identifier (trace ID, session ID, or deterministic display_block ID). | | annotated_interactions | created_at [*required*] | date-time | Timestamp when the interaction was added to the queue. | | annotated_interactions | display_block | [object] | List of content blocks that make up a `display_block` interaction. Must contain at least one block. | | display_block | alt | string | Alternative text for an `image` block. | | display_block | content | | Block payload. A string for `markdown`, `header`, and `text`; an arbitrary JSON value (object, array, or scalar) for `json`. Omitted for `image`, `widget`, and `llmobs_trace`. | | display_block | height | int64 | Optional rendered height. Must be positive when set. | | display_block | interactionType | enum | Upstream interaction type referenced by an `llmobs_trace` block. Restricted to `trace` or `experiment_trace`. Allowed enum values: `trace,experiment_trace` | | display_block | label | string | Optional label rendered alongside the block. | | display_block | level | enum | Visual size for a `header` block. Allowed enum values: `sm,md,lg,xl` | | display_block | tileDef | | Tile definition for a `widget` block. Required for `widget`. The schema is owned by the frontend renderer. | | display_block | timeFrame | object | Unix-millis time range used by chart blocks. | | timeFrame | end [*required*] | int64 | End of the range, in Unix milliseconds. | | timeFrame | start [*required*] | int64 | Start of the range, in Unix milliseconds. | | display_block | traceId | string | Trace identifier. Required for `llmobs_trace` blocks. | | display_block | type [*required*] | enum | Discriminator for a single `display_block` content block. Adding a variant requires coordinated changes in the frontend renderer. Allowed enum values: `markdown,header,text,json,image,widget,llmobs_trace` | | display_block | url | string | URL of the image. Required for `image` blocks. | | annotated_interactions | id [*required*] | string | Unique identifier of the interaction. | | annotated_interactions | modified_at [*required*] | date-time | Timestamp when the interaction was last updated. | | annotated_interactions | queue_id [*required*] | string | Identifier of the annotation queue this interaction belongs to. | | annotated_interactions | queue_name [*required*] | string | Name of the annotation queue this interaction belongs to. | | annotated_interactions | type [*required*] | enum | Type of an annotated interaction. Allowed enum values: `trace,experiment_trace,session,display_block` | | attributes | total_count [*required*] | int32 | Total number of annotated interactions matching the query. | | data | id [*required*] | string | Opaque identifier for the response object. | | data | type [*required*] | enum | Resource type for cross-queue annotated interactions lookup. Allowed enum values: `annotated_interactions_by_trace` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "annotated_interactions": [ { "annotations": [ { "created_at": "2024-01-15T10:30:00Z", "created_by": "00000000-0000-0000-0000-000000000002", "id": "annotation-789", "interaction_id": "interaction-456", "label_values": { "quality": "good" }, "modified_at": "2024-01-15T10:30:00Z", "modified_by": "00000000-0000-0000-0000-000000000002" } ], "content_id": "trace-abc-123", "created_at": "2025-06-01T12:00:00Z", "display_block": [ { "alt": "Example image", "content": "## Triage Instructions", "height": 240, "interactionType": "trace", "label": "Triage Instructions", "level": "md", "tileDef": { "requests": [ { "queries": [ { "data_source": "metrics", "name": "q", "query": "avg:system.cpu.user{*}" } ], "response_format": "timeseries", "type": "line" } ], "viz": "timeseries" }, "timeFrame": { "end": 1705315800000, "start": 1705312200000 }, "traceId": "69fcc2bb0000000003113989d83069ba", "type": "markdown", "url": "https://example.com/image.png" } ], "id": "interaction-456", "modified_at": "2025-06-01T12:00:00Z", "queue_id": "queue-uuid-001", "queue_name": "My Annotation Queue", "type": "trace" } ], "total_count": 1 }, "id": "trace-query", "type": "annotated_interactions_by_trace" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Required query argumentsexport contentIds="CHANGE_ME"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotated-interactions?contentIds=${contentIds}" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Get LLM Observability dataset draft state{% #get-llm-observability-dataset-draft-state %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | -------------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state | ### Overview Retrieve the draft state of a dataset, including whether it is currently locked for editing and which user holds the lock. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing the draft state of an LLM Observability dataset. | Parent field | Field | Type | Description | | ------------ | -------------------------------- | --------- | -------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an LLM Observability dataset draft state. | | data | attributes [*required*] | object | Attributes of an LLM Observability dataset draft state. | | attributes | drafting_since [*required*] | date-time | Timestamp when the dataset draft session started. | | attributes | user [*required*] | object | User information associated with a dataset draft state. | | user | email | string | Email address of the user. | | user | handle | string | Handle of the user. | | user | icon | string | Icon for the user. | | user | id [*required*] | string | Unique identifier of the user holding the draft lock. | | user | name | string | Display name of the user. | | data | id [*required*] | string | Unique identifier of the dataset draft state. Matches the dataset ID. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset draft state. Allowed enum values: `draft_state_data` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "drafting_since": "2024-01-15T10:30:00Z", "user": { "email": "jane.doe@example.com", "handle": "jane.doe@example.com", "icon": "", "id": "00000000-0000-0000-0000-000000000010", "name": "Jane Doe" } }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "draft_state_data" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/draft_state" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Lock LLM Observability dataset draft state{% #lock-llm-observability-dataset-draft-state %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | | ap2.datadoghq.com | PATCH https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | | app.datadoghq.eu | PATCH https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | | app.ddog-gov.com | PATCH https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | | us2.ddog-gov.com | PATCH https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | | app.datadoghq.com | PATCH https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | | us3.datadoghq.com | PATCH https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | | us5.datadoghq.com | PATCH https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/lock | ### Overview Acquire the draft lock on a dataset for the calling user. The lock prevents other users from concurrently editing the dataset draft. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing the draft state of an LLM Observability dataset. | Parent field | Field | Type | Description | | ------------ | -------------------------------- | --------- | -------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Data object for an LLM Observability dataset draft state. | | data | attributes [*required*] | object | Attributes of an LLM Observability dataset draft state. | | attributes | drafting_since [*required*] | date-time | Timestamp when the dataset draft session started. | | attributes | user [*required*] | object | User information associated with a dataset draft state. | | user | email | string | Email address of the user. | | user | handle | string | Handle of the user. | | user | icon | string | Icon for the user. | | user | id [*required*] | string | Unique identifier of the user holding the draft lock. | | user | name | string | Display name of the user. | | data | id [*required*] | string | Unique identifier of the dataset draft state. Matches the dataset ID. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset draft state. Allowed enum values: `draft_state_data` | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "drafting_since": "2024-01-15T10:30:00Z", "user": { "email": "jane.doe@example.com", "handle": "jane.doe@example.com", "icon": "", "id": "00000000-0000-0000-0000-000000000010", "name": "Jane Doe" } }, "id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "type": "draft_state_data" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/draft_state/lock" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## Unlock LLM Observability dataset draft state{% #unlock-llm-observability-dataset-draft-state %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ----------------------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | | ap2.datadoghq.com | PATCH https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | | app.datadoghq.eu | PATCH https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | | app.ddog-gov.com | PATCH https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | | us2.ddog-gov.com | PATCH https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | | app.datadoghq.com | PATCH https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | | us3.datadoghq.com | PATCH https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | | us5.datadoghq.com | PATCH https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/draft_state/unlock | ### Overview Release the draft lock on a dataset held by the calling user, allowing other users to edit the dataset draft. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Response {% tab title="200" %} OK {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/draft_state/unlock" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %} ## List LLM Observability dataset versions{% #list-llm-observability-dataset-versions %} {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ----------------------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/llm-obs/v1/{project_id}/datasets/{dataset_id}/versions | ### Overview List the active versions of a dataset. A version is created each time a dataset is referenced by an experiment run. ### Arguments #### Path Parameters | Name | Type | Description | | ---------------------------- | ------ | ---------------------------------------- | | project_id [*required*] | string | The ID of the LLM Observability project. | | dataset_id [*required*] | string | The ID of the LLM Observability dataset. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing the active versions of an LLM Observability dataset. | Parent field | Field | Type | Description | | ------------ | -------------------------------- | --------- | ------------------------------------------------------------------------------------------------- | | | data [*required*] | [object] | List of dataset versions. | | data | attributes [*required*] | object | Attributes of an LLM Observability dataset version. | | attributes | dataset_id [*required*] | string | Unique identifier of the dataset this version belongs to. | | attributes | last_used [*required*] | date-time | Timestamp when this dataset version was last referenced. Null if the version has never been used. | | attributes | version_number [*required*] | int32 | Sequential version number for this dataset version. | | data | id [*required*] | string | Unique identifier of the dataset version. | | data | type [*required*] | enum | Resource type of an LLM Observability dataset version. Allowed enum values: `dataset_version` | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "last_used": "2024-01-15T10:30:00Z", "version_number": 1 }, "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012", "type": "dataset_version" } ] } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751"export dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/${project_id}/datasets/${dataset_id}/versions" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" {% /tab %}