For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/api/latest/llm-observability/create-or-update-annotations.md. A documentation index is available at /llms.txt.

Create or update annotations

Note: This endpoint is in preview and is subject to change. If you have any feedback, contact Datadog support.

POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.uk1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotations

Overview

Create or update annotations on interactions in a queue. Each annotation is matched by interaction_id and the requesting user’s identity. Results and errors in the response are linked to request items by interaction_id. Errors for individual items are returned in the errors field without blocking the rest of the batch.

Arguments

Path Parameters

Name

Type

Description

queue_id [required]

string

The ID of the LLM Observability annotation queue.

Request

Body Data (required)

Payload for creating or updating annotations.

Expand All

Field

Type

Description

data [required]

object

Data object for creating or updating annotations.

attributes [required]

object

Attributes for creating or updating annotations.

annotations [required]

[object]

List of annotations to create or update. Must contain at least one item.

interaction_id [required]

string

ID of the interaction to annotate.

label_values [required]

[object]

Label values for this annotation. Each entry references a label schema by ID and provides the corresponding value validated against the schema type constraints.

assessment

enum

Assessment result for a label value. Allowed enum values: pass,fail

label_schema_id [required]

string

ID of the label schema this value corresponds to.

reasoning

string

Free text reasoning for this label value.

value [required]

 <oneOf>

The value for this label. Must comply with the label schema type constraints.

Option 1

double

A scalar numeric value.

Option 2

string

A scalar string value.

Option 3

[string]

For categorical-type labels allowing multiple selections.

Option 4

boolean

A scalar boolean value.

type [required]

enum

Resource type for LLM Observability annotations. Allowed enum values: annotations

{
  "data": {
    "attributes": {
      "annotations": [
        {
          "interaction_id": "00000000-0000-0000-0000-000000000001",
          "label_values": [
            {
              "assessment": "pass",
              "label_schema_id": "abc-123",
              "reasoning": "The response was accurate and well-structured.",
              "value": 0
            }
          ]
        }
      ]
    },
    "type": "annotations"
  }
}

Response

OK — annotations created or updated. Per-item errors are listed in `errors`.

Response containing the created or updated annotations.

Expand All

Field

Type

Description

data [required]

object

Data object for the annotations response.

attributes [required]

object

Attributes of the annotations response.

annotations [required]

[object]

Successfully created or updated annotations.

created_at [required]

date-time

Timestamp when the annotation was created.

created_by [required]

string

Identifier of the user who created the annotation.

id [required]

string

Unique identifier of the annotation.

interaction_id [required]

string

Identifier of the interaction this annotation belongs to.

label_values [required]

[object]

Label values for this annotation. Each entry references a label schema by ID and provides the corresponding value.

assessment

enum

Assessment result for a label value. Allowed enum values: pass,fail

label_schema_id [required]

string

ID of the label schema this value corresponds to.

name_when_saved

string

Name of the label schema at the time the annotation was created.

reasoning

string

Free text reasoning for this label value.

type

enum

Type of a label in an annotation queue label schema. Allowed enum values: score,categorical,boolean,text

value [required]

 <oneOf>

The value for this label. Must comply with the label schema type constraints.

Option 1

double

A scalar numeric value.

Option 2

string

A scalar string value.

Option 3

[string]

For categorical-type labels allowing multiple selections.

Option 4

boolean

A scalar boolean value.

modified_at [required]

date-time

Timestamp when the annotation was last modified.

modified_by [required]

string

Identifier of the user who last modified the annotation.

errors

[object]

Partial errors for annotations that could not be processed.

annotation_id

string

ID of the annotation that failed, if applicable.

error [required]

string

Error message.

interaction_id [required]

string

ID of the interaction that failed.

id [required]

string

The annotation queue ID.

type [required]

enum

Resource type for LLM Observability annotations. Allowed enum values: annotations

{
  "data": {
    "attributes": {
      "annotations": [
        {
          "created_at": "2024-01-15T10:30:00Z",
          "created_by": "00000000-0000-0000-0000-000000000002",
          "id": "annotation-789",
          "interaction_id": "interaction-456",
          "label_values": [
            {
              "assessment": "pass",
              "label_schema_id": "abc-123",
              "name_when_saved": "quality",
              "reasoning": "The response was accurate and well-structured.",
              "type": "score",
              "value": 0
            }
          ],
          "modified_at": "2024-01-15T10:30:00Z",
          "modified_by": "00000000-0000-0000-0000-000000000002"
        }
      ],
      "errors": [
        {
          "annotation_id": "00000000-0000-0000-0000-000000000000",
          "error": "interaction not found",
          "interaction_id": "00000000-0000-0000-0000-000000000001"
        }
      ]
    },
    "id": "00000000-0000-0000-0000-000000000001",
    "type": "annotations"
  }
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Unauthorized

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Forbidden

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Not Found — the queue does not exist.

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  ## default
# 

# Path parameters
export queue_id="00000000-0000-0000-0000-000000000001"
# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.uk1.datadoghq.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/annotations" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotations": [ { "interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [ { "label_schema_id": "abc-123", "value": "good" }, { "label_schema_id": "ef56gh78", "value": "positive" } ] } ] }, "type": "annotations" } } EOF
"""
Create or update annotations returns "OK — annotations created or updated. Per-item errors are listed in `errors`."
response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_annotation_label_value import LLMObsAnnotationLabelValue
from datadog_api_client.v2.model.llm_obs_annotations_data_attributes_request import (
    LLMObsAnnotationsDataAttributesRequest,
)
from datadog_api_client.v2.model.llm_obs_annotations_data_request import LLMObsAnnotationsDataRequest
from datadog_api_client.v2.model.llm_obs_annotations_request import LLMObsAnnotationsRequest
from datadog_api_client.v2.model.llm_obs_annotations_type import LLMObsAnnotationsType
from datadog_api_client.v2.model.llm_obs_upsert_annotation_item import LLMObsUpsertAnnotationItem

body = LLMObsAnnotationsRequest(
    data=LLMObsAnnotationsDataRequest(
        attributes=LLMObsAnnotationsDataAttributesRequest(
            annotations=[
                LLMObsUpsertAnnotationItem(
                    interaction_id="00000000-0000-0000-0000-000000000001",
                    label_values=[
                        LLMObsAnnotationLabelValue(
                            label_schema_id="abc-123",
                            value="good",
                        ),
                        LLMObsAnnotationLabelValue(
                            label_schema_id="ef56gh78",
                            value="positive",
                        ),
                    ],
                ),
            ],
        ),
        type=LLMObsAnnotationsType.ANNOTATIONS,
    ),
)

configuration = Configuration()
configuration.unstable_operations["upsert_llm_obs_annotations"] = True
with ApiClient(configuration) as api_client:
    api_instance = LLMObservabilityApi(api_client)
    response = api_instance.upsert_llm_obs_annotations(queue_id="queue_id", body=body)

    print(response)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" python3 "example.py"
# Create or update annotations returns "OK — annotations created or updated. Per-item errors are listed in `errors`."
# response

require "datadog_api_client"
DatadogAPIClient.configure do |config|
  config.unstable_operations["v2.upsert_llm_obs_annotations".to_sym] = true
end
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new

body = DatadogAPIClient::V2::LLMObsAnnotationsRequest.new({
  data: DatadogAPIClient::V2::LLMObsAnnotationsDataRequest.new({
    attributes: DatadogAPIClient::V2::LLMObsAnnotationsDataAttributesRequest.new({
      annotations: [
        DatadogAPIClient::V2::LLMObsUpsertAnnotationItem.new({
          interaction_id: "00000000-0000-0000-0000-000000000001",
          label_values: [
            DatadogAPIClient::V2::LLMObsAnnotationLabelValue.new({
              label_schema_id: "abc-123",
              value: "good",
            }),
            DatadogAPIClient::V2::LLMObsAnnotationLabelValue.new({
              label_schema_id: "ef56gh78",
              value: "positive",
            }),
          ],
        }),
      ],
    }),
    type: DatadogAPIClient::V2::LLMObsAnnotationsType::ANNOTATIONS,
  }),
})
p api_instance.upsert_llm_obs_annotations("queue_id", body)

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" rb "example.rb"
// Create or update annotations returns "OK — annotations created or updated. Per-item errors are listed in `errors`."
// response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)

func main() {
	body := datadogV2.LLMObsAnnotationsRequest{
		Data: datadogV2.LLMObsAnnotationsDataRequest{
			Attributes: datadogV2.LLMObsAnnotationsDataAttributesRequest{
				Annotations: []datadogV2.LLMObsUpsertAnnotationItem{
					{
						InteractionId: "00000000-0000-0000-0000-000000000001",
						LabelValues: []datadogV2.LLMObsAnnotationLabelValue{
							{
								LabelSchemaId: "abc-123",
								Value: datadogV2.LLMObsAnnotationLabelValueValue{
									AnyValueString: datadog.PtrString("good")},
							},
							{
								LabelSchemaId: "ef56gh78",
								Value: datadogV2.LLMObsAnnotationLabelValueValue{
									AnyValueString: datadog.PtrString("positive")},
							},
						},
					},
				},
			},
			Type: datadogV2.LLMOBSANNOTATIONSTYPE_ANNOTATIONS,
		},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	configuration.SetUnstableOperationEnabled("v2.UpsertLLMObsAnnotations", true)
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV2.NewLLMObservabilityApi(apiClient)
	resp, r, err := api.UpsertLLMObsAnnotations(ctx, "queue_id", body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LLMObservabilityApi.UpsertLLMObsAnnotations`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LLMObservabilityApi.UpsertLLMObsAnnotations`:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" go run "main.go"
// Create or update annotations returns "OK — annotations created or updated. Per-item errors are
// listed in `errors`."
// response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.LlmObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsAnnotationLabelValue;
import com.datadog.api.client.v2.model.LLMObsAnnotationLabelValueValue;
import com.datadog.api.client.v2.model.LLMObsAnnotationsDataAttributesRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationsDataRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationsRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationsResponse;
import com.datadog.api.client.v2.model.LLMObsAnnotationsType;
import com.datadog.api.client.v2.model.LLMObsUpsertAnnotationItem;
import java.util.Arrays;
import java.util.Collections;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    defaultClient.setUnstableOperationEnabled("v2.upsertLLMObsAnnotations", true);
    LlmObservabilityApi apiInstance = new LlmObservabilityApi(defaultClient);

    LLMObsAnnotationsRequest body =
        new LLMObsAnnotationsRequest()
            .data(
                new LLMObsAnnotationsDataRequest()
                    .attributes(
                        new LLMObsAnnotationsDataAttributesRequest()
                            .annotations(
                                Collections.singletonList(
                                    new LLMObsUpsertAnnotationItem()
                                        .interactionId("00000000-0000-0000-0000-000000000001")
                                        .labelValues(
                                            Arrays.asList(
                                                new LLMObsAnnotationLabelValue()
                                                    .labelSchemaId("abc-123")
                                                    .value(
                                                        new LLMObsAnnotationLabelValueValue(
                                                            "good")),
                                                new LLMObsAnnotationLabelValue()
                                                    .labelSchemaId("ef56gh78")
                                                    .value(
                                                        new LLMObsAnnotationLabelValueValue(
                                                            "positive")))))))
                    .type(LLMObsAnnotationsType.ANNOTATIONS));

    try {
      LLMObsAnnotationsResponse result =
          apiInstance.upsertLLMObsAnnotations("00000000-0000-0000-0000-000000000001", body);
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LlmObservabilityApi#upsertLLMObsAnnotations");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" java "Example.java"
// Create or update annotations returns "OK — annotations created or updated.
// Per-item errors are listed in `errors`." response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_llm_observability::LLMObservabilityAPI;
use datadog_api_client::datadogV2::model::LLMObsAnnotationLabelValue;
use datadog_api_client::datadogV2::model::LLMObsAnnotationLabelValueValue;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsDataAttributesRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsDataRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsType;
use datadog_api_client::datadogV2::model::LLMObsUpsertAnnotationItem;

#[tokio::main]
async fn main() {
    let body = LLMObsAnnotationsRequest::new(LLMObsAnnotationsDataRequest::new(
        LLMObsAnnotationsDataAttributesRequest::new(vec![LLMObsUpsertAnnotationItem::new(
            "00000000-0000-0000-0000-000000000001".to_string(),
            vec![
                LLMObsAnnotationLabelValue::new(
                    "abc-123".to_string(),
                    LLMObsAnnotationLabelValueValue::AnyValueString("good".to_string()),
                ),
                LLMObsAnnotationLabelValue::new(
                    "ef56gh78".to_string(),
                    LLMObsAnnotationLabelValueValue::AnyValueString("positive".to_string()),
                ),
            ],
        )]),
        LLMObsAnnotationsType::ANNOTATIONS,
    ));
    let mut configuration = datadog::Configuration::new();
    configuration.set_unstable_operation_enabled("v2.UpsertLLMObsAnnotations", true);
    let api = LLMObservabilityAPI::with_config(configuration);
    let resp = api
        .upsert_llm_obs_annotations("queue_id".to_string(), body)
        .await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run
/**
 * Create or update annotations returns "OK — annotations created or updated. Per-item errors are listed in `errors`."
 * response
 */

import { client, v2 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
configuration.unstableOperations["v2.upsertLLMObsAnnotations"] = true;
const apiInstance = new v2.LLMObservabilityApi(configuration);

const params: v2.LLMObservabilityApiUpsertLLMObsAnnotationsRequest = {
  body: {
    data: {
      attributes: {
        annotations: [
          {
            interactionId: "00000000-0000-0000-0000-000000000001",
            labelValues: [
              {
                labelSchemaId: "abc-123",
                value: "good",
              },
              {
                labelSchemaId: "ef56gh78",
                value: "positive",
              },
            ],
          },
        ],
      },
      type: "annotations",
    },
  },
  queueId: "queue_id",
};

apiInstance
  .upsertLLMObsAnnotations(params)
  .then((data: v2.LLMObsAnnotationsResponse) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" tsc "example.ts"