Create or update annotations

Note: This endpoint is in preview and is subject to change. If you have any feedback, contact Datadog support.

POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.uk1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotationshttps://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}/annotations

Overview

Create or update annotations on interactions in a queue. Each annotation is matched by interaction_id and the requesting user’s identity. Results and errors in the response are linked to request items by interaction_id. Errors for individual items are returned in the errors field without blocking the rest of the batch. Requests where every item is denied by queue access rules return 403.

Arguments

Path Parameters

Name

Type

Description

queue_id [required]

string

The ID of the Agent Observability annotation queue.

Request

Body Data (required)

Payload for creating or updating annotations.

Expand All

Field

Type

Description

data [required]

object

Data object for creating or updating annotations.

attributes [required]

object

Attributes for creating or updating annotations.

annotations [required]

[object]

List of annotations to create or update. Must contain at least one item.

interaction_id [required]

string

ID of the interaction to annotate.

label_values [required]

[object]

Label values for this annotation. Each entry references a label schema by ID and provides the corresponding value validated against the schema type constraints.

assessment

enum

Assessment result for a label value. Allowed enum values: pass,fail

label_schema_id [required]

string

ID of the label schema this value corresponds to.

reasoning

string

Free text reasoning for this label value.

value [required]

 <oneOf>

The value for this label. Must comply with the label schema type constraints.

Object 1

double

A scalar numeric value.

Object 2

string

A scalar value represented as a string.

Object 3

[string]

For categorical-type labels allowing multiple selections.

Object 4

boolean

A scalar boolean value.

type [required]

enum

Resource type for Agent Observability annotations. Allowed enum values: annotations

{
  "data": {
    "attributes": {
      "annotations": [
        {
          "interaction_id": "00000000-0000-0000-0000-000000000001",
          "label_values": [
            {
              "assessment": "pass",
              "label_schema_id": "abc-123",
              "reasoning": "The response was accurate and well-structured.",
              "value": 0
            }
          ]
        }
      ]
    },
    "type": "annotations"
  }
}

Response

OK — annotations created or updated. For mixed batches, denied items and other per-item errors are listed in `errors`.

Response containing the created or updated annotations.

Expand All

Field

Type

Description

data [required]

object

Data object for the annotations response.

attributes [required]

object

Attributes of the annotations response.

annotations [required]

[object]

Successfully created or updated annotations.

created_at [required]

date-time

Timestamp when the annotation was created.

created_by [required]

string

Identifier of the user who created the annotation.

id [required]

string

Unique identifier of the annotation.

interaction_id [required]

string

Identifier of the interaction this annotation belongs to.

label_values [required]

[object]

Label values for this annotation. Each entry references a label schema by ID and provides the corresponding value.

assessment

enum

Assessment result for a label value. Allowed enum values: pass,fail

label_schema_id [required]

string

ID of the label schema this value corresponds to.

name_when_saved

string

Name of the label schema at the time the annotation was created.

reasoning

string

Free text reasoning for this label value.

type

enum

Type of a label in an annotation queue label schema. Allowed enum values: score,categorical,boolean,text

value [required]

 <oneOf>

The value for this label. Must comply with the label schema type constraints.

Object 1

double

A scalar numeric value.

Object 2

string

A scalar value represented as a string.

Object 3

[string]

For categorical-type labels allowing multiple selections.

Object 4

boolean

A scalar boolean value.

modified_at [required]

date-time

Timestamp when the annotation was last modified.

modified_by [required]

string

Identifier of the user who last modified the annotation.

errors

[object]

Partial errors for annotations that could not be processed.

annotation_id

string

ID of the annotation that failed, if applicable.

code

enum

Stable error code. permission_denied indicates the item was rejected by queue access rules. Allowed enum values: permission_denied

error [required]

string

Error message.

interaction_id [required]

string

ID of the interaction that failed.

id [required]

string

The annotation queue ID.

type [required]

enum

Resource type for Agent Observability annotations. Allowed enum values: annotations

{
  "data": {
    "attributes": {
      "annotations": [
        {
          "created_at": "2024-01-15T10:30:00Z",
          "created_by": "00000000-0000-0000-0000-000000000002",
          "id": "annotation-789",
          "interaction_id": "interaction-456",
          "label_values": [
            {
              "assessment": "pass",
              "label_schema_id": "abc-123",
              "name_when_saved": "quality",
              "reasoning": "The response was accurate and well-structured.",
              "type": "score",
              "value": 0
            }
          ],
          "modified_at": "2024-01-15T10:30:00Z",
          "modified_by": "00000000-0000-0000-0000-000000000002"
        }
      ],
      "errors": [
        {
          "annotation_id": "00000000-0000-0000-0000-000000000000",
          "code": "permission_denied",
          "error": "interaction not found",
          "interaction_id": "00000000-0000-0000-0000-000000000001"
        }
      ]
    },
    "id": "00000000-0000-0000-0000-000000000001",
    "type": "annotations"
  }
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Unauthorized

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Forbidden — all request items were denied by queue access rules

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Not Found — the queue does not exist.

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  ## default
# 

# Path parameters
export queue_id="00000000-0000-0000-0000-000000000001"
# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.uk1.datadoghq.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}/annotations" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "annotations": [ { "interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [ { "label_schema_id": "abc-123", "value": "good" }, { "label_schema_id": "ef56gh78", "value": "positive" } ] } ] }, "type": "annotations" } } EOF
"""
Create or update annotations returns "OK — annotations created or updated. For mixed batches, denied items and other
per-item errors are listed in `errors`." response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.agent_observability_api import AgentObservabilityApi
from datadog_api_client.v2.model.llm_obs_annotation_label_value import LLMObsAnnotationLabelValue
from datadog_api_client.v2.model.llm_obs_annotations_data_attributes_request import (
    LLMObsAnnotationsDataAttributesRequest,
)
from datadog_api_client.v2.model.llm_obs_annotations_data_request import LLMObsAnnotationsDataRequest
from datadog_api_client.v2.model.llm_obs_annotations_request import LLMObsAnnotationsRequest
from datadog_api_client.v2.model.llm_obs_annotations_type import LLMObsAnnotationsType
from datadog_api_client.v2.model.llm_obs_upsert_annotation_item import LLMObsUpsertAnnotationItem

body = LLMObsAnnotationsRequest(
    data=LLMObsAnnotationsDataRequest(
        attributes=LLMObsAnnotationsDataAttributesRequest(
            annotations=[
                LLMObsUpsertAnnotationItem(
                    interaction_id="00000000-0000-0000-0000-000000000001",
                    label_values=[
                        LLMObsAnnotationLabelValue(
                            label_schema_id="abc-123",
                            value="good",
                        ),
                        LLMObsAnnotationLabelValue(
                            label_schema_id="ef56gh78",
                            value="positive",
                        ),
                    ],
                ),
            ],
        ),
        type=LLMObsAnnotationsType.ANNOTATIONS,
    ),
)

configuration = Configuration()
configuration.unstable_operations["upsert_llm_obs_annotations"] = True
with ApiClient(configuration) as api_client:
    api_instance = AgentObservabilityApi(api_client)
    response = api_instance.upsert_llm_obs_annotations(queue_id="queue_id", body=body)

    print(response)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" python3 "example.py"
# Create or update annotations returns "OK — annotations created or updated. For mixed batches, denied items and other
# per-item errors are listed in `errors`." response

require "datadog_api_client"
DatadogAPIClient.configure do |config|
  config.unstable_operations["v2.upsert_llm_obs_annotations".to_sym] = true
end
api_instance = DatadogAPIClient::V2::AgentObservabilityAPI.new

body = DatadogAPIClient::V2::LLMObsAnnotationsRequest.new({
  data: DatadogAPIClient::V2::LLMObsAnnotationsDataRequest.new({
    attributes: DatadogAPIClient::V2::LLMObsAnnotationsDataAttributesRequest.new({
      annotations: [
        DatadogAPIClient::V2::LLMObsUpsertAnnotationItem.new({
          interaction_id: "00000000-0000-0000-0000-000000000001",
          label_values: [
            DatadogAPIClient::V2::LLMObsAnnotationLabelValue.new({
              label_schema_id: "abc-123",
              value: "good",
            }),
            DatadogAPIClient::V2::LLMObsAnnotationLabelValue.new({
              label_schema_id: "ef56gh78",
              value: "positive",
            }),
          ],
        }),
      ],
    }),
    type: DatadogAPIClient::V2::LLMObsAnnotationsType::ANNOTATIONS,
  }),
})
p api_instance.upsert_llm_obs_annotations("queue_id", body)

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" rb "example.rb"
// Create or update annotations returns "OK — annotations created or updated. For mixed batches, denied items and other
// per-item errors are listed in `errors`." response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)

func main() {
	body := datadogV2.LLMObsAnnotationsRequest{
		Data: datadogV2.LLMObsAnnotationsDataRequest{
			Attributes: datadogV2.LLMObsAnnotationsDataAttributesRequest{
				Annotations: []datadogV2.LLMObsUpsertAnnotationItem{
					{
						InteractionId: "00000000-0000-0000-0000-000000000001",
						LabelValues: []datadogV2.LLMObsAnnotationLabelValue{
							{
								LabelSchemaId: "abc-123",
								Value: datadogV2.LLMObsAnnotationLabelValueValue{
									AnyValueString: datadog.PtrString("good")},
							},
							{
								LabelSchemaId: "ef56gh78",
								Value: datadogV2.LLMObsAnnotationLabelValueValue{
									AnyValueString: datadog.PtrString("positive")},
							},
						},
					},
				},
			},
			Type: datadogV2.LLMOBSANNOTATIONSTYPE_ANNOTATIONS,
		},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	configuration.SetUnstableOperationEnabled("v2.UpsertLLMObsAnnotations", true)
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV2.NewAgentObservabilityApi(apiClient)
	resp, r, err := api.UpsertLLMObsAnnotations(ctx, "queue_id", body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `AgentObservabilityApi.UpsertLLMObsAnnotations`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `AgentObservabilityApi.UpsertLLMObsAnnotations`:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" go run "main.go"
// Create or update annotations returns "OK — annotations created or updated. For mixed batches,
// denied items and other
// per-item errors are listed in `errors`." response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.AgentObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsAnnotationLabelValue;
import com.datadog.api.client.v2.model.LLMObsAnnotationLabelValueValue;
import com.datadog.api.client.v2.model.LLMObsAnnotationsDataAttributesRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationsDataRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationsRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationsResponse;
import com.datadog.api.client.v2.model.LLMObsAnnotationsType;
import com.datadog.api.client.v2.model.LLMObsUpsertAnnotationItem;
import java.util.Arrays;
import java.util.Collections;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    defaultClient.setUnstableOperationEnabled("v2.upsertLLMObsAnnotations", true);
    AgentObservabilityApi apiInstance = new AgentObservabilityApi(defaultClient);

    LLMObsAnnotationsRequest body =
        new LLMObsAnnotationsRequest()
            .data(
                new LLMObsAnnotationsDataRequest()
                    .attributes(
                        new LLMObsAnnotationsDataAttributesRequest()
                            .annotations(
                                Collections.singletonList(
                                    new LLMObsUpsertAnnotationItem()
                                        .interactionId("00000000-0000-0000-0000-000000000001")
                                        .labelValues(
                                            Arrays.asList(
                                                new LLMObsAnnotationLabelValue()
                                                    .labelSchemaId("abc-123")
                                                    .value(
                                                        new LLMObsAnnotationLabelValueValue(
                                                            "good")),
                                                new LLMObsAnnotationLabelValue()
                                                    .labelSchemaId("ef56gh78")
                                                    .value(
                                                        new LLMObsAnnotationLabelValueValue(
                                                            "positive")))))))
                    .type(LLMObsAnnotationsType.ANNOTATIONS));

    try {
      LLMObsAnnotationsResponse result =
          apiInstance.upsertLLMObsAnnotations("00000000-0000-0000-0000-000000000001", body);
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling AgentObservabilityApi#upsertLLMObsAnnotations");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" java "Example.java"
// Create or update annotations returns "OK — annotations created or updated. For
// mixed batches, denied items and other per-item errors are listed in `errors`."
// response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_agent_observability::AgentObservabilityAPI;
use datadog_api_client::datadogV2::model::LLMObsAnnotationLabelValue;
use datadog_api_client::datadogV2::model::LLMObsAnnotationLabelValueValue;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsDataAttributesRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsDataRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationsType;
use datadog_api_client::datadogV2::model::LLMObsUpsertAnnotationItem;

#[tokio::main]
async fn main() {
    let body = LLMObsAnnotationsRequest::new(LLMObsAnnotationsDataRequest::new(
        LLMObsAnnotationsDataAttributesRequest::new(vec![LLMObsUpsertAnnotationItem::new(
            "00000000-0000-0000-0000-000000000001".to_string(),
            vec![
                LLMObsAnnotationLabelValue::new(
                    "abc-123".to_string(),
                    LLMObsAnnotationLabelValueValue::AnyValueString("good".to_string()),
                ),
                LLMObsAnnotationLabelValue::new(
                    "ef56gh78".to_string(),
                    LLMObsAnnotationLabelValueValue::AnyValueString("positive".to_string()),
                ),
            ],
        )]),
        LLMObsAnnotationsType::ANNOTATIONS,
    ));
    let mut configuration = datadog::Configuration::new();
    configuration.set_unstable_operation_enabled("v2.UpsertLLMObsAnnotations", true);
    let api = AgentObservabilityAPI::with_config(configuration);
    let resp = api
        .upsert_llm_obs_annotations("queue_id".to_string(), body)
        .await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run
/**
 * Create or update annotations returns "OK — annotations created or updated. For mixed batches, denied items and other
 * per-item errors are listed in `errors`." response
 */

import { client, v2 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
configuration.unstableOperations["v2.upsertLLMObsAnnotations"] = true;
const apiInstance = new v2.AgentObservabilityApi(configuration);

const params: v2.AgentObservabilityApiUpsertLLMObsAnnotationsRequest = {
  body: {
    data: {
      attributes: {
        annotations: [
          {
            interactionId: "00000000-0000-0000-0000-000000000001",
            labelValues: [
              {
                labelSchemaId: "abc-123",
                value: "good",
              },
              {
                labelSchemaId: "ef56gh78",
                value: "positive",
              },
            ],
          },
        ],
      },
      type: "annotations",
    },
  },
  queueId: "queue_id",
};

apiInstance
  .upsertLLMObsAnnotations(params)
  .then((data: v2.LLMObsAnnotationsResponse) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" tsc "example.ts"