Update an LLM Observability annotation queue

v2 (latest)

Note: This endpoint is in preview and is subject to change. If you have any feedback, contact Datadog support.

PATCH https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}https://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queues/{queue_id}https://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}https://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}https://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/{queue_id}

Overview

Partially update an annotation queue. The name, description, and annotation_schema fields can be updated.

Arguments

Path Parameters

Name

Type

Description

queue_id [required]

string

The ID of the LLM Observability annotation queue.

Request

Body Data (required)

Update annotation queue payload.

Expand All

Field

Type

Description

data [required]

object

Data object for updating an LLM Observability annotation queue.

{
  "data": {
    "attributes": {
      "annotation_schema": {
        "label_schemas": [
          {
            "description": "Rating of the response quality.",
            "has_assessment": false,
            "has_reasoning": false,
            "id": "abc-123",
            "is_assessment": false,
            "is_integer": false,
            "is_required": true,
            "max": 5,
            "min": 0,
            "name": "quality",
            "type": "score",
            "values": [
              "good",
              "bad",
              "neutral"
            ]
          }
        ]
      },
      "description": "Updated description",
      "name": "Updated queue name"
    },
    "type": "queues"
  }
}

Response

Response containing a single LLM Observability annotation queue.

Expand All

Field

Type

Description

data [required]

object

Data object for an LLM Observability annotation queue.

{
  "data": {
    "attributes": {
      "annotation_schema": {
        "label_schemas": [
          {
            "description": "Rating of the response quality.",
            "has_assessment": false,
            "has_reasoning": false,
            "id": "abc-123",
            "is_assessment": false,
            "is_integer": false,
            "is_required": true,
            "max": 5,
            "min": 0,
            "name": "quality",
            "type": "score",
            "values": [
              "good",
              "bad",
              "neutral"
            ]
          }
        ]
      },
      "created_at": "2024-01-15T10:30:00Z",
      "created_by": "00000000-0000-0000-0000-000000000002",
      "description": "Queue for annotating customer support traces",
      "modified_at": "2024-01-15T10:30:00Z",
      "modified_by": "00000000-0000-0000-0000-000000000002",
      "name": "My annotation queue",
      "owned_by": "00000000-0000-0000-0000-000000000002",
      "project_id": "00000000-0000-0000-0000-000000000002"
    },
    "id": "00000000-0000-0000-0000-000000000001",
    "type": "queues"
  }
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Unauthorized

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Forbidden

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Not Found

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  ## default
# 

# Path parameters
export queue_id="00000000-0000-0000-0000-000000000001"
# Curl command
curl -X PATCH "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues/${queue_id}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d @- << EOF
{
  "data": {
    "attributes": {
      "description": "Updated description",
      "name": "Updated queue name"
    },
    "type": "queues"
  }
}
EOF

                

"""
Update an LLM Observability annotation queue returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_annotation_queue_type import LLMObsAnnotationQueueType
from datadog_api_client.v2.model.llm_obs_annotation_queue_update_data_attributes_request import (
    LLMObsAnnotationQueueUpdateDataAttributesRequest,
)
from datadog_api_client.v2.model.llm_obs_annotation_queue_update_data_request import (
    LLMObsAnnotationQueueUpdateDataRequest,
)
from datadog_api_client.v2.model.llm_obs_annotation_queue_update_request import LLMObsAnnotationQueueUpdateRequest
from datadog_api_client.v2.model.llm_obs_annotation_schema import LLMObsAnnotationSchema
from datadog_api_client.v2.model.llm_obs_label_schema import LLMObsLabelSchema
from datadog_api_client.v2.model.llm_obs_label_schema_type import LLMObsLabelSchemaType

body = LLMObsAnnotationQueueUpdateRequest(
    data=LLMObsAnnotationQueueUpdateDataRequest(
        attributes=LLMObsAnnotationQueueUpdateDataAttributesRequest(
            annotation_schema=LLMObsAnnotationSchema(
                label_schemas=[
                    LLMObsLabelSchema(
                        description="Rating of the response quality.",
                        has_assessment=False,
                        has_reasoning=False,
                        id="abc-123",
                        is_assessment=False,
                        is_integer=False,
                        is_required=True,
                        max=5.0,
                        min=0.0,
                        name="quality",
                        type=LLMObsLabelSchemaType.SCORE,
                        values=[
                            "good",
                            "bad",
                            "neutral",
                        ],
                    ),
                ],
            ),
            description="Updated description",
            name="Updated queue name",
        ),
        type=LLMObsAnnotationQueueType.QUEUES,
    ),
)

configuration = Configuration()
configuration.unstable_operations["update_llm_obs_annotation_queue"] = True
with ApiClient(configuration) as api_client:
    api_instance = LLMObservabilityApi(api_client)
    response = api_instance.update_llm_obs_annotation_queue(queue_id="queue_id", body=body)

    print(response)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" python3 "example.py"

# Update an LLM Observability annotation queue returns "OK" response

require "datadog_api_client"
DatadogAPIClient.configure do |config|
  config.unstable_operations["v2.update_llm_obs_annotation_queue".to_sym] = true
end
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new

body = DatadogAPIClient::V2::LLMObsAnnotationQueueUpdateRequest.new({
  data: DatadogAPIClient::V2::LLMObsAnnotationQueueUpdateDataRequest.new({
    attributes: DatadogAPIClient::V2::LLMObsAnnotationQueueUpdateDataAttributesRequest.new({
      annotation_schema: DatadogAPIClient::V2::LLMObsAnnotationSchema.new({
        label_schemas: [
          DatadogAPIClient::V2::LLMObsLabelSchema.new({
            description: "Rating of the response quality.",
            has_assessment: false,
            has_reasoning: false,
            id: "abc-123",
            is_assessment: false,
            is_integer: false,
            is_required: true,
            max: 5.0,
            min: 0.0,
            name: "quality",
            type: DatadogAPIClient::V2::LLMObsLabelSchemaType::SCORE,
            values: [
              "good",
              "bad",
              "neutral",
            ],
          }),
        ],
      }),
      description: "Updated description",
      name: "Updated queue name",
    }),
    type: DatadogAPIClient::V2::LLMObsAnnotationQueueType::QUEUES,
  }),
})
p api_instance.update_llm_obs_annotation_queue("queue_id", body)

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" rb "example.rb"

// Update an LLM Observability annotation queue returns "OK" response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)

func main() {
	body := datadogV2.LLMObsAnnotationQueueUpdateRequest{
		Data: datadogV2.LLMObsAnnotationQueueUpdateDataRequest{
			Attributes: datadogV2.LLMObsAnnotationQueueUpdateDataAttributesRequest{
				AnnotationSchema: &datadogV2.LLMObsAnnotationSchema{
					LabelSchemas: []datadogV2.LLMObsLabelSchema{
						{
							Description:   datadog.PtrString("Rating of the response quality."),
							HasAssessment: datadog.PtrBool(false),
							HasReasoning:  datadog.PtrBool(false),
							Id:            datadog.PtrString("abc-123"),
							IsAssessment:  datadog.PtrBool(false),
							IsInteger:     datadog.PtrBool(false),
							IsRequired:    datadog.PtrBool(true),
							Max:           datadog.PtrFloat64(5.0),
							Min:           datadog.PtrFloat64(0.0),
							Name:          "quality",
							Type:          datadogV2.LLMOBSLABELSCHEMATYPE_SCORE,
							Values: []string{
								"good",
								"bad",
								"neutral",
							},
						},
					},
				},
				Description: datadog.PtrString("Updated description"),
				Name:        datadog.PtrString("Updated queue name"),
			},
			Type: datadogV2.LLMOBSANNOTATIONQUEUETYPE_QUEUES,
		},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	configuration.SetUnstableOperationEnabled("v2.UpdateLLMObsAnnotationQueue", true)
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV2.NewLLMObservabilityApi(apiClient)
	resp, r, err := api.UpdateLLMObsAnnotationQueue(ctx, "queue_id", body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LLMObservabilityApi.UpdateLLMObsAnnotationQueue`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LLMObservabilityApi.UpdateLLMObsAnnotationQueue`:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" go run "main.go"

// Update an LLM Observability annotation queue returns "OK" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.LlmObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsAnnotationQueueResponse;
import com.datadog.api.client.v2.model.LLMObsAnnotationQueueType;
import com.datadog.api.client.v2.model.LLMObsAnnotationQueueUpdateDataAttributesRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationQueueUpdateDataRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationQueueUpdateRequest;
import com.datadog.api.client.v2.model.LLMObsAnnotationSchema;
import com.datadog.api.client.v2.model.LLMObsLabelSchema;
import com.datadog.api.client.v2.model.LLMObsLabelSchemaType;
import java.util.Arrays;
import java.util.Collections;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    defaultClient.setUnstableOperationEnabled("v2.updateLLMObsAnnotationQueue", true);
    LlmObservabilityApi apiInstance = new LlmObservabilityApi(defaultClient);

    LLMObsAnnotationQueueUpdateRequest body =
        new LLMObsAnnotationQueueUpdateRequest()
            .data(
                new LLMObsAnnotationQueueUpdateDataRequest()
                    .attributes(
                        new LLMObsAnnotationQueueUpdateDataAttributesRequest()
                            .annotationSchema(
                                new LLMObsAnnotationSchema()
                                    .labelSchemas(
                                        Collections.singletonList(
                                            new LLMObsLabelSchema()
                                                .description("Rating of the response quality.")
                                                .hasAssessment(false)
                                                .hasReasoning(false)
                                                .id("abc-123")
                                                .isAssessment(false)
                                                .isInteger(false)
                                                .isRequired(true)
                                                .max(5.0)
                                                .name("quality")
                                                .type(LLMObsLabelSchemaType.SCORE)
                                                .values(Arrays.asList("good", "bad", "neutral")))))
                            .description("Updated description")
                            .name("Updated queue name"))
                    .type(LLMObsAnnotationQueueType.QUEUES));

    try {
      LLMObsAnnotationQueueResponse result =
          apiInstance.updateLLMObsAnnotationQueue("00000000-0000-0000-0000-000000000001", body);
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LlmObservabilityApi#updateLLMObsAnnotationQueue");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" java "Example.java"

// Update an LLM Observability annotation queue returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_llm_observability::LLMObservabilityAPI;
use datadog_api_client::datadogV2::model::LLMObsAnnotationQueueType;
use datadog_api_client::datadogV2::model::LLMObsAnnotationQueueUpdateDataAttributesRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationQueueUpdateDataRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationQueueUpdateRequest;
use datadog_api_client::datadogV2::model::LLMObsAnnotationSchema;
use datadog_api_client::datadogV2::model::LLMObsLabelSchema;
use datadog_api_client::datadogV2::model::LLMObsLabelSchemaType;

#[tokio::main]
async fn main() {
    let body =
        LLMObsAnnotationQueueUpdateRequest::new(LLMObsAnnotationQueueUpdateDataRequest::new(
            LLMObsAnnotationQueueUpdateDataAttributesRequest::new()
                .annotation_schema(LLMObsAnnotationSchema::new(vec![LLMObsLabelSchema::new(
                    "quality".to_string(),
                    LLMObsLabelSchemaType::SCORE,
                )
                .description("Rating of the response quality.".to_string())
                .has_assessment(false)
                .has_reasoning(false)
                .id("abc-123".to_string())
                .is_assessment(false)
                .is_integer(false)
                .is_required(true)
                .max(5.0 as f64)
                .min(0.0 as f64)
                .values(vec![
                    "good".to_string(),
                    "bad".to_string(),
                    "neutral".to_string(),
                ])]))
                .description("Updated description".to_string())
                .name("Updated queue name".to_string()),
            LLMObsAnnotationQueueType::QUEUES,
        ));
    let mut configuration = datadog::Configuration::new();
    configuration.set_unstable_operation_enabled("v2.UpdateLLMObsAnnotationQueue", true);
    let api = LLMObservabilityAPI::with_config(configuration);
    let resp = api
        .update_llm_obs_annotation_queue("queue_id".to_string(), body)
        .await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run

/**
 * Update an LLM Observability annotation queue returns "OK" response
 */

import { client, v2 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
configuration.unstableOperations["v2.updateLLMObsAnnotationQueue"] = true;
const apiInstance = new v2.LLMObservabilityApi(configuration);

const params: v2.LLMObservabilityApiUpdateLLMObsAnnotationQueueRequest = {
  body: {
    data: {
      attributes: {
        annotationSchema: {
          labelSchemas: [
            {
              description: "Rating of the response quality.",
              hasAssessment: false,
              hasReasoning: false,
              id: "abc-123",
              isAssessment: false,
              isInteger: false,
              isRequired: true,
              max: 5.0,
              min: 0.0,
              name: "quality",
              type: "score",
              values: ["good", "bad", "neutral"],
            },
          ],
        },
        description: "Updated description",
        name: "Updated queue name",
      },
      type: "queues",
    },
  },
  queueId: "queue_id",
};

apiInstance
  .updateLLMObsAnnotationQueue(params)
  .then((data: v2.LLMObsAnnotationQueueResponse) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" tsc "example.ts"

Update an LLM Observability annotation queue

Overview

Arguments

Path Parameters

Request

Body Data (required)

Response

Code Example

Update an LLM Observability annotation queue

Update an LLM Observability annotation queue

Instructions

Update an LLM Observability annotation queue

Instructions

Update an LLM Observability annotation queue

Instructions

Update an LLM Observability annotation queue

Instructions

Update an LLM Observability annotation queue

Instructions

Update an LLM Observability annotation queue

Instructions

How can I help you today?