List LLM Observability annotation queues

Note: This endpoint is in preview and is subject to change. If you have any feedback, contact Datadog support.

GET https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/annotation-queueshttps://api.ap2.datadoghq.com/api/v2/llm-obs/v1/annotation-queueshttps://api.datadoghq.eu/api/v2/llm-obs/v1/annotation-queueshttps://api.ddog-gov.com/api/v2/llm-obs/v1/annotation-queueshttps://api.us2.ddog-gov.com/api/v2/llm-obs/v1/annotation-queueshttps://api.datadoghq.com/api/v2/llm-obs/v1/annotation-queueshttps://api.us3.datadoghq.com/api/v2/llm-obs/v1/annotation-queueshttps://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues

Overview

List annotation queues. Optionally filter by project ID or queue IDs. These parameters are mutually exclusive. If neither is provided, all queues in the organization are returned.

Arguments

Query Strings

Name

Type

Description

projectId

string

Filter annotation queues by project ID. Cannot be used together with queueIds.

queueIds

array

Filter annotation queues by queue IDs (comma-separated). Cannot be used together with projectId.

Response

OK

Response containing a list of LLM Observability annotation queues.

Expand All

Field

Type

Description

data [required]

[object]

List of annotation queues.

attributes [required]

object

Attributes of an LLM Observability annotation queue.

annotation_schema

object

Schema defining the labels for an annotation queue.

label_schemas [required]

[object]

List of label schema definitions.

description

string

Description of the label.

has_assessment

boolean

Whether this label includes an assessment field.

has_reasoning

boolean

Whether this label includes a reasoning field.

id

string

Unique identifier of the label schema. Assigned by the server if not provided.

is_assessment

boolean

Whether the boolean label represents an assessment. Requires has_assessment to be true.

is_integer

boolean

Whether score values must be integers. Applicable to score-type labels.

is_required

boolean

Whether this label is required for an annotation.

max

double

Maximum value for score-type labels.

min

double

Minimum value for score-type labels.

name [required]

string

Name of the label. Must match the pattern ^[a-zA-Z0-9_-]+$ and be unique within the queue.

type [required]

enum

Type of a label in an annotation queue label schema. Allowed enum values: score,categorical,boolean,text

values

[string]

Allowed values for categorical-type labels. Must contain at least one non-empty, unique value.

created_at [required]

date-time

Timestamp when the queue was created.

created_by [required]

string

Identifier of the user who created the queue.

description [required]

string

Description of the annotation queue.

modified_at [required]

date-time

Timestamp when the queue was last modified.

modified_by [required]

string

Identifier of the user who last modified the queue.

name [required]

string

Name of the annotation queue.

owned_by [required]

string

Identifier of the user who owns the queue.

project_id [required]

string

Identifier of the project this queue belongs to.

id [required]

string

Unique identifier of the annotation queue.

type [required]

enum

Resource type of an LLM Observability annotation queue. Allowed enum values: queues

{
  "data": [
    {
      "attributes": {
        "annotation_schema": {
          "label_schemas": [
            {
              "description": "Rating of the response quality.",
              "has_assessment": false,
              "has_reasoning": false,
              "id": "abc-123",
              "is_assessment": false,
              "is_integer": false,
              "is_required": true,
              "max": 5,
              "min": 0,
              "name": "quality",
              "type": "score",
              "values": [
                "good",
                "bad",
                "neutral"
              ]
            }
          ]
        },
        "created_at": "2024-01-15T10:30:00Z",
        "created_by": "00000000-0000-0000-0000-000000000002",
        "description": "Queue for annotating customer support traces",
        "modified_at": "2024-01-15T10:30:00Z",
        "modified_by": "00000000-0000-0000-0000-000000000002",
        "name": "My annotation queue",
        "owned_by": "00000000-0000-0000-0000-000000000002",
        "project_id": "00000000-0000-0000-0000-000000000002"
      },
      "id": "00000000-0000-0000-0000-000000000001",
      "type": "queues"
    }
  ]
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Unauthorized

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Forbidden

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  # Curl command
curl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/annotation-queues" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
"""
List LLM Observability annotation queues returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi

configuration = Configuration()
configuration.unstable_operations["list_llm_obs_annotation_queues"] = True
with ApiClient(configuration) as api_client:
    api_instance = LLMObservabilityApi(api_client)
    response = api_instance.list_llm_obs_annotation_queues()

    print(response)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" python3 "example.py"
# List LLM Observability annotation queues returns "OK" response

require "datadog_api_client"
DatadogAPIClient.configure do |config|
  config.unstable_operations["v2.list_llm_obs_annotation_queues".to_sym] = true
end
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
p api_instance.list_llm_obs_annotation_queues()

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" rb "example.rb"
// List LLM Observability annotation queues returns "OK" response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)

func main() {
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	configuration.SetUnstableOperationEnabled("v2.ListLLMObsAnnotationQueues", true)
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV2.NewLLMObservabilityApi(apiClient)
	resp, r, err := api.ListLLMObsAnnotationQueues(ctx, *datadogV2.NewListLLMObsAnnotationQueuesOptionalParameters())

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LLMObservabilityApi.ListLLMObsAnnotationQueues`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LLMObservabilityApi.ListLLMObsAnnotationQueues`:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" go run "main.go"
// List LLM Observability annotation queues returns "OK" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.LlmObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsAnnotationQueuesResponse;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    defaultClient.setUnstableOperationEnabled("v2.listLLMObsAnnotationQueues", true);
    LlmObservabilityApi apiInstance = new LlmObservabilityApi(defaultClient);

    try {
      LLMObsAnnotationQueuesResponse result = apiInstance.listLLMObsAnnotationQueues();
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LlmObservabilityApi#listLLMObsAnnotationQueues");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" java "Example.java"
// List LLM Observability annotation queues returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_llm_observability::LLMObservabilityAPI;
use datadog_api_client::datadogV2::api_llm_observability::ListLLMObsAnnotationQueuesOptionalParams;

#[tokio::main]
async fn main() {
    let mut configuration = datadog::Configuration::new();
    configuration.set_unstable_operation_enabled("v2.ListLLMObsAnnotationQueues", true);
    let api = LLMObservabilityAPI::with_config(configuration);
    let resp = api
        .list_llm_obs_annotation_queues(ListLLMObsAnnotationQueuesOptionalParams::default())
        .await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run
/**
 * List LLM Observability annotation queues returns "OK" response
 */

import { client, v2 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
configuration.unstableOperations["v2.listLLMObsAnnotationQueues"] = true;
const apiInstance = new v2.LLMObservabilityApi(configuration);

apiInstance
  .listLLMObsAnnotationQueues()
  .then((data: v2.LLMObsAnnotationQueuesResponse) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" tsc "example.ts"