Create an LLM Observability experiment

Note: This endpoint is in preview and is subject to change. If you have any feedback, contact Datadog support.

POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experimentshttps://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experimentshttps://api.datadoghq.eu/api/v2/llm-obs/v1/experimentshttps://api.ddog-gov.com/api/v2/llm-obs/v1/experimentshttps://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experimentshttps://api.datadoghq.com/api/v2/llm-obs/v1/experimentshttps://api.us3.datadoghq.com/api/v2/llm-obs/v1/experimentshttps://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments

Overview

Create a new LLM Observability experiment.

Request

Body Data (required)

Create experiment payload.

Expand All

Field

Type

Description

data [required]

object

Data object for creating an LLM Observability experiment.

attributes [required]

object

Attributes for creating an LLM Observability experiment.

config

object

Configuration parameters for the experiment.

dataset_id

string

Identifier of the dataset used in this experiment.

dataset_version

int64

Version of the dataset to use. Defaults to the current version if not specified.

description

string

Description of the experiment.

ensure_unique

boolean

Whether to ensure the experiment name is unique. Defaults to true.

metadata

object

Arbitrary metadata associated with the experiment.

name [required]

string

Name of the experiment.

parent_experiment_id

string

Identifier of the parent (baseline) experiment this experiment is run against.

project_id [required]

string

Identifier of the project this experiment belongs to.

run_count

int32

Number of runs configured for this experiment.

type [required]

enum

Resource type of an LLM Observability experiment. Allowed enum values: experiments

{
  "data": {
    "attributes": {
      "config": {},
      "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
      "dataset_version": "integer",
      "description": "string",
      "ensure_unique": false,
      "metadata": {},
      "name": "My Experiment v1",
      "parent_experiment_id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012",
      "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
      "run_count": "integer"
    },
    "type": "experiments"
  }
}

Response

OK

Response containing a single LLM Observability experiment.

Expand All

Field

Type

Description

data [required]

object

Data object for an LLM Observability experiment.

attributes [required]

object

Attributes of an LLM Observability experiment.

aggregate_data

object

Pre-computed aggregate metrics for this experiment run, including eval score distributions, token costs, and error rates.

author

object

User data for the author of an experiment. Only present when include[user_data] is true.

email

string

Email address of the user.

handle

string

Username or handle associated with the user's Datadog account.

icon

string

URL of the user's icon.

id

string

Unique identifier of the user.

name

string

Display name of the user.

config [required]

object

Configuration parameters for the experiment.

created_at [required]

date-time

Timestamp when the experiment was created.

dataset_id [required]

string

Identifier of the dataset used in this experiment.

dataset_name

string

Name of the dataset used in this experiment. Only present when include[dataset_names] is true.

dataset_version

int64

Version of the dataset used in this experiment.

deleted_at

date-time

Timestamp when the experiment was soft-deleted, if applicable.

description [required]

string

Description of the experiment.

error

string

Error message describing why the experiment failed, if applicable.

experiment

string

Logical name of the experiment, shared across all runs of the same pipeline.

metadata [required]

object

Arbitrary metadata associated with the experiment.

name [required]

string

Name of the experiment.

parent_experiment_id

string

Identifier of the parent (baseline) experiment this experiment was run against, if any.

project_id [required]

string

Identifier of the project this experiment belongs to.

run_count

int32

Expected number of runs for this experiment.

status

enum

Execution status of an LLM Observability experiment. Allowed enum values: running,completed,failed,interrupted

updated_at [required]

date-time

Timestamp when the experiment was last updated.

id [required]

string

Unique identifier of the experiment.

type [required]

enum

Resource type of an LLM Observability experiment. Allowed enum values: experiments

{
  "data": {
    "attributes": {
      "aggregate_data": {},
      "author": {
        "email": "jane.doe@example.com",
        "handle": "jane.doe@example.com",
        "icon": "https://example.com/icon.png",
        "id": "00000000-0000-0000-0000-000000000010",
        "name": "Jane Doe"
      },
      "config": {},
      "created_at": "2024-01-15T10:30:00Z",
      "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
      "dataset_name": "string",
      "dataset_version": "integer",
      "deleted_at": "2019-09-19T10:00:00.000Z",
      "description": "",
      "error": "string",
      "experiment": "my-pipeline",
      "metadata": {},
      "name": "My Experiment v1",
      "parent_experiment_id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012",
      "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
      "run_count": "integer",
      "status": "completed",
      "updated_at": "2024-01-15T10:30:00Z"
    },
    "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012",
    "type": "experiments"
  }
}

Created

Response containing a single LLM Observability experiment.

Expand All

Field

Type

Description

data [required]

object

Data object for an LLM Observability experiment.

attributes [required]

object

Attributes of an LLM Observability experiment.

aggregate_data

object

Pre-computed aggregate metrics for this experiment run, including eval score distributions, token costs, and error rates.

author

object

User data for the author of an experiment. Only present when include[user_data] is true.

email

string

Email address of the user.

handle

string

Username or handle associated with the user's Datadog account.

icon

string

URL of the user's icon.

id

string

Unique identifier of the user.

name

string

Display name of the user.

config [required]

object

Configuration parameters for the experiment.

created_at [required]

date-time

Timestamp when the experiment was created.

dataset_id [required]

string

Identifier of the dataset used in this experiment.

dataset_name

string

Name of the dataset used in this experiment. Only present when include[dataset_names] is true.

dataset_version

int64

Version of the dataset used in this experiment.

deleted_at

date-time

Timestamp when the experiment was soft-deleted, if applicable.

description [required]

string

Description of the experiment.

error

string

Error message describing why the experiment failed, if applicable.

experiment

string

Logical name of the experiment, shared across all runs of the same pipeline.

metadata [required]

object

Arbitrary metadata associated with the experiment.

name [required]

string

Name of the experiment.

parent_experiment_id

string

Identifier of the parent (baseline) experiment this experiment was run against, if any.

project_id [required]

string

Identifier of the project this experiment belongs to.

run_count

int32

Expected number of runs for this experiment.

status

enum

Execution status of an LLM Observability experiment. Allowed enum values: running,completed,failed,interrupted

updated_at [required]

date-time

Timestamp when the experiment was last updated.

id [required]

string

Unique identifier of the experiment.

type [required]

enum

Resource type of an LLM Observability experiment. Allowed enum values: experiments

{
  "data": {
    "attributes": {
      "aggregate_data": {},
      "author": {
        "email": "jane.doe@example.com",
        "handle": "jane.doe@example.com",
        "icon": "https://example.com/icon.png",
        "id": "00000000-0000-0000-0000-000000000010",
        "name": "Jane Doe"
      },
      "config": {},
      "created_at": "2024-01-15T10:30:00Z",
      "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
      "dataset_name": "string",
      "dataset_version": "integer",
      "deleted_at": "2019-09-19T10:00:00.000Z",
      "description": "",
      "error": "string",
      "experiment": "my-pipeline",
      "metadata": {},
      "name": "My Experiment v1",
      "parent_experiment_id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012",
      "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
      "run_count": "integer",
      "status": "completed",
      "updated_at": "2024-01-15T10:30:00Z"
    },
    "id": "3fd6b5e0-8910-4b1c-a7d0-5b84de329012",
    "type": "experiments"
  }
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Unauthorized

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Forbidden

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  ## default
# 

# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "name": "My Experiment v1", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" }, "type": "experiments" } } EOF
"""
Create an LLM Observability experiment returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_experiment_data_attributes_request import LLMObsExperimentDataAttributesRequest
from datadog_api_client.v2.model.llm_obs_experiment_data_request import LLMObsExperimentDataRequest
from datadog_api_client.v2.model.llm_obs_experiment_request import LLMObsExperimentRequest
from datadog_api_client.v2.model.llm_obs_experiment_type import LLMObsExperimentType

body = LLMObsExperimentRequest(
    data=LLMObsExperimentDataRequest(
        attributes=LLMObsExperimentDataAttributesRequest(
            dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
            name="My Experiment v1",
            project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
        ),
        type=LLMObsExperimentType.EXPERIMENTS,
    ),
)

configuration = Configuration()
configuration.unstable_operations["create_llm_obs_experiment"] = True
with ApiClient(configuration) as api_client:
    api_instance = LLMObservabilityApi(api_client)
    response = api_instance.create_llm_obs_experiment(body=body)

    print(response)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" python3 "example.py"
# Create an LLM Observability experiment returns "OK" response

require "datadog_api_client"
DatadogAPIClient.configure do |config|
  config.unstable_operations["v2.create_llm_obs_experiment".to_sym] = true
end
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new

body = DatadogAPIClient::V2::LLMObsExperimentRequest.new({
  data: DatadogAPIClient::V2::LLMObsExperimentDataRequest.new({
    attributes: DatadogAPIClient::V2::LLMObsExperimentDataAttributesRequest.new({
      dataset_id: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
      name: "My Experiment v1",
      project_id: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
    }),
    type: DatadogAPIClient::V2::LLMObsExperimentType::EXPERIMENTS,
  }),
})
p api_instance.create_llm_obs_experiment(body)

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" rb "example.rb"
// Create an LLM Observability experiment returns "OK" response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)

func main() {
	body := datadogV2.LLMObsExperimentRequest{
		Data: datadogV2.LLMObsExperimentDataRequest{
			Attributes: datadogV2.LLMObsExperimentDataAttributesRequest{
				DatasetId: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
				Name:      "My Experiment v1",
				ProjectId: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
			},
			Type: datadogV2.LLMOBSEXPERIMENTTYPE_EXPERIMENTS,
		},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	configuration.SetUnstableOperationEnabled("v2.CreateLLMObsExperiment", true)
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV2.NewLLMObservabilityApi(apiClient)
	resp, r, err := api.CreateLLMObsExperiment(ctx, body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LLMObservabilityApi.CreateLLMObsExperiment`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LLMObservabilityApi.CreateLLMObsExperiment`:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" go run "main.go"
// Create an LLM Observability experiment returns "OK" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.LlmObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsExperimentDataAttributesRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentDataRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentResponse;
import com.datadog.api.client.v2.model.LLMObsExperimentType;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    defaultClient.setUnstableOperationEnabled("v2.createLLMObsExperiment", true);
    LlmObservabilityApi apiInstance = new LlmObservabilityApi(defaultClient);

    LLMObsExperimentRequest body =
        new LLMObsExperimentRequest()
            .data(
                new LLMObsExperimentDataRequest()
                    .attributes(
                        new LLMObsExperimentDataAttributesRequest()
                            .datasetId("9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d")
                            .name("My Experiment v1")
                            .projectId("a33671aa-24fd-4dcd-9b33-a8ec7dde7751"))
                    .type(LLMObsExperimentType.EXPERIMENTS));

    try {
      LLMObsExperimentResponse result = apiInstance.createLLMObsExperiment(body);
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LlmObservabilityApi#createLLMObsExperiment");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" java "Example.java"
// Create an LLM Observability experiment returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_llm_observability::LLMObservabilityAPI;
use datadog_api_client::datadogV2::model::LLMObsExperimentDataAttributesRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentDataRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentType;

#[tokio::main]
async fn main() {
    let body = LLMObsExperimentRequest::new(LLMObsExperimentDataRequest::new(
        LLMObsExperimentDataAttributesRequest::new(
            "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d".to_string(),
            "My Experiment v1".to_string(),
            "a33671aa-24fd-4dcd-9b33-a8ec7dde7751".to_string(),
        ),
        LLMObsExperimentType::EXPERIMENTS,
    ));
    let mut configuration = datadog::Configuration::new();
    configuration.set_unstable_operation_enabled("v2.CreateLLMObsExperiment", true);
    let api = LLMObservabilityAPI::with_config(configuration);
    let resp = api.create_llm_obs_experiment(body).await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run
/**
 * Create an LLM Observability experiment returns "OK" response
 */

import { client, v2 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
configuration.unstableOperations["v2.createLLMObsExperiment"] = true;
const apiInstance = new v2.LLMObservabilityApi(configuration);

const params: v2.LLMObservabilityApiCreateLLMObsExperimentRequest = {
  body: {
    data: {
      attributes: {
        datasetId: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
        name: "My Experiment v1",
        projectId: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
      },
      type: "experiments",
    },
  },
};

apiInstance
  .createLLMObsExperiment(params)
  .then((data: v2.LLMObsExperimentResponse) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" tsc "example.ts"