Note : This endpoint is in preview and is subject to change.
If you have any feedback, contact Datadog support .
POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments https://api.datadoghq.eu/api/v2/llm-obs/v1/experiments https://api.ddog-gov.com/api/v2/llm-obs/v1/experiments https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments https://api.datadoghq.com/api/v2/llm-obs/v1/experiments https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments
Overview Create a new LLM Observability experiment.
Request Body Data (required) Create experiment payload.
Expand All
Data object for creating an LLM Observability experiment.
Attributes for creating an LLM Observability experiment.
Configuration parameters for the experiment.
Identifier of the dataset used in this experiment.
Version of the dataset to use. Defaults to the current version if not specified.
Description of the experiment.
Whether to ensure the experiment name is unique. Defaults to true.
Arbitrary metadata associated with the experiment.
Identifier of the parent (baseline) experiment this experiment is run against.
Identifier of the project this experiment belongs to.
Number of runs configured for this experiment.
Resource type of an LLM Observability experiment.
Allowed enum values: experiments
{
"data" : {
"attributes" : {
"config" : {},
"dataset_id" : "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ,
"dataset_version" : "integer" ,
"description" : "string" ,
"ensure_unique" : false ,
"metadata" : {},
"name" : "My Experiment v1" ,
"parent_experiment_id" : "3fd6b5e0-8910-4b1c-a7d0-5b84de329012" ,
"project_id" : "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ,
"run_count" : "integer"
},
"type" : "experiments"
}
} Response OK
Response containing a single LLM Observability experiment.
Expand All
Data object for an LLM Observability experiment.
Attributes of an LLM Observability experiment.
Pre-computed aggregate metrics for this experiment run, including eval score distributions, token costs, and error rates.
User data for the author of an experiment. Only present when include[user_data] is true.
Email address of the user.
Username or handle associated with the user's Datadog account.
Unique identifier of the user.
Display name of the user.
Configuration parameters for the experiment.
Timestamp when the experiment was created.
Identifier of the dataset used in this experiment.
Name of the dataset used in this experiment.
Only present when include[dataset_names] is true.
Version of the dataset used in this experiment.
Timestamp when the experiment was soft-deleted, if applicable.
Description of the experiment.
Error message describing why the experiment failed, if applicable.
Logical name of the experiment, shared across all runs of the same pipeline.
Arbitrary metadata associated with the experiment.
Identifier of the parent (baseline) experiment this experiment was run against, if any.
Identifier of the project this experiment belongs to.
Expected number of runs for this experiment.
Execution status of an LLM Observability experiment.
Allowed enum values: running,completed,failed,interrupted
Timestamp when the experiment was last updated.
Unique identifier of the experiment.
Resource type of an LLM Observability experiment.
Allowed enum values: experiments
{
"data" : {
"attributes" : {
"aggregate_data" : {},
"author" : {
"email" : "jane.doe@example.com" ,
"handle" : "jane.doe@example.com" ,
"icon" : "https://example.com/icon.png" ,
"id" : "00000000-0000-0000-0000-000000000010" ,
"name" : "Jane Doe"
},
"config" : {},
"created_at" : "2024-01-15T10:30:00Z" ,
"dataset_id" : "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ,
"dataset_name" : "string" ,
"dataset_version" : "integer" ,
"deleted_at" : "2019-09-19T10:00:00.000Z" ,
"description" : "" ,
"error" : "string" ,
"experiment" : "my-pipeline" ,
"metadata" : {},
"name" : "My Experiment v1" ,
"parent_experiment_id" : "3fd6b5e0-8910-4b1c-a7d0-5b84de329012" ,
"project_id" : "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ,
"run_count" : "integer" ,
"status" : "completed" ,
"updated_at" : "2024-01-15T10:30:00Z"
},
"id" : "3fd6b5e0-8910-4b1c-a7d0-5b84de329012" ,
"type" : "experiments"
}
} Created
Response containing a single LLM Observability experiment.
Expand All
Data object for an LLM Observability experiment.
Attributes of an LLM Observability experiment.
Pre-computed aggregate metrics for this experiment run, including eval score distributions, token costs, and error rates.
User data for the author of an experiment. Only present when include[user_data] is true.
Email address of the user.
Username or handle associated with the user's Datadog account.
Unique identifier of the user.
Display name of the user.
Configuration parameters for the experiment.
Timestamp when the experiment was created.
Identifier of the dataset used in this experiment.
Name of the dataset used in this experiment.
Only present when include[dataset_names] is true.
Version of the dataset used in this experiment.
Timestamp when the experiment was soft-deleted, if applicable.
Description of the experiment.
Error message describing why the experiment failed, if applicable.
Logical name of the experiment, shared across all runs of the same pipeline.
Arbitrary metadata associated with the experiment.
Identifier of the parent (baseline) experiment this experiment was run against, if any.
Identifier of the project this experiment belongs to.
Expected number of runs for this experiment.
Execution status of an LLM Observability experiment.
Allowed enum values: running,completed,failed,interrupted
Timestamp when the experiment was last updated.
Unique identifier of the experiment.
Resource type of an LLM Observability experiment.
Allowed enum values: experiments
{
"data" : {
"attributes" : {
"aggregate_data" : {},
"author" : {
"email" : "jane.doe@example.com" ,
"handle" : "jane.doe@example.com" ,
"icon" : "https://example.com/icon.png" ,
"id" : "00000000-0000-0000-0000-000000000010" ,
"name" : "Jane Doe"
},
"config" : {},
"created_at" : "2024-01-15T10:30:00Z" ,
"dataset_id" : "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ,
"dataset_name" : "string" ,
"dataset_version" : "integer" ,
"deleted_at" : "2019-09-19T10:00:00.000Z" ,
"description" : "" ,
"error" : "string" ,
"experiment" : "my-pipeline" ,
"metadata" : {},
"name" : "My Experiment v1" ,
"parent_experiment_id" : "3fd6b5e0-8910-4b1c-a7d0-5b84de329012" ,
"project_id" : "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ,
"run_count" : "integer" ,
"status" : "completed" ,
"updated_at" : "2024-01-15T10:30:00Z"
},
"id" : "3fd6b5e0-8910-4b1c-a7d0-5b84de329012" ,
"type" : "experiments"
}
} Bad Request
API error response.
Expand All
A human-readable explanation specific to this occurrence of the error.
Non-standard meta-information about the error
References to the source of the error.
A string indicating the name of a single request header which caused the error.
A string indicating which URI query parameter caused the error.
A JSON pointer to the value in the request document that caused the error.
Status code of the response.
Short human-readable summary of the error.
{
"errors" : [
{
"detail" : "Missing required attribute in body" ,
"meta" : {},
"source" : {
"header" : "Authorization" ,
"parameter" : "limit" ,
"pointer" : "/data/attributes/title"
},
"status" : "400" ,
"title" : "Bad Request"
}
]
} Unauthorized
API error response.
Expand All
A human-readable explanation specific to this occurrence of the error.
Non-standard meta-information about the error
References to the source of the error.
A string indicating the name of a single request header which caused the error.
A string indicating which URI query parameter caused the error.
A JSON pointer to the value in the request document that caused the error.
Status code of the response.
Short human-readable summary of the error.
{
"errors" : [
{
"detail" : "Missing required attribute in body" ,
"meta" : {},
"source" : {
"header" : "Authorization" ,
"parameter" : "limit" ,
"pointer" : "/data/attributes/title"
},
"status" : "400" ,
"title" : "Bad Request"
}
]
} Forbidden
API error response.
Expand All
A human-readable explanation specific to this occurrence of the error.
Non-standard meta-information about the error
References to the source of the error.
A string indicating the name of a single request header which caused the error.
A string indicating which URI query parameter caused the error.
A JSON pointer to the value in the request document that caused the error.
Status code of the response.
Short human-readable summary of the error.
{
"errors" : [
{
"detail" : "Missing required attribute in body" ,
"meta" : {},
"source" : {
"header" : "Authorization" ,
"parameter" : "limit" ,
"pointer" : "/data/attributes/title"
},
"status" : "400" ,
"title" : "Bad Request"
}
]
} Too many requests
{
"errors" : [
"Bad Request"
]
} Code Example Copy
## default
#
# Curl command curl -X POST "https://api.ap1.datadoghq.com "https://api.ap2.datadoghq.com "https://api.datadoghq.eu "https://api.ddog-gov.com "https://api.us2.ddog-gov.com "https://api.datadoghq.com "https://api.us3.datadoghq.com "https://api.us5.datadoghq.com /api/v2/llm-obs/v1/experiments " \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY} " \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY} " \
-d @- << EOF
{
"data": {
"attributes": {
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"name": "My Experiment v1",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751"
},
"type": "experiments"
}
}
EOF
"""
Create an LLM Observability experiment returns "OK" response
"""
from datadog_api_client import ApiClient , Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_experiment_data_attributes_request import LLMObsExperimentDataAttributesRequest
from datadog_api_client.v2.model.llm_obs_experiment_data_request import LLMObsExperimentDataRequest
from datadog_api_client.v2.model.llm_obs_experiment_request import LLMObsExperimentRequest
from datadog_api_client.v2.model.llm_obs_experiment_type import LLMObsExperimentType
body = LLMObsExperimentRequest (
data = LLMObsExperimentDataRequest (
attributes = LLMObsExperimentDataAttributesRequest (
dataset_id = "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ,
name = "My Experiment v1" ,
project_id = "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ,
),
type = LLMObsExperimentType . EXPERIMENTS ,
),
)
configuration = Configuration ()
configuration . unstable_operations [ "create_llm_obs_experiment" ] = True
with ApiClient ( configuration ) as api_client :
api_instance = LLMObservabilityApi ( api_client )
response = api_instance . create_llm_obs_experiment ( body = body )
print ( response )
Instructions First install the library and its dependencies and then save the example to example.py and run following commands:
DD_SITE = "datadoghq.com us3.datadoghq.com us5.datadoghq.com datadoghq.eu ap1.datadoghq.com ap2.datadoghq.com ddog-gov.com us2.ddog-gov.com " DD_API_KEY = "<DD_API_KEY>" DD_APP_KEY = "<DD_APP_KEY>" python3 "example.py"
# Create an LLM Observability experiment returns "OK" response
require "datadog_api_client"
DatadogAPIClient . configure do | config |
config . unstable_operations [ "v2.create_llm_obs_experiment" . to_sym ] = true
end
api_instance = DatadogAPIClient :: V2 :: LLMObservabilityAPI . new
body = DatadogAPIClient :: V2 :: LLMObsExperimentRequest . new ({
data : DatadogAPIClient :: V2 :: LLMObsExperimentDataRequest . new ({
attributes : DatadogAPIClient :: V2 :: LLMObsExperimentDataAttributesRequest . new ({
dataset_id : "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ,
name : "My Experiment v1" ,
project_id : "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ,
}),
type : DatadogAPIClient :: V2 :: LLMObsExperimentType :: EXPERIMENTS ,
}),
})
p api_instance . create_llm_obs_experiment ( body )
Instructions First install the library and its dependencies and then save the example to example.rb and run following commands:
DD_SITE = "datadoghq.com us3.datadoghq.com us5.datadoghq.com datadoghq.eu ap1.datadoghq.com ap2.datadoghq.com ddog-gov.com us2.ddog-gov.com " DD_API_KEY = "<DD_API_KEY>" DD_APP_KEY = "<DD_APP_KEY>" rb "example.rb"
// Create an LLM Observability experiment returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)
func main () {
body := datadogV2 . LLMObsExperimentRequest {
Data : datadogV2 . LLMObsExperimentDataRequest {
Attributes : datadogV2 . LLMObsExperimentDataAttributesRequest {
DatasetId : "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ,
Name : "My Experiment v1" ,
ProjectId : "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ,
},
Type : datadogV2 . LLMOBSEXPERIMENTTYPE_EXPERIMENTS ,
},
}
ctx := datadog . NewDefaultContext ( context . Background ())
configuration := datadog . NewConfiguration ()
configuration . SetUnstableOperationEnabled ( "v2.CreateLLMObsExperiment" , true )
apiClient := datadog . NewAPIClient ( configuration )
api := datadogV2 . NewLLMObservabilityApi ( apiClient )
resp , r , err := api . CreateLLMObsExperiment ( ctx , body )
if err != nil {
fmt . Fprintf ( os . Stderr , "Error when calling `LLMObservabilityApi.CreateLLMObsExperiment`: %v\n" , err )
fmt . Fprintf ( os . Stderr , "Full HTTP response: %v\n" , r )
}
responseContent , _ := json . MarshalIndent ( resp , "" , " " )
fmt . Fprintf ( os . Stdout , "Response from `LLMObservabilityApi.CreateLLMObsExperiment`:\n%s\n" , responseContent )
}
Instructions First install the library and its dependencies and then save the example to main.go and run following commands:
DD_SITE = "datadoghq.com us3.datadoghq.com us5.datadoghq.com datadoghq.eu ap1.datadoghq.com ap2.datadoghq.com ddog-gov.com us2.ddog-gov.com " DD_API_KEY = "<DD_API_KEY>" DD_APP_KEY = "<DD_APP_KEY>" go run "main.go"
// Create an LLM Observability experiment returns "OK" response
import com.datadog.api.client.ApiClient ;
import com.datadog.api.client.ApiException ;
import com.datadog.api.client.v2.api.LlmObservabilityApi ;
import com.datadog.api.client.v2.model.LLMObsExperimentDataAttributesRequest ;
import com.datadog.api.client.v2.model.LLMObsExperimentDataRequest ;
import com.datadog.api.client.v2.model.LLMObsExperimentRequest ;
import com.datadog.api.client.v2.model.LLMObsExperimentResponse ;
import com.datadog.api.client.v2.model.LLMObsExperimentType ;
public class Example {
public static void main ( String [] args ) {
ApiClient defaultClient = ApiClient . getDefaultApiClient ();
defaultClient . setUnstableOperationEnabled ( "v2.createLLMObsExperiment" , true );
LlmObservabilityApi apiInstance = new LlmObservabilityApi ( defaultClient );
LLMObsExperimentRequest body =
new LLMObsExperimentRequest ()
. data (
new LLMObsExperimentDataRequest ()
. attributes (
new LLMObsExperimentDataAttributesRequest ()
. datasetId ( "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" )
. name ( "My Experiment v1" )
. projectId ( "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ))
. type ( LLMObsExperimentType . EXPERIMENTS ));
try {
LLMObsExperimentResponse result = apiInstance . createLLMObsExperiment ( body );
System . out . println ( result );
} catch ( ApiException e ) {
System . err . println ( "Exception when calling LlmObservabilityApi#createLLMObsExperiment" );
System . err . println ( "Status code: " + e . getCode ());
System . err . println ( "Reason: " + e . getResponseBody ());
System . err . println ( "Response headers: " + e . getResponseHeaders ());
e . printStackTrace ();
}
}
}
Instructions First install the library and its dependencies and then save the example to Example.java and run following commands:
DD_SITE = "datadoghq.com us3.datadoghq.com us5.datadoghq.com datadoghq.eu ap1.datadoghq.com ap2.datadoghq.com ddog-gov.com us2.ddog-gov.com " DD_API_KEY = "<DD_API_KEY>" DD_APP_KEY = "<DD_APP_KEY>" java "Example.java"
// Create an LLM Observability experiment returns "OK" response
use datadog_api_client ::datadog ;
use datadog_api_client ::datadogV2 ::api_llm_observability ::LLMObservabilityAPI ;
use datadog_api_client ::datadogV2 ::model ::LLMObsExperimentDataAttributesRequest ;
use datadog_api_client ::datadogV2 ::model ::LLMObsExperimentDataRequest ;
use datadog_api_client ::datadogV2 ::model ::LLMObsExperimentRequest ;
use datadog_api_client ::datadogV2 ::model ::LLMObsExperimentType ;
#[tokio::main]
async fn main () {
let body = LLMObsExperimentRequest ::new ( LLMObsExperimentDataRequest ::new (
LLMObsExperimentDataAttributesRequest ::new (
"9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" . to_string (),
"My Experiment v1" . to_string (),
"a33671aa-24fd-4dcd-9b33-a8ec7dde7751" . to_string (),
),
LLMObsExperimentType ::EXPERIMENTS ,
));
let mut configuration = datadog ::Configuration ::new ();
configuration . set_unstable_operation_enabled ( "v2.CreateLLMObsExperiment" , true );
let api = LLMObservabilityAPI ::with_config ( configuration );
let resp = api . create_llm_obs_experiment ( body ). await ;
if let Ok ( value ) = resp {
println! ( " {:#?} " , value );
} else {
println! ( " {:#?} " , resp . unwrap_err ());
}
}
Instructions First install the library and its dependencies and then save the example to src/main.rs and run following commands:
DD_SITE = "datadoghq.com us3.datadoghq.com us5.datadoghq.com datadoghq.eu ap1.datadoghq.com ap2.datadoghq.com ddog-gov.com us2.ddog-gov.com " DD_API_KEY = "<DD_API_KEY>" DD_APP_KEY = "<DD_APP_KEY>" cargo run
/**
* Create an LLM Observability experiment returns "OK" response
*/
import { client , v2 } from "@datadog/datadog-api-client" ;
const configuration = client . createConfiguration ();
configuration . unstableOperations [ "v2.createLLMObsExperiment" ] = true ;
const apiInstance = new v2 . LLMObservabilityApi ( configuration );
const params : v2.LLMObservabilityApiCreateLLMObsExperimentRequest = {
body : {
data : {
attributes : {
datasetId : "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d" ,
name : "My Experiment v1" ,
projectId : "a33671aa-24fd-4dcd-9b33-a8ec7dde7751" ,
},
type : "experiments" ,
},
},
};
apiInstance
. createLLMObsExperiment ( params )
. then (( data : v2.LLMObsExperimentResponse ) => {
console . log (
"API called successfully. Returned data: " + JSON . stringify ( data )
);
})
. catch (( error : any ) => console . error ( error ));
Instructions First install the library and its dependencies and then save the example to example.ts and run following commands:
DD_SITE = "datadoghq.com us3.datadoghq.com us5.datadoghq.com datadoghq.eu ap1.datadoghq.com ap2.datadoghq.com ddog-gov.com us2.ddog-gov.com " DD_API_KEY = "<DD_API_KEY>" DD_APP_KEY = "<DD_APP_KEY>" tsc "example.ts"