Note : This endpoint is in preview and is subject to change.
If you have any feedback, contact Datadog support .
POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics https://api.datadoghq.eu/api/v2/llm-obs/v1/experimentation/analytics https://api.ddog-gov.com/api/v2/llm-obs/v1/experimentation/analytics https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experimentation/analytics https://api.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics https://api.us3.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experimentation/analytics
Overview
Execute an analytics aggregation over LLM Observability experimentation data.
Use this endpoint to compute metrics (for example average eval scores) grouped by fields such as span_id or experiment_id.
At least one compute definition and one index must be provided.
Request Body Data (required) Analytics payload.
Expand All
Data object for an analytics request.
Attributes for an analytics request.
Analytics aggregation parameters.
List of metric computations to perform.
Name of the metric to compute.
Optional alias for this computation in the response.
Filter to a specific dataset version.
Fields to group results by.
Data indexes to query. At least one is required.
Maximum number of results to return.
Search query for filtering analytics data.
Unix-millisecond time range for filtering analytics data.
Start of the time range in milliseconds since Unix epoch.
End of the time range in milliseconds since Unix epoch.
Resource type for experimentation search and analytics operations.
Allowed enum values: experimentation
{
"data" : {
"attributes" : {
"aggregate" : {
"compute" : [
{
"metric" : "score_value" ,
"name" : "avg_faithfulness"
}
],
"dataset_version" : "integer" ,
"group_by" : [
{
"field" : "span_id"
}
],
"indexes" : [
"experiment-evals"
],
"limit" : 1000 ,
"search" : {
"query" : "@experiment_id:3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
},
"time" : {
"from" : 1705312200000 ,
"to" : 1705315800000
}
}
},
"type" : "experimentation"
}
} Response OK
Response to an analytics query.
Expand All
JSON:API data object for an analytics response.
Attributes of an analytics response.
Total number of events matched by the query before grouping.
Analytics query result containing all buckets.
The group-by field values for this bucket.
Computed metric values for this bucket.
Server-generated identifier for this analytics result.
Resource type for experimentation search and analytics operations.
Allowed enum values: experimentation
{
"data" : {
"attributes" : {
"hit_count" : 1500 ,
"result" : {
"values" : [
{
"by" : {
"span_id" : "span-7a1b2c3d"
},
"metrics" : {
"score_value" : 0.85
}
}
]
}
},
"id" : "00000000-0000-0000-0000-000000000001" ,
"type" : "experimentation"
}
} Bad Request
API error response.
Expand All
A human-readable explanation specific to this occurrence of the error.
Non-standard meta-information about the error
References to the source of the error.
A string indicating the name of a single request header which caused the error.
A string indicating which URI query parameter caused the error.
A JSON pointer to the value in the request document that caused the error.
Status code of the response.
Short human-readable summary of the error.
{
"errors" : [
{
"detail" : "Missing required attribute in body" ,
"meta" : {},
"source" : {
"header" : "Authorization" ,
"parameter" : "limit" ,
"pointer" : "/data/attributes/title"
},
"status" : "400" ,
"title" : "Bad Request"
}
]
} Unauthorized
API error response.
Expand All
A human-readable explanation specific to this occurrence of the error.
Non-standard meta-information about the error
References to the source of the error.
A string indicating the name of a single request header which caused the error.
A string indicating which URI query parameter caused the error.
A JSON pointer to the value in the request document that caused the error.
Status code of the response.
Short human-readable summary of the error.
{
"errors" : [
{
"detail" : "Missing required attribute in body" ,
"meta" : {},
"source" : {
"header" : "Authorization" ,
"parameter" : "limit" ,
"pointer" : "/data/attributes/title"
},
"status" : "400" ,
"title" : "Bad Request"
}
]
} Forbidden
API error response.
Expand All
A human-readable explanation specific to this occurrence of the error.
Non-standard meta-information about the error
References to the source of the error.
A string indicating the name of a single request header which caused the error.
A string indicating which URI query parameter caused the error.
A JSON pointer to the value in the request document that caused the error.
Status code of the response.
Short human-readable summary of the error.
{
"errors" : [
{
"detail" : "Missing required attribute in body" ,
"meta" : {},
"source" : {
"header" : "Authorization" ,
"parameter" : "limit" ,
"pointer" : "/data/attributes/title"
},
"status" : "400" ,
"title" : "Bad Request"
}
]
} Too many requests
{
"errors" : [
"Bad Request"
]
} Internal Server Error
API error response.
Expand All
A human-readable explanation specific to this occurrence of the error.
Non-standard meta-information about the error
References to the source of the error.
A string indicating the name of a single request header which caused the error.
A string indicating which URI query parameter caused the error.
A JSON pointer to the value in the request document that caused the error.
Status code of the response.
Short human-readable summary of the error.
{
"errors" : [
{
"detail" : "Missing required attribute in body" ,
"meta" : {},
"source" : {
"header" : "Authorization" ,
"parameter" : "limit" ,
"pointer" : "/data/attributes/title"
},
"status" : "400" ,
"title" : "Bad Request"
}
]
} Code Example Copy
## default
#
# Curl command curl -X POST "https://api.ap1.datadoghq.com "https://api.ap2.datadoghq.com "https://api.datadoghq.eu "https://api.ddog-gov.com "https://api.us2.ddog-gov.com "https://api.datadoghq.com "https://api.us3.datadoghq.com "https://api.us5.datadoghq.com /api/v2/llm-obs/v1/experimentation/analytics " \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY} " \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY} " \
-d @- << EOF
{
"data": {
"attributes": {
"aggregate": {
"compute": [
{
"metric": "score_value",
"name": "avg_faithfulness"
}
],
"group_by": [
{
"field": "span_id"
}
],
"indexes": [
"experiment-evals"
],
"search": {
"query": "@experiment_id:3fd6b5e0-8910-4b1c-a7d0-5b84de329012 @label:faithfulness"
}
}
},
"type": "experimentation"
}
}
EOF
