Push events for an LLM Observability experiment

Note: This endpoint is in preview and is subject to change. If you have any feedback, contact Datadog support.

POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.datadoghq.eu/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events

Overview

Push spans and metrics for an LLM Observability experiment.

Arguments

Path Parameters

Name

Type

Description

experiment_id [required]

string

The ID of the LLM Observability experiment.

Request

Body Data (required)

Experiment events payload.

Expand All

Field

Type

Description

data [required]

object

Data object for pushing experiment events.

attributes [required]

object

Attributes for pushing experiment events including spans and metrics.

metrics

[object]

List of metrics to push for the experiment.

assessment

enum

Assessment result for an LLM Observability experiment metric. Allowed enum values: pass,fail

boolean_value

boolean

Boolean value. Used when metric_type is boolean.

categorical_value

string

Categorical value. Used when metric_type is categorical.

error

object

Error details for an experiment metric evaluation.

message

string

Error message associated with the metric evaluation.

json_value

object

JSON value. Used when metric_type is json.

label [required]

string

Label or name for the metric.

metadata

object

Arbitrary metadata associated with the metric.

metric_type [required]

enum

Type of metric recorded for an LLM Observability experiment. Allowed enum values: score,categorical,boolean,json

reasoning

string

Human-readable reasoning for the metric value.

score_value

double

Numeric score value. Used when metric_type is score.

span_id [required]

string

The ID of the span this metric measures.

tags

[string]

List of tags associated with the metric.

timestamp_ms [required]

int64

Timestamp when the metric was recorded, in milliseconds since Unix epoch.

spans

[object]

List of spans to push for the experiment.

dataset_id [required]

string

Dataset ID associated with this span.

duration [required]

int64

Duration of the span in nanoseconds.

meta

object

Metadata associated with an experiment span.

error

object

Error details for an experiment span.

message

string

Error message.

stack

string

Stack trace of the error.

type

string

The error type or exception class name.

expected_output

object

Expected output for the span, used for evaluation.

input

object <oneOf>

Represents any valid JSON value.

Option 1

string

A scalar string value.

Option 2

double

A scalar numeric value.

Option 3

object

An arbitrary object value with additional properties.

Option 4

[ <oneOf>]

An array of arbitrary values.

Option 1

string

A scalar string value.

Option 2

double

A scalar numeric value.

Option 3

object

An arbitrary object value with additional properties.

Option 4

boolean

A scalar boolean value.

Option 5

boolean

A scalar boolean value.

output

object <oneOf>

Represents any valid JSON value.

Option 1

string

A scalar string value.

Option 2

double

A scalar numeric value.

Option 3

object

An arbitrary object value with additional properties.

Option 4

[ <oneOf>]

An array of arbitrary values.

Option 1

string

A scalar string value.

Option 2

double

A scalar numeric value.

Option 3

object

An arbitrary object value with additional properties.

Option 4

boolean

A scalar boolean value.

Option 5

boolean

A scalar boolean value.

name [required]

string

Name of the span.

project_id [required]

string

Project ID associated with this span.

span_id [required]

string

Unique identifier of the span.

start_ns [required]

int64

Start time of the span in nanoseconds since Unix epoch.

status [required]

enum

Status of the span. Allowed enum values: ok,error

tags

[string]

List of tags associated with the span.

trace_id [required]

string

Trace ID for the span.

type [required]

enum

Resource type for LLM Observability experiment events. Allowed enum values: events

{
  "data": {
    "attributes": {
      "metrics": [
        {
          "assessment": "pass",
          "boolean_value": false,
          "categorical_value": "string",
          "error": {
            "message": "string"
          },
          "json_value": {},
          "label": "faithfulness",
          "metadata": {},
          "metric_type": "score",
          "reasoning": "string",
          "score_value": "number",
          "span_id": "span-7a1b2c3d",
          "tags": [],
          "timestamp_ms": 1705314600000
        }
      ],
      "spans": [
        {
          "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
          "duration": 1500000000,
          "meta": {
            "error": {
              "message": "Model response timed out",
              "stack": "Traceback (most recent call last):\n  File \"main.py\", line 10, in <module>\n    response = model.generate(input)\n  File \"model.py\", line 45, in generate\n    raise TimeoutError(\"Model response timed out\")\nTimeoutError: Model response timed out",
              "type": "TimeoutError"
            },
            "expected_output": {},
            "input": {
              "description": "undefined",
              "type": "undefined"
            },
            "output": {
              "description": "undefined",
              "type": "undefined"
            }
          },
          "name": "llm_call",
          "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
          "span_id": "span-7a1b2c3d",
          "start_ns": 1705314600000000000,
          "status": "ok",
          "tags": [],
          "trace_id": "abc123def456"
        }
      ]
    },
    "type": "events"
  }
}

Response

Accepted

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Unauthorized

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Forbidden

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Not Found

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  ## default
# 

# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "metrics": [ { "assessment": "pass", "label": "faithfulness", "metric_type": "score", "span_id": "span-7a1b2c3d", "timestamp_ms": 1705314600000 } ], "spans": [ { "dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d", "duration": 1500000000, "name": "llm_call", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751", "span_id": "span-7a1b2c3d", "start_ns": 1705314600000000000, "status": "ok", "trace_id": "abc123def456" } ] }, "type": "events" } } EOF
"""
Push events for an LLM Observability experiment returns "Accepted" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_event_type import LLMObsEventType
from datadog_api_client.v2.model.llm_obs_experiment_events_data_attributes_request import (
    LLMObsExperimentEventsDataAttributesRequest,
)
from datadog_api_client.v2.model.llm_obs_experiment_events_data_request import LLMObsExperimentEventsDataRequest
from datadog_api_client.v2.model.llm_obs_experiment_events_request import LLMObsExperimentEventsRequest
from datadog_api_client.v2.model.llm_obs_experiment_metric import LLMObsExperimentMetric
from datadog_api_client.v2.model.llm_obs_experiment_metric_error import LLMObsExperimentMetricError
from datadog_api_client.v2.model.llm_obs_experiment_span import LLMObsExperimentSpan
from datadog_api_client.v2.model.llm_obs_experiment_span_error import LLMObsExperimentSpanError
from datadog_api_client.v2.model.llm_obs_experiment_span_meta import LLMObsExperimentSpanMeta
from datadog_api_client.v2.model.llm_obs_experiment_span_status import LLMObsExperimentSpanStatus
from datadog_api_client.v2.model.llm_obs_metric_assessment import LLMObsMetricAssessment
from datadog_api_client.v2.model.llm_obs_metric_score_type import LLMObsMetricScoreType

body = LLMObsExperimentEventsRequest(
    data=LLMObsExperimentEventsDataRequest(
        attributes=LLMObsExperimentEventsDataAttributesRequest(
            metrics=[
                LLMObsExperimentMetric(
                    assessment=LLMObsMetricAssessment.PASS,
                    error=LLMObsExperimentMetricError(),
                    label="faithfulness",
                    metric_type=LLMObsMetricScoreType.SCORE,
                    span_id="span-7a1b2c3d",
                    tags=[],
                    timestamp_ms=1705314600000,
                ),
            ],
            spans=[
                LLMObsExperimentSpan(
                    dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
                    duration=1500000000,
                    meta=LLMObsExperimentSpanMeta(
                        error=LLMObsExperimentSpanError(
                            message="Model response timed out",
                            stack='Traceback (most recent call last):\n  File "main.py", line 10, in <module>\n    response = model.generate(input)\n  File "model.py", line 45, in generate\n    raise TimeoutError("Model response timed out")\nTimeoutError: Model response timed out',
                            type="TimeoutError",
                        ),
                        input=None,
                        output=None,
                    ),
                    name="llm_call",
                    project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
                    span_id="span-7a1b2c3d",
                    start_ns=1705314600000000000,
                    status=LLMObsExperimentSpanStatus.OK,
                    tags=[],
                    trace_id="abc123def456",
                ),
            ],
        ),
        type=LLMObsEventType.EVENTS,
    ),
)

configuration = Configuration()
configuration.unstable_operations["create_llm_obs_experiment_events"] = True
with ApiClient(configuration) as api_client:
    api_instance = LLMObservabilityApi(api_client)
    api_instance.create_llm_obs_experiment_events(experiment_id="experiment_id", body=body)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" python3 "example.py"
# Push events for an LLM Observability experiment returns "Accepted" response

require "datadog_api_client"
DatadogAPIClient.configure do |config|
  config.unstable_operations["v2.create_llm_obs_experiment_events".to_sym] = true
end
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new

body = DatadogAPIClient::V2::LLMObsExperimentEventsRequest.new({
  data: DatadogAPIClient::V2::LLMObsExperimentEventsDataRequest.new({
    attributes: DatadogAPIClient::V2::LLMObsExperimentEventsDataAttributesRequest.new({
      metrics: [
        DatadogAPIClient::V2::LLMObsExperimentMetric.new({
          assessment: DatadogAPIClient::V2::LLMObsMetricAssessment::PASS,
          error: DatadogAPIClient::V2::LLMObsExperimentMetricError.new({}),
          label: "faithfulness",
          metric_type: DatadogAPIClient::V2::LLMObsMetricScoreType::SCORE,
          span_id: "span-7a1b2c3d",
          tags: [],
          timestamp_ms: 1705314600000,
        }),
      ],
      spans: [
        DatadogAPIClient::V2::LLMObsExperimentSpan.new({
          dataset_id: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
          duration: 1500000000,
          meta: DatadogAPIClient::V2::LLMObsExperimentSpanMeta.new({
            error: DatadogAPIClient::V2::LLMObsExperimentSpanError.new({
              message: "Model response timed out",
              stack: 'Traceback (most recent call last):\n  File "main.py", line 10, in <module>\n    response = model.generate(input)\n  File "model.py", line 45, in generate\n    raise TimeoutError("Model response timed out")\nTimeoutError: Model response timed out',
              type: "TimeoutError",
            }),
            input: nil,
            output: nil,
          }),
          name: "llm_call",
          project_id: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
          span_id: "span-7a1b2c3d",
          start_ns: 1705314600000000000,
          status: DatadogAPIClient::V2::LLMObsExperimentSpanStatus::OK,
          tags: [],
          trace_id: "abc123def456",
        }),
      ],
    }),
    type: DatadogAPIClient::V2::LLMObsEventType::EVENTS,
  }),
})
p api_instance.create_llm_obs_experiment_events("experiment_id", body)

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" rb "example.rb"
// Push events for an LLM Observability experiment returns "Accepted" response

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)

func main() {
	body := datadogV2.LLMObsExperimentEventsRequest{
		Data: datadogV2.LLMObsExperimentEventsDataRequest{
			Attributes: datadogV2.LLMObsExperimentEventsDataAttributesRequest{
				Metrics: []datadogV2.LLMObsExperimentMetric{
					{
						Assessment:  datadogV2.LLMOBSMETRICASSESSMENT_PASS.Ptr(),
						Error:       &datadogV2.LLMObsExperimentMetricError{},
						Label:       "faithfulness",
						MetricType:  datadogV2.LLMOBSMETRICSCORETYPE_SCORE,
						SpanId:      "span-7a1b2c3d",
						Tags:        []string{},
						TimestampMs: 1705314600000,
					},
				},
				Spans: []datadogV2.LLMObsExperimentSpan{
					{
						DatasetId: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
						Duration:  1500000000,
						Meta: &datadogV2.LLMObsExperimentSpanMeta{
							Error: &datadogV2.LLMObsExperimentSpanError{
								Message: datadog.PtrString("Model response timed out"),
								Stack: datadog.PtrString(`Traceback (most recent call last):
  File "main.py", line 10, in <module>
    response = model.generate(input)
  File "model.py", line 45, in generate
    raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out`),
								Type: datadog.PtrString("TimeoutError"),
							},
							Input:  *datadogV2.NewNullableAnyValue(nil),
							Output: *datadogV2.NewNullableAnyValue(nil),
						},
						Name:      "llm_call",
						ProjectId: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
						SpanId:    "span-7a1b2c3d",
						StartNs:   1705314600000000000,
						Status:    datadogV2.LLMOBSEXPERIMENTSPANSTATUS_OK,
						Tags:      []string{},
						TraceId:   "abc123def456",
					},
				},
			},
			Type: datadogV2.LLMOBSEVENTTYPE_EVENTS,
		},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	configuration.SetUnstableOperationEnabled("v2.CreateLLMObsExperimentEvents", true)
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV2.NewLLMObservabilityApi(apiClient)
	r, err := api.CreateLLMObsExperimentEvents(ctx, "experiment_id", body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LLMObservabilityApi.CreateLLMObsExperimentEvents`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" go run "main.go"
// Push events for an LLM Observability experiment returns "Accepted" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.LlmObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsEventType;
import com.datadog.api.client.v2.model.LLMObsExperimentEventsDataAttributesRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentEventsDataRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentEventsRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentMetric;
import com.datadog.api.client.v2.model.LLMObsExperimentMetricError;
import com.datadog.api.client.v2.model.LLMObsExperimentSpan;
import com.datadog.api.client.v2.model.LLMObsExperimentSpanError;
import com.datadog.api.client.v2.model.LLMObsExperimentSpanMeta;
import com.datadog.api.client.v2.model.LLMObsExperimentSpanStatus;
import com.datadog.api.client.v2.model.LLMObsMetricAssessment;
import com.datadog.api.client.v2.model.LLMObsMetricScoreType;
import java.util.Collections;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    defaultClient.setUnstableOperationEnabled("v2.createLLMObsExperimentEvents", true);
    LlmObservabilityApi apiInstance = new LlmObservabilityApi(defaultClient);

    LLMObsExperimentEventsRequest body =
        new LLMObsExperimentEventsRequest()
            .data(
                new LLMObsExperimentEventsDataRequest()
                    .attributes(
                        new LLMObsExperimentEventsDataAttributesRequest()
                            .metrics(
                                Collections.singletonList(
                                    new LLMObsExperimentMetric()
                                        .assessment(LLMObsMetricAssessment.PASS)
                                        .error(new LLMObsExperimentMetricError())
                                        .label("faithfulness")
                                        .metricType(LLMObsMetricScoreType.SCORE)
                                        .spanId("span-7a1b2c3d")
                                        .timestampMs(1705314600000L)))
                            .spans(
                                Collections.singletonList(
                                    new LLMObsExperimentSpan()
                                        .datasetId("9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d")
                                        .duration(1500000000L)
                                        .meta(
                                            new LLMObsExperimentSpanMeta()
                                                .error(
                                                    new LLMObsExperimentSpanError()
                                                        .message("Model response timed out")
                                                        .stack(
                                                            """
Traceback (most recent call last):
  File "main.py", line 10, in <module>
    response = model.generate(input)
  File "model.py", line 45, in generate
    raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out
""")
                                                        .type("TimeoutError"))
                                                .input(null)
                                                .output(null))
                                        .name("llm_call")
                                        .projectId("a33671aa-24fd-4dcd-9b33-a8ec7dde7751")
                                        .spanId("span-7a1b2c3d")
                                        .startNs(1705314600000000000L)
                                        .status(LLMObsExperimentSpanStatus.OK)
                                        .traceId("abc123def456"))))
                    .type(LLMObsEventType.EVENTS));

    try {
      apiInstance.createLLMObsExperimentEvents("3fd6b5e0-8910-4b1c-a7d0-5b84de329012", body);
    } catch (ApiException e) {
      System.err.println("Exception when calling LlmObservabilityApi#createLLMObsExperimentEvents");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" java "Example.java"
// Push events for an LLM Observability experiment returns "Accepted" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_llm_observability::LLMObservabilityAPI;
use datadog_api_client::datadogV2::model::LLMObsEventType;
use datadog_api_client::datadogV2::model::LLMObsExperimentEventsDataAttributesRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentEventsDataRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentEventsRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentMetric;
use datadog_api_client::datadogV2::model::LLMObsExperimentMetricError;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpan;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpanError;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpanMeta;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpanStatus;
use datadog_api_client::datadogV2::model::LLMObsMetricAssessment;
use datadog_api_client::datadogV2::model::LLMObsMetricScoreType;

#[tokio::main]
async fn main() {
    let body = LLMObsExperimentEventsRequest::new(LLMObsExperimentEventsDataRequest::new(
        LLMObsExperimentEventsDataAttributesRequest::new()
            .metrics(vec![LLMObsExperimentMetric::new(
                "faithfulness".to_string(),
                LLMObsMetricScoreType::SCORE,
                "span-7a1b2c3d".to_string(),
                1705314600000,
            )
            .assessment(LLMObsMetricAssessment::PASS)
            .error(LLMObsExperimentMetricError::new())
            .tags(vec![])])
            .spans(vec![LLMObsExperimentSpan::new(
                "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d".to_string(),
                1500000000,
                "llm_call".to_string(),
                "a33671aa-24fd-4dcd-9b33-a8ec7dde7751".to_string(),
                "span-7a1b2c3d".to_string(),
                1705314600000000000,
                LLMObsExperimentSpanStatus::OK,
                "abc123def456".to_string(),
            )
            .meta(
                LLMObsExperimentSpanMeta::new()
                    .error(
                        LLMObsExperimentSpanError::new()
                            .message("Model response timed out".to_string())
                            .stack(
                                r#"Traceback (most recent call last):
  File "main.py", line 10, in <module>
    response = model.generate(input)
  File "model.py", line 45, in generate
    raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out"#
                                    .to_string(),
                            )
                            .type_("TimeoutError".to_string()),
                    )
                    .input(None)
                    .output(None),
            )
            .tags(vec![])]),
        LLMObsEventType::EVENTS,
    ));
    let mut configuration = datadog::Configuration::new();
    configuration.set_unstable_operation_enabled("v2.CreateLLMObsExperimentEvents", true);
    let api = LLMObservabilityAPI::with_config(configuration);
    let resp = api
        .create_llm_obs_experiment_events("experiment_id".to_string(), body)
        .await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run
/**
 * Push events for an LLM Observability experiment returns "Accepted" response
 */

import { client, v2 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
configuration.unstableOperations["v2.createLLMObsExperimentEvents"] = true;
const apiInstance = new v2.LLMObservabilityApi(configuration);

const params: v2.LLMObservabilityApiCreateLLMObsExperimentEventsRequest = {
  body: {
    data: {
      attributes: {
        metrics: [
          {
            assessment: "pass",
            error: {},
            label: "faithfulness",
            metricType: "score",
            spanId: "span-7a1b2c3d",
            tags: [],
            timestampMs: 1705314600000,
          },
        ],
        spans: [
          {
            datasetId: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
            duration: 1500000000,
            meta: {
              error: {
                message: "Model response timed out",
                stack: `Traceback (most recent call last):
  File "main.py", line 10, in <module>
    response = model.generate(input)
  File "model.py", line 45, in generate
    raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out`,
                type: "TimeoutError",
              },
              input: undefined,
              output: undefined,
            },
            name: "llm_call",
            projectId: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
            spanId: "span-7a1b2c3d",
            startNs: 1705314600000000000,
            status: "ok",
            tags: [],
            traceId: "abc123def456",
          },
        ],
      },
      type: "events",
    },
  },
  experimentId: "experiment_id",
};

apiInstance
  .createLLMObsExperimentEvents(params)
  .then((data: any) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" tsc "example.ts"