Get a custom evaluator configuration

Note: This endpoint is in preview and is subject to change. If you have any feedback, contact Datadog support.

GET https://api.ap1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.ap2.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.datadoghq.eu/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.us2.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.us3.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}

Overview

Retrieve a custom LLM Observability evaluator configuration by its name.

Arguments

Path Parameters

Name

Type

Description

eval_name [required]

string

The name of the custom LLM Observability evaluator configuration.

Response

OK

Response containing a custom LLM Observability evaluator configuration.

Expand All

Field

Type

Description

data [required]

object

Data object for a custom LLM Observability evaluator configuration.

attributes [required]

object

Attributes of a custom LLM Observability evaluator configuration.

category

string

Category of the evaluator.

created_at [required]

date-time

Timestamp when the evaluator configuration was created.

created_by

object

A Datadog user associated with a custom evaluator configuration.

email

string

Email address of the user.

eval_name [required]

string

Name of the custom evaluator.

last_updated_by

object

A Datadog user associated with a custom evaluator configuration.

email

string

Email address of the user.

llm_judge_config

object

LLM judge configuration for a custom evaluator.

assessment_criteria

object

Criteria used to assess the pass/fail result of a custom evaluator.

max_threshold

double

Maximum numeric threshold for a passing result.

min_threshold

double

Minimum numeric threshold for a passing result.

pass_values

[string]

Specific output values considered as a passing result.

pass_when

boolean

When true, a boolean output of true is treated as passing.

inference_params [required]

object

LLM inference parameters for a custom evaluator.

frequency_penalty

double

Frequency penalty to reduce repetition.

max_tokens

int64

Maximum number of tokens to generate.

presence_penalty

double

Presence penalty to reduce repetition.

temperature

double

Sampling temperature for the LLM.

top_k

int64

Top-k sampling parameter.

top_p

double

Top-p (nucleus) sampling parameter.

last_used_library_prompt_template_name

string

Name of the last library prompt template used.

modified_library_prompt_template

boolean

Whether the library prompt template was modified.

output_schema

object

JSON schema describing the expected output format of the LLM judge.

parsing_type

enum

Output parsing type for a custom LLM judge evaluator. Allowed enum values: structured_output,json

prompt_template

[object]

List of messages forming the LLM judge prompt template.

content

string

Text content of the message.

contents

[object]

Multi-part content blocks for the message.

type [required]

string

Content block type.

value [required]

object

Value of a prompt message content block.

text

string

Text content of the message block.

tool_call

object

A tool call within a prompt message.

arguments

string

JSON-encoded arguments for the tool call.

id

string

Unique identifier of the tool call.

name

string

Name of the tool being called.

type

string

Type of the tool call.

tool_call_result

object

A tool call result within a prompt message.

name

string

Name of the tool that produced this result.

result

string

The result returned by the tool.

tool_id

string

Identifier of the tool call this result corresponds to.

type

string

Type of the tool result.

role [required]

string

Role of the message author.

llm_provider

object

LLM provider configuration for a custom evaluator.

bedrock

object

AWS Bedrock-specific options for LLM provider configuration.

region

string

AWS region for Bedrock.

integration_account_id

string

Integration account identifier.

integration_provider

enum

Name of the LLM integration provider. Allowed enum values: openai,amazon-bedrock,anthropic,azure-openai,vertex-ai,llm-proxy

model_name

string

Name of the LLM model.

vertex_ai

object

Google Vertex AI-specific options for LLM provider configuration.

location

string

Google Cloud region.

project

string

Google Cloud project ID.

target

object

Target application configuration for a custom evaluator.

application_name [required]

string

Name of the ML application this evaluator targets.

enabled [required]

boolean

Whether the evaluator is active for the target application.

eval_scope

enum

Scope at which to evaluate spans. Allowed enum values: span,trace,session

filter

string

Filter expression to select which spans to evaluate.

root_spans_only

boolean

When true, only root spans are evaluated.

sampling_percentage

double

Percentage of traces to evaluate. Must be greater than 0 and at most 100.

updated_at [required]

date-time

Timestamp when the evaluator configuration was last updated.

id [required]

string

Unique name identifier of the evaluator configuration.

type [required]

enum

Type of the custom LLM Observability evaluator configuration resource. Allowed enum values: evaluator_config

{
  "data": {
    "attributes": {
      "category": "Custom",
      "created_at": "2024-01-15T10:30:00Z",
      "created_by": {
        "email": "user@example.com"
      },
      "eval_name": "my-custom-evaluator",
      "last_updated_by": {
        "email": "user@example.com"
      },
      "llm_judge_config": {
        "assessment_criteria": {
          "max_threshold": 1,
          "min_threshold": 0.7,
          "pass_values": [
            "pass",
            "yes"
          ],
          "pass_when": true
        },
        "inference_params": {
          "frequency_penalty": 0,
          "max_tokens": 1024,
          "presence_penalty": 0,
          "temperature": 0.7,
          "top_k": 50,
          "top_p": 1
        },
        "last_used_library_prompt_template_name": "sentiment-analysis-v1",
        "modified_library_prompt_template": false,
        "output_schema": {},
        "parsing_type": "structured_output",
        "prompt_template": [
          {
            "content": "Rate the quality of the following response:",
            "contents": [
              {
                "type": "text",
                "value": {
                  "text": "What is the sentiment of this review?",
                  "tool_call": {
                    "arguments": "{\"location\": \"San Francisco\"}",
                    "id": "call_abc123",
                    "name": "get_weather",
                    "type": "function"
                  },
                  "tool_call_result": {
                    "name": "get_weather",
                    "result": "sunny, 72F",
                    "tool_id": "call_abc123",
                    "type": "function"
                  }
                }
              }
            ],
            "role": "user"
          }
        ]
      },
      "llm_provider": {
        "bedrock": {
          "region": "us-east-1"
        },
        "integration_account_id": "my-account-id",
        "integration_provider": "openai",
        "model_name": "gpt-4o",
        "vertex_ai": {
          "location": "us-central1",
          "project": "my-gcp-project"
        }
      },
      "target": {
        "application_name": "my-llm-app",
        "enabled": true,
        "eval_scope": "span",
        "filter": "@service:my-service",
        "root_spans_only": true,
        "sampling_percentage": 50
      },
      "updated_at": "2024-01-15T10:30:00Z"
    },
    "id": "my-custom-evaluator",
    "type": "evaluator_config"
  }
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Unauthorized

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Forbidden

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Not Found

API error response.

Expand All

Field

Type

Description

errors [required]

[object]

A list of errors.

detail

string

A human-readable explanation specific to this occurrence of the error.

meta

object

Non-standard meta-information about the error

source

object

References to the source of the error.

header

string

A string indicating the name of a single request header which caused the error.

parameter

string

A string indicating which URI query parameter caused the error.

pointer

string

A JSON pointer to the value in the request document that caused the error.

status

string

Status code of the response.

title

string

Short human-readable summary of the error.

{
  "errors": [
    {
      "detail": "Missing required attribute in body",
      "meta": {},
      "source": {
        "header": "Authorization",
        "parameter": "limit",
        "pointer": "/data/attributes/title"
      },
      "status": "400",
      "title": "Bad Request"
    }
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  # Path parameters
export eval_name="my-custom-evaluator"
# Curl command
curl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
"""
Get a custom evaluator configuration returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi

configuration = Configuration()
configuration.unstable_operations["get_llm_obs_custom_eval_config"] = True
with ApiClient(configuration) as api_client:
    api_instance = LLMObservabilityApi(api_client)
    response = api_instance.get_llm_obs_custom_eval_config(
        eval_name="eval_name",
    )

    print(response)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" python3 "example.py"
# Get a custom evaluator configuration returns "OK" response

require "datadog_api_client"
DatadogAPIClient.configure do |config|
  config.unstable_operations["v2.get_llm_obs_custom_eval_config".to_sym] = true
end
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
p api_instance.get_llm_obs_custom_eval_config("eval_name")

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" rb "example.rb"
// Get a custom evaluator configuration returns "OK" response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)

func main() {
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	configuration.SetUnstableOperationEnabled("v2.GetLLMObsCustomEvalConfig", true)
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV2.NewLLMObservabilityApi(apiClient)
	resp, r, err := api.GetLLMObsCustomEvalConfig(ctx, "eval_name")

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LLMObservabilityApi.GetLLMObsCustomEvalConfig`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LLMObservabilityApi.GetLLMObsCustomEvalConfig`:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" go run "main.go"
// Get a custom evaluator configuration returns "OK" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.LlmObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigResponse;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    defaultClient.setUnstableOperationEnabled("v2.getLLMObsCustomEvalConfig", true);
    LlmObservabilityApi apiInstance = new LlmObservabilityApi(defaultClient);

    try {
      LLMObsCustomEvalConfigResponse result =
          apiInstance.getLLMObsCustomEvalConfig("my-custom-evaluator");
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LlmObservabilityApi#getLLMObsCustomEvalConfig");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" java "Example.java"
// Get a custom evaluator configuration returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_llm_observability::LLMObservabilityAPI;

#[tokio::main]
async fn main() {
    let mut configuration = datadog::Configuration::new();
    configuration.set_unstable_operation_enabled("v2.GetLLMObsCustomEvalConfig", true);
    let api = LLMObservabilityAPI::with_config(configuration);
    let resp = api
        .get_llm_obs_custom_eval_config("eval_name".to_string())
        .await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run
/**
 * Get a custom evaluator configuration returns "OK" response
 */

import { client, v2 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
configuration.unstableOperations["v2.getLLMObsCustomEvalConfig"] = true;
const apiInstance = new v2.LLMObservabilityApi(configuration);

const params: v2.LLMObservabilityApiGetLLMObsCustomEvalConfigRequest = {
  evalName: "eval_name",
};

apiInstance
  .getLLMObsCustomEvalConfig(params)
  .then((data: v2.LLMObsCustomEvalConfigResponse) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" tsc "example.ts"