--- title: List custom evaluator configurations description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > API Reference > LLM Observability --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # List custom evaluator configurations{% #list-custom-evaluator-configurations %} Copy pageCopied {% tab title="v2" %} **Note**: This endpoint is in preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/unstable/llm-obs/config/evaluators/custom | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom | | us2.ddog-gov.com | GET https://api.us2.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom | | uk1.datadoghq.com | GET https://api.uk1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom | | app.datadoghq.com | GET https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom | ### Overview List all custom LLM Observability evaluator configurations for the organization. ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing a list of custom LLM Observability evaluator configurations. | Parent field | Field | Type | Description | | ------------------- | -------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | [object] | List of custom evaluator configuration data objects. | | data | attributes [*required*] | object | Attributes of a custom LLM Observability evaluator configuration. | | attributes | category | string | Category of the evaluator. | | attributes | created_at [*required*] | date-time | Timestamp when the evaluator configuration was created. | | attributes | created_by | object | A Datadog user associated with a custom evaluator configuration. | | created_by | email | string | Email address of the user. | | attributes | eval_name [*required*] | string | Name of the custom evaluator. | | attributes | last_updated_by | object | A Datadog user associated with a custom evaluator configuration. | | last_updated_by | email | string | Email address of the user. | | attributes | llm_judge_config | object | LLM judge configuration for a custom evaluator. | | llm_judge_config | assessment_criteria | object | Criteria used to assess the pass/fail result of a custom evaluator. | | assessment_criteria | max_threshold | double | Maximum numeric threshold for a passing result. | | assessment_criteria | min_threshold | double | Minimum numeric threshold for a passing result. | | assessment_criteria | pass_values | [string] | Specific output values considered as a passing result. | | assessment_criteria | pass_when | boolean | When true, a boolean output of true is treated as passing. | | llm_judge_config | context_query | string | Query used to extract additional context for the evaluation. | | llm_judge_config | inference_params [*required*] | object | LLM inference parameters for a custom evaluator. | | inference_params | frequency_penalty | double | Frequency penalty to reduce repetition. | | inference_params | max_tokens | int64 | Maximum number of tokens to generate. | | inference_params | presence_penalty | double | Presence penalty to reduce repetition. | | inference_params | temperature | double | Sampling temperature for the LLM. | | inference_params | top_k | int64 | Top-k sampling parameter. | | inference_params | top_p | double | Top-p (nucleus) sampling parameter. | | llm_judge_config | last_used_library_prompt_template_name | string | Name of the last library prompt template used. | | llm_judge_config | modified_library_prompt_template | boolean | Whether the library prompt template was modified. | | llm_judge_config | output_schema | object | JSON schema describing the expected output format of the LLM judge. | | llm_judge_config | parsing_type | enum | Output parsing type for a custom LLM judge evaluator. Allowed enum values: `structured_output,json,keyword_search` | | llm_judge_config | prompt_template | [object] | List of messages forming the LLM judge prompt template. | | prompt_template | content | string | Text content of the message. | | prompt_template | contents | [object] | Multi-part content blocks for the message. | | contents | type [*required*] | string | Content block type. | | contents | value [*required*] | object | Value of a prompt message content block. | | value | text | string | Text content of the message block. | | value | tool_call | object | A tool call within a prompt message. | | tool_call | arguments | string | JSON-encoded arguments for the tool call. | | tool_call | id | string | Unique identifier of the tool call. | | tool_call | name | string | Name of the tool being called. | | tool_call | type | string | Type of the tool call. | | value | tool_call_result | object | A tool call result within a prompt message. | | tool_call_result | name | string | Name of the tool that produced this result. | | tool_call_result | result | string | The result returned by the tool. | | tool_call_result | tool_id | string | Identifier of the tool call this result corresponds to. | | tool_call_result | type | string | Type of the tool result. | | prompt_template | role [*required*] | string | Role of the message author. | | llm_judge_config | target_query | string | Query used to extract the target value to evaluate. | | llm_judge_config | user_specified_json_post_processing_function | string | User-provided function applied to post-process the JSON output of the LLM judge. | | attributes | llm_provider | object | LLM provider configuration for a custom evaluator. | | llm_provider | bedrock | object | AWS Bedrock-specific options for LLM provider configuration. | | bedrock | inference_profile | string | Bedrock inference profile identifier, such as an application inference profile ARN. | | bedrock | region | string | AWS region for Bedrock. | | llm_provider | integration_account_id | string | Integration account identifier. | | llm_provider | integration_provider | enum | Name of the LLM integration provider. Allowed enum values: `openai,amazon-bedrock,anthropic,azure-openai,vertex-ai,llm-proxy` | | llm_provider | model_name | string | Name of the LLM model. | | llm_provider | vertex_ai | object | Google Vertex AI-specific options for LLM provider configuration. | | vertex_ai | location | string | Google Cloud region. | | vertex_ai | project | string | Google Cloud project ID. | | attributes | target | object | Target application configuration for a custom evaluator. | | target | application_name [*required*] | string | Name of the ML application this evaluator targets. | | target | enabled [*required*] | boolean | Whether the evaluator is active for the target application. | | target | eval_scope | enum | Scope at which to evaluate spans. Allowed enum values: `span,trace,session` | | target | experiment_project_ids | [string] | Experiment project IDs this evaluator is scoped to. | | target | filter | string | Filter expression to select which spans to evaluate. | | target | root_spans_only | boolean | When true, only root spans are evaluated. | | target | sampling_percentage | double | Percentage of traces to evaluate. Must be greater than 0 and at most 100. | | attributes | updated_at [*required*] | date-time | Timestamp when the evaluator configuration was last updated. | | data | id [*required*] | string | Unique name identifier of the evaluator configuration. | | data | type [*required*] | enum | Type of the custom LLM Observability evaluator configuration resource. Allowed enum values: `evaluator_config` | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "category": "Custom", "created_at": "2024-01-15T10:30:00Z", "created_by": { "email": "user@example.com" }, "eval_name": "my-custom-evaluator", "last_updated_by": { "email": "user@example.com" }, "llm_judge_config": { "assessment_criteria": { "max_threshold": 1, "min_threshold": 0.7, "pass_values": [ "pass", "yes" ], "pass_when": true }, "context_query": "@input.context", "inference_params": { "frequency_penalty": 0, "max_tokens": 1024, "presence_penalty": 0, "temperature": 0.7, "top_k": 50, "top_p": 1 }, "last_used_library_prompt_template_name": "sentiment-analysis-v1", "modified_library_prompt_template": false, "output_schema": {}, "parsing_type": "structured_output", "prompt_template": [ { "content": "Rate the quality of the following response:", "contents": [ { "type": "text", "value": { "text": "What is the sentiment of this review?", "tool_call": { "arguments": "{\"location\": \"San Francisco\"}", "id": "call_abc123", "name": "get_weather", "type": "function" }, "tool_call_result": { "name": "get_weather", "result": "sunny, 72F", "tool_id": "call_abc123", "type": "function" } } } ], "role": "user" } ], "target_query": "@output.value", "user_specified_json_post_processing_function": "string" }, "llm_provider": { "bedrock": { "inference_profile": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123", "region": "us-east-1" }, "integration_account_id": "my-account-id", "integration_provider": "openai", "model_name": "gpt-4o", "vertex_ai": { "location": "us-central1", "project": "my-gcp-project" } }, "target": { "application_name": "my-llm-app", "enabled": true, "eval_scope": "span", "experiment_project_ids": [], "filter": "@service:my-service", "root_spans_only": true, "sampling_percentage": 50 }, "updated_at": "2024-01-15T10:30:00Z" }, "id": "my-custom-evaluator", "type": "evaluator_config" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl command curl -X GET "https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" ##### ```python """ List custom evaluator configurations returns "OK" response """ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi configuration = Configuration() configuration.unstable_operations["list_llm_obs_custom_eval_configs"] = True with ApiClient(configuration) as api_client: api_instance = LLMObservabilityApi(api_client) response = api_instance.list_llm_obs_custom_eval_configs() print(response) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest.md?code-lang=python) and then save the example to `example.py` and run following commands: DD_SITE="datadoghq.com" DD_API_KEY="" DD_APP_KEY="" python3 "example.py" ##### ```ruby # List custom evaluator configurations returns "OK" response require "datadog_api_client" DatadogAPIClient.configure do |config| config.unstable_operations["v2.list_llm_obs_custom_eval_configs".to_sym] = true end api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new p api_instance.list_llm_obs_custom_eval_configs() ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest.md?code-lang=ruby) and then save the example to `example.rb` and run following commands: DD_SITE="datadoghq.com" DD_API_KEY="" DD_APP_KEY="" rb "example.rb" ##### ```go // List custom evaluator configurations returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() configuration.SetUnstableOperationEnabled("v2.ListLLMObsCustomEvalConfigs", true) apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewLLMObservabilityApi(apiClient) resp, r, err := api.ListLLMObsCustomEvalConfigs(ctx) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `LLMObservabilityApi.ListLLMObsCustomEvalConfigs`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `LLMObservabilityApi.ListLLMObsCustomEvalConfigs`:\n%s\n", responseContent) } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest.md?code-lang=go) and then save the example to `main.go` and run following commands: DD_SITE="datadoghq.com" DD_API_KEY="" DD_APP_KEY="" go run "main.go" ##### ```java // List custom evaluator configurations returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.LlmObservabilityApi; import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigListResponse; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); defaultClient.setUnstableOperationEnabled("v2.listLLMObsCustomEvalConfigs", true); LlmObservabilityApi apiInstance = new LlmObservabilityApi(defaultClient); try { LLMObsCustomEvalConfigListResponse result = apiInstance.listLLMObsCustomEvalConfigs(); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling LlmObservabilityApi#listLLMObsCustomEvalConfigs"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest.md?code-lang=java) and then save the example to `Example.java` and run following commands: DD_SITE="datadoghq.com" DD_API_KEY="" DD_APP_KEY="" java "Example.java" ##### ```rust // List custom evaluator configurations returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_llm_observability::LLMObservabilityAPI; #[tokio::main] async fn main() { let mut configuration = datadog::Configuration::new(); configuration.set_unstable_operation_enabled("v2.ListLLMObsCustomEvalConfigs", true); let api = LLMObservabilityAPI::with_config(configuration); let resp = api.list_llm_obs_custom_eval_configs().await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest.md?code-lang=rust) and then save the example to `src/main.rs` and run following commands: DD_SITE="datadoghq.com" DD_API_KEY="" DD_APP_KEY="" cargo run ##### ```typescript /** * List custom evaluator configurations returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); configuration.unstableOperations["v2.listLLMObsCustomEvalConfigs"] = true; const apiInstance = new v2.LLMObservabilityApi(configuration); apiInstance .listLLMObsCustomEvalConfigs() .then((data: v2.LLMObsCustomEvalConfigListResponse) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest.md?code-lang=typescript) and then save the example to `example.ts` and run following commands: DD_SITE="datadoghq.com" DD_API_KEY="" DD_APP_KEY="" tsc "example.ts" {% /tab %}