---
title: Evaluator compatibility
description: Learn about the compatibility requirements for evaluators.
breadcrumbs: >-
  Docs > Agent Observability > Investigate > Evaluations > Evaluator
  compatibility
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Evaluator compatibility

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Evaluator compatibility{% #evaluator-compatibility %}

The supported third party LLM providers are OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, Vertex AI, and Custom Provider.

### Managed evaluators{% #managed-evaluators %}

Managed evaluators are supported for the following configurations.

| Evaluator                                                                                                      | DD-trace version | LLM Provider | Applicable span |
| -------------------------------------------------------------------------------------------------------------- | ---------------- | ------------ | --------------- |
| [Language Mismatch](https://docs.datadoghq.com/llm_observability/investigate/evaluations/language_mismatch.md) | Fully supported  | Self hosted  | All span kinds  |

### Custom LLM-as-a-judge evaluators{% #custom-llm-as-a-judge-evaluators %}

Custom LLM-as-a-judge evaluators are supported for the following configurations.

| Evaluator                                                                                                                                      | DD-trace version | LLM Provider                  | Applicable span |
| ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ----------------------------- | --------------- |
| [Boolean](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md#define-the-evaluation-output)     | Fully supported  | All third party LLM providers | All span kinds  |
| [Score](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md#define-the-evaluation-output)       | Fully supported  | All third party LLM providers | All span kinds  |
| [Categorical](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md#define-the-evaluation-output) | Fully supported  | All third party LLM providers | All span kinds  |
| [JSON](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md#define-the-evaluation-output)        | Fully supported  | All third party LLM providers | All span kinds  |

#### LLM-as-a-judge evaluator templates{% #llm-as-a-judge-evaluator-templates %}

Existing templates for custom LLM-as-a-judge evaluators are supported for the following configurations.

| Evaluator                                                                                                                                                                      | DD-trace version | LLM Provider                  | Applicable span |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------- | ----------------------------- | --------------- |
| [Failure to Answer](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#failure-to-answer)                 | Fully supported  | All third party LLM providers | All span kinds  |
| [Hallucination](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#hallucination)                         | Fully supported  | All third party LLM providers | LLM only        |
| [Sentiment](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#sentiment)                                 | Fully supported  | All third party LLM providers | All span kinds  |
| [Toxicity](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#toxicity)                                   | Fully supported  | All third party LLM providers | All span kinds  |
| [Prompt Injection](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#prompt-injection)                   | Fully supported  | All third party LLM providers | All span kinds  |
| [Topic Relevancy](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#topic-relevancy)                     | Fully supported  | All third party LLM providers | All span kinds  |
| [Tool Selection](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#tool-selection)                       | Fully supported  | All third party LLM providers | LLM only        |
| [Tool Argument Correctness](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#tool-argument-correctness) | Fully supported  | All third party LLM providers | LLM only        |
| [Goal Completeness](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md#goal-completeness)                 | Fully supported  | All third party LLM providers | LLM only        |