Custom LLM-as-a-judge evaluations use an LLM to judge the performance of another LLM. You can define evaluation logic with natural language prompts, capture subjective or objective criteria (like tone, helpfulness, or factuality), and run these evaluations at scale across your traces and spans.

Create a custom LLM-as-a-judge evaluation

You can create and manage custom evaluations from the Evaluations page in LLM Observability.

Learn more about the compatibility requirements.

Configure the prompt

1. In Datadog, navigate to the LLM Observability Evaluations page. Select Create Evaluation, then select Create your own.

2. Provide a clear, descriptive evaluation name (for example, factuality-check or tone-eval). You can use this name when querying evaluation results. The name must be unique within your application.

3. Use the Account drop-down menu to select the LLM provider and corresponding account to use for your LLM judge. To connect a new account, see connect an LLM provider.

4. Use the Model drop-down menu to select a model to use for your LLM judge.

5. Under Evaluation Prompt section, use the Prompt Template drop-down menu:

   • Create from scratch: Use your own custom prompt (defined in the next step).
   • Failure to Answer, Prompt Injection, Sentiment, etc.: Populate a pre-existing prompt template. You can use these templates as-is, or modify them to match your specific evaluation logic.

6. In the System Prompt field, enter your custom prompt or modify a prompt template. For custom prompts, provide clear instructions describing what the evaluator should assess.

   • Focus on a single evaluation goal
   • Include 2–3 few-shot examples showing input/output pairs, expected results, and reasoning.

You will be looking at interactions between a user and a budgeting AI agent. Your job is to classify the user's intent when it comes to using the budgeting AI agent.

You will be given a Span Input, which represents the user's message to the agent, which you will then classify. Here are some examples.

Span Input: What are the core things I should know about budgeting?
Classification: general_financial_advice

Span Input: Did I go over budget with my grocery bills last month?
Classification: budgeting_question

Span Input: What is the category for which I have the highest budget?
Classification: budgeting_question

Span Input: Based on my past months, what is my ideal budget for subscriptions?
Classification: budgeting_advice

Span Input: Raise my restaurant budget by $50
Classification: budgeting_request

Span Input: Help me plan a trip to the Maldives
Classification: unrelated

Span Input: {{span_input}}

7. In the User field, provide your user prompt. Explicitly specify what parts of the span to evaluate: Span Input ({{span_input}}), Output ({{span_output}}), or both.

Define the evaluation output

For OpenAI or Azure OpenAI models, configure Structured Output.

For Anthropic or Amazon Bedrock models, configure Keyword Search Output.

{
    "name": "categorical_eval",
    "schema": {
        "type": "object",
        "required": [
            "categorical_eval",
            "reasoning"
        ],
        "properties": {
            "categorical_eval": {
                "type": "string",
                "anyOf": [
                    {
                        "const": "budgeting_question",
                        "description": "The user is asking a question about their budget. The answer can be directly determined by looking at their budget and spending."
                    },
                    {
                        "const": "budgeting_request",
                        "description": "The user is asking to change something about their budget. This should involve an action that changes their budget."
                    },
                    {
                        "const": "budgeting_advice",
                        "description": "The user is asking for advice on their budget. This should not require a change to their budget, but it should require an analysis of their budget and spending."
                    },
                    {
                        "const": "general_financial_advice",
                        "description": "The user is asking for general financial advice which is not directly related to their specific budget. However, this can include advice about budgeting in general."
                    },
                    {
                        "const": "unrelated",
                        "description": "This is a catch-all category for things not related to budgeting or financial advice."
                    }
                ]
            },
            "reasoning": {
                "type": "string",
                "description": "Describe how you decided the category"
            }
        },
        "additionalProperties": false
    },
    "strict": true
}

Define the evaluation scope: Filtering and sampling

Under Evaluation Scope, define where and how your evaluation runs. This helps control coverage (which spans are included) and cost (how many spans are sampled).

• Application: Select the application you want to evaluate.
• Evaluate On: Choose one of the following:
  • Traces: Evaluate only root spans
  • All Spans: Evaluate both root and child spans
• Span Names: (Optional) Limit evaluation to spans with certain names.
• Tags: (Optional) Limit evaluation to spans with certain tags.
• Sampling Rate: (Optional) Apply sampling (for example, 10%) to control evaluation cost.

Test and preview

Use the Test Evaluation panel on the right to preview results. You can enter sample {{span_input}} and {{span_output}} values and click Run Evaluation to see both the result, the reasoning explanation, and whether it passed or failed returned by your LLM judge.

Refine your prompt and schema until outputs are consistent and interpretable.

Viewing and using results

After you save your evaluation, Datadog automatically runs your evaluation on targeted spans. Results are available across LLM Observability in near-real-time. You can find your custom LLM-as-a-judge results for a specific span in the Evaluations tab, alongside other evaluations.

Each evaluation result includes:

• The evaluated value (for example True, 9, or Neutral)
• The reasoning (when enabled)
• The pass/fail indicator (based on your assessment criteria)

Use the syntax @evaluations.custom.<evaluation_name> to query or visualize results.

For example:

@evaluations.custom.helpfulness-check

You can:

• Filter traces by evaluation results (example, @evaluations.custom.helpfulness-check)
• Filter by pass/fail assessment status (example, @evaluations.assessment.custom.helpfulness-check:fail)
• Use evaluation results as facets
• View aggregate results in the LLM Observability Overview page’s Evaluation section
• Create monitors to alert on performance changes or regression

Best practices for reliable custom evaluations

• Start small: Target a single, well-defined failure mode before scaling.
• Enable reasoning when you need explainable decisions and to improve the accuracy on complex reasoning tasks.
• Iterate: Run, inspect outputs, and refine your prompt.
• Validate: Periodically check evaluator accuracy using sampled traces.
• Document your rubric: Clearly define what “Pass” and “Fail” mean to avoid drift over time.
• Re-align your evaluator: Reassess prompt and few-shot examples when the underlying LLM updates.

Further Reading

Additional helpful documentation, links, and articles:

Learn about LLM Observability terms and conceptsDOCUMENTATION Learn how to set up LLM ObservabilityDOCUMENTATION Learn about managed evaluationsDOCUMENTATION Building an LLM evaluation framework: best practicesBLOG Using LLM-as-a-judge for an automated and versatile evaluationHUGGING FACE