---
title: Code Evaluators
description: >-
  Create custom evaluators that run a JavaScript function on your spans, traces,
  or sessions instead of calling an LLM judge.
breadcrumbs: Docs > Agent Observability > Investigate > Evaluations > Code Evaluators
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Code Evaluators

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Code evaluators are custom evaluators that run a JavaScript function you write, instead of sending a prompt to an LLM judge. Use them for checks that have a definite answer, such as:

- Detecting empty or malformed tool results
- Checking whether user input or model output contains a keyword
- Counting spans, tool calls, or retries in a trace
- Computing a rate across the traces in a session

Because no LLM is called, code evaluators do not require an [LLM provider account](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/connect_to_account.md), consume no LLM tokens, and return the same result every time for the same input.

Code evaluators run on the same scopes as [custom LLM-as-a-judge evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md): span, trace, and session. Results appear across Agent Observability alongside your other evaluations.

## Create a code evaluator{% #create-a-code-evaluator %}

1. In Datadog, navigate to the Agent Observability [Evaluators page](https://app.datadoghq.com/llm/evaluations). Select Create Evaluator.

1. Start creating the evaluator in one of these ways:

   - Under Templates, select a template from the Code Evaluations group. The editor opens with Type set to Code (deterministic).
   - Under Scratch, select Code. The editor opens with Type set to Code (deterministic) and a minimal starting function.

1. Provide a clear, descriptive Name (for example, `empty-tool-result`). The name must be unique within your application.

1. In Runs on, select the application you want to evaluate, what you want to evaluate on (span, trace, or session), and the sampling rate. You can add more filtering criteria, for example, `@meta.span.kind:tool`, to evaluate only tool spans. See [Define the evaluation scope](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md#define-the-evaluation-scope-filtering-and-sampling) for details.

1. (Optional) To change your starting point, use the Template picker:

   - Create from scratch: Starts from a minimal function.
   - Empty tool result: Flags tool calls that return an empty result.
   - User input contains text: Checks whether the user input contains a given string.

Templates have a version for each scope. If you change the scope before editing the template's code, the editor switches to the template's version for the new scope. Code you have edited is not replaced.

1. Write or edit your function in Evaluation code. See Write the evaluation function.

1. Test the evaluator using the pane on the left. See Test and preview.

1. Select Publish to start running the evaluator, or Save as Draft to finish it later.

When Type is set to Code (deterministic), the model, prompt, and output sections of the editor are hidden, because the function defines the whole evaluation.

You can switch an existing evaluator between LLM judge and Code (deterministic) with the Type selector.

## Write the evaluation function{% #write-the-evaluation-function %}

The function must be named `__evalDeterministic`. It receives one argument, `data`, and returns the evaluation result:

```javascript
function __evalDeterministic(data) {
    return {
        value: true,
        assessment: "pass",
        reasoning: "Explain why the evaluation returned this value",
        metadata: { example_key: "example value" },
    };
}
```

### Input data{% #input-data %}

`data` object shapes depend on the evaluation scope:

| Scope   | `data`                                                                             | Example fields                                                                                        |
| ------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Span    | The span                                                                           | `data.name`, `data.span_id`, `data.meta.span.kind`, `data.meta.input.value`, `data.meta.output.value` |
| Trace   | `{ spans: [...] }`, every span in the trace                                        | `data.spans[0].name`, `data.spans.filter((s) => s.meta?.span?.kind === "tool")`                       |
| Session | `{ session_id, traces: [{ trace_id, spans: [...] }] }`, every trace in the session | `data.session_id`, `data.traces[0].spans`                                                             |

To see the exact fields available, select Test Evaluator > Test Manually. Then, select a span, trace, or session in the left pane of the editor to view its JSON.

Fields can be missing on some spans. Use optional chaining (`?.`) and defaults (`??`) to avoid errors, for example `data.meta?.output?.value ?? ""`.

### Return value{% #return-value %}

Your function's return value type determines the evaluation's metric type:

| Return value                                    | Metric type                           |
| ----------------------------------------------- | ------------------------------------- |
| `true` or `false`                               | Boolean                               |
| A number                                        | Score                                 |
| A non-empty string                              | Categorical                           |
| An object without a `value` or `assessment` key | JSON                                  |
| `null`, `undefined`, or `""`                    | No result. The evaluation is skipped. |

To add an assessment, reasoning, or metadata, return an object with a `value` key, an `assessment` key, or both. If `value` is an object, the result is JSON.

| Key          | Description                                                                       |
| ------------ | --------------------------------------------------------------------------------- |
| `value`      | The evaluation value. Mapped to a metric type using the rules above.              |
| `assessment` | Optional. `"pass"` or `"fail"`.                                                   |
| `reasoning`  | Optional. A string explaining the result.                                         |
| `metadata`   | Optional. An object with extra details, such as the IDs of the spans that failed. |

If the object has an `assessment` but no `value`, the result is a Boolean: `true` when the assessment is `"pass"`, and `false` when it is `"fail"`.

Return `null` to skip spans, traces, or sessions that the evaluation doesn't apply to, for example, a span that is not a tool call.

### Errors{% #errors %}

The evaluation fails with an error if your function returns:

- An array, or any other type not listed in the tables above
- An `assessment` other than `"pass"` or `"fail"`
- A `reasoning` that is not a string
- A `metadata` that is not an object

### Example: Empty tool result (span scope){% #example-empty-tool-result-span-scope %}

```javascript
function __evalDeterministic(data) {
    if (data.meta?.span?.kind !== "tool") {
        return null;
    }
    const output = String(data.meta?.output?.value ?? "").trim();
    const isEmpty = output === "" || output === "[]" || output === "{}";
    return {
        value: isEmpty,
        assessment: isEmpty ? "fail" : "pass",
        reasoning: isEmpty
            ? `Tool ${data.name} returned an empty result`
            : `Tool ${data.name} returned a result`,
    };
}
```

### Example: Empty tool results in a trace (trace scope){% #example-empty-tool-results-in-a-trace-trace-scope %}

```javascript
function __evalDeterministic(data) {
    const isEmpty = (span) => {
        const output = String(span.meta?.output?.value ?? "").trim();
        return output === "" || output === "[]" || output === "{}";
    };
    const toolSpans = (data.spans ?? []).filter((s) => s.meta?.span?.kind === "tool");
    if (toolSpans.length === 0) {
        return null;
    }
    const empty = toolSpans.filter(isEmpty);
    return {
        value: empty.length > 0,
        assessment: empty.length > 0 ? "fail" : "pass",
        reasoning: empty.length > 0
            ? `${empty.length} of ${toolSpans.length} tool call(s) returned an empty result`
            : `All ${toolSpans.length} tool call(s) returned a result`,
        metadata: { empty_tool_span_ids: empty.map((s) => s.span_id) },
    };
}
```

### Example: Empty tool call rate (session scope){% #example-empty-tool-call-rate-session-scope %}

This function returns a number, so the result is a score.

```javascript
function __evalDeterministic(data) {
    const toolSpans = (data.traces ?? []).flatMap((t) =>
        (t.spans ?? []).filter((s) => s.meta?.span?.kind === "tool"),
    );
    if (toolSpans.length === 0) {
        return null;
    }
    const empty = toolSpans.filter((s) => {
        const output = String(s.meta?.output?.value ?? "").trim();
        return output === "" || output === "[]" || output === "{}";
    });
    return empty.length / toolSpans.length;
}
```

## Test and preview{% #test-and-preview %}

The pane on the left shows the spans, traces, or sessions that match the evaluation scope. Select Test Evaluator, then:

- Select Auto Test to run your function on a sample of the matching spans, traces, or sessions.
- Select Test Manually to run your function on a span, trace, or session that you choose.

For each run, the result panel shows the returned value, any `console.log` output, and any error the function raised.

## Viewing and using results{% #viewing-and-using-results %}

Code evaluator results appear on the evaluator's Results tab, alongside other evaluation results. You can also search for them on the Agent Observability [Traces page](https://app.datadoghq.com/llm/traces), using the same syntax as other evaluations. For example:

- `@evaluation.empty-tool-result.value:true`
- `@evaluation.empty-tool-result.assessment:fail`

See [Viewing and using results](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md#viewing-and-using-results) for details.

## Configure code evaluators from the API{% #configure-code-evaluators-from-the-api %}

You can create and update code evaluators with the [custom evaluator configuration API](https://docs.datadoghq.com/api/latest/agent-observability.md#create-or-update-a-custom-evaluator-configuration). In the request body:

- Set `llm_provider.integration_provider` to `deterministic` and `llm_provider.model_name` to `javascript`.
- Omit `llm_provider.integration_account_id`, `prompt_template`, `output_schema`, and `inference_params`.
- Set `llm_judge_config.parsing_type` to `json`, and put your function in `llm_judge_config.user_specified_json_post_processing_function`.

Set `DD_SITE` to your [Datadog site](https://docs.datadoghq.com/getting_started/site.md) (for example, <YOUR_DATADOG_SITE>).

```bash
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/empty-tool-result" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -d @- <<'EOF'
{
  "data": {
    "type": "evaluator_config",
    "id": "empty-tool-result",
    "attributes": {
      "eval_name": "empty-tool-result",
      "target": {
        "application_name": "my-app",
        "eval_scope": "span",
        "filter": "@meta.span.kind:tool"
      },
      "llm_provider": {
        "integration_provider": "deterministic",
        "model_name": "javascript"
      },
      "llm_judge_config": {
        "parsing_type": "json",
        "user_specified_json_post_processing_function": "function __evalDeterministic(data) {\n  const out = String(data.meta?.output?.value ?? \"\").trim();\n  const empty = out === \"\" || out === \"[]\" || out === \"{}\";\n  return { value: empty, assessment: empty ? \"fail\" : \"pass\" };\n}"
      }
    }
  }
}
EOF
```

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Learn about custom LLM-as-a-judge evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md)
- [Prompt templating reference](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/prompt_templating.md)
- [Learn about managed evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations/managed_evaluations.md)
