Custom LLM-as-a-judge prompts inject span or trace data into the User message by wrapping a field path in {{ ... }}. The System Prompt holds the static instructions to the LLM judge and does not resolve placeholders. The same syntax works in both the test pane and at evaluation time.

At a glance

The autocomplete dropdown opens after you type {{ and lists fields available on the selected sample.

Span-scope syntax

Span-scope evaluations expose a single span per evaluation. Reference fields by their JSON path on the span.

Built-in aliases

The aliases adapt to the kind of span being evaluated, so you don’t have to branch on whether the span is an LLM call or an agent step.

Direct field paths

{{name}}
{{meta.input.value}}
{{meta.output.value}}
{{metrics.input_tokens}}

Array access

{{meta.input.messages[0].content}}     # First message only
{{meta.input.messages[*].content}}     # All messages, joined with newlines
{{meta.input.messages[0,2].content}}   # Inclusive range; out-of-bounds ends are clamped
{{meta.input.messages.content}}        # Implicit fan-out, equivalent to [*]

Trace-scope syntax

Trace-scope evaluations expose every span in the trace under the spans array. Use {{spans...}} paths to read across spans. The {{span_input}} and {{span_output}} aliases are not available in trace scope.

Reference the whole trace

{{spans}}    # JSON of every span in the trace
{{*}}        # Entire trace payload as JSON, including top-level metadata

Pick a span by index

{{spans[0].meta.input.value}}    # First span
{{spans[*].name}}                # Newline-joined names of every span

Filter spans by attribute

{{spans[name:my-span].meta.input.value}}
{{spans[meta.span.kind:llm].meta.output.value}}
{{spans[meta.span.kind:tool].meta.input.parameters}}

[field.path:value] keeps only the spans whose field at field.path equals value. Combine with deeper paths to extract the inputs or outputs of the matching spans. The filter falls back to an empty string if no span matches.

Resolution rules

For example, given a span where meta.input.messages is:

[
  { "role": "user", "content": "hello" },
  { "role": "user", "content": "help please" }
]

Tips

• Type {{ in the prompt editor to open the autocomplete dropdown. The list adapts to the scope (span or trace) and to the sample selected on the right.
• Pick a sample row in the Filtered Spans panel (span scope) or the Spans in Selected Trace panel (trace scope), then click Test Evaluation to preview how each placeholder resolves on real data before saving the configuration.
• Use the three-dots menu on a sample’s JSON view and select Add variable to message to insert a field path into the prompt without typing it.
• Pass {{*}} when you want the LLM judge to see the full payload—useful for free-form prompts that decide for themselves which fields matter.

Further Reading

Additional helpful documentation, links, and articles:

Custom LLM-as-a-Judge EvaluationsDOCUMENTATION Trace-Level EvaluationsDOCUMENTATION