Overview

Annotation Queues provide a structured workflow for human review of LLM traces. Use annotation queues to:

• Review traces with complete context including spans, metadata, tool calls, inputs, outputs, and evaluation results
• Apply structured labels and free-form observations to traces
• Identify and categorize failure patterns
• Validate LLM-as-a-Judge evaluation accuracy
• Build golden datasets with human-verified labels for testing and validation

Creating an annotation queue

Step 1: Configure queue settings

1. Navigate to AI Observability > Experiment > Annotations and select your project.

2. Click Create Queue.

3. On the About tab, configure:

   • Name: Descriptive name reflecting the queue’s purpose (for example, “Failed Evaluations Review - Q1 2026”)
   • Project: Agent Observability project this queue belongs to
   • Description (optional): Explain the queue’s purpose and any special instructions for annotators

4. Then click Next.

5. On the Schema tab, define your new queue’s label schema. Use the Preview pane to see how labels appear to annotators as you configure them. Each label can be marked as required and can optionally include:

   • Assessment criteria: Allow annotators to indicate pass/fail for that label value
   • Reasoning: Allow annotators to add a short explanation

6. Review your queue configuration and click Create to create the queue.

Step 2: Select traces for annotation

You can create add traces to queue manually from Trace Explorer or populate queues automatically using Automation Rules.

Annotating traces

Accessing your queues

Navigate to AI Observability > Experiment > Annotations to see all available annotation queues. Click on a queue to see the trace list, then click Review to begin annotating.

Review Mode displays:

• Full trace context (right panel):

  • Complete span tree with inputs, outputs, metadata
  • Tool calls and intermediate reasoning steps
  • Evaluation results on trace and individual spans

• Annotation controls (left panel):

  • Configured labels for this queue
  • Progress indicator showing position in queue
  • Navigation controls (Previous, Next)

Applying labels

For each trace:

1. Review the full trace context: Expand spans as needed to understand inputs, outputs, tool calls, and evaluation results.
2. Apply labels: Fill in the configured labels based on your assessment.
3. Annotations are be autosaved.

Best practices for annotation

Be consistent:

• Review the queue description and label definitions before starting.
• When multiple annotators work on the same queue, establish shared understanding of criteria.
• Document reasoning in notes for borderline cases.

Provide reasoning:

• Use free-form notes to document why you applied specific labels.
• Note patterns you observe across multiple traces.
• Reasoning helps refine evaluation criteria and understand failure modes.

Managing queues

Tracking queue progress

The Annotations list page displays a progress bar for each queue showing the ratio of reviewed interactions to total interactions. Use this to monitor annotation completion across queues at a glance.

Filtering traces by annotation labels

use the Annotation Labels facet to filter traces by labels applied in annotation queues. This allows you to:

• Find all traces tagged with a specific failure mode (for example, failure_type: hallucination)
• Build targeted samples for downstream review, dataset creation, or CSV export for data analysis

Editing queue schema

You can modify a queue’s label schema after creation:

1. Navigate to AI Observability > Experiment > Annotations.
2. Open the queue.
3. If the Details panel is hidden, click View Details.
4. Click Edit.
5. Add, remove, or modify labels.
6. Click Save Changes.

Exporting annotated data

Export annotated traces for analysis or use in other workflows:

1. Navigate to AI Observability > Experiment > Annotations.
2. Open the queue.
3. Select traces (or select all).
4. Click Export.

The file downloads as annotations_<queue-id>.csv. You can also retrieve span data programmatically using the Export API.

Content ID,Type,Input,Output,Alice_Johnson_quality,Alice_Johnson_quality_assessment,Alice_Johnson_quality_reasoning,Alice_Johnson_failure_type,Alice_Johnson_failure_type_assessment,Alice_Johnson_failure_type_reasoning,Bob_Smith_quality,...

Retrieve spans by trace ID or session ID

After exporting annotation data, use the Export API to retrieve the full span data for traces or sessions in the CSV and join it with your annotation labels.

By trace ID:

curl -G "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events" \
  -H "DD-API-KEY: <YOUR_DATADOG_API_KEY>" \
  -H "DD-APPLICATION-KEY: <YOUR_DATADOG_APPLICATION_KEY>" \
  --data-urlencode "filter[trace_id]=<TRACE_ID>"

By session ID:

curl -G "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events" \
  -H "DD-API-KEY: <YOUR_DATADOG_API_KEY>" \
  -H "DD-APPLICATION-KEY: <YOUR_DATADOG_APPLICATION_KEY>" \
  --data-urlencode "filter[query]=@session_id:<SESSION_ID>"

Adding to datasets

Transfer annotated traces to datasets for experiment evaluation:

1. Navigate to AI Observability > Experiment > Annotations.
2. Open the queue.
3. Select traces to transfer.
4. Click Add to Dataset.
5. Set the dataset’s expected output:
   • From interaction: use each trace’s actual output. For experiment traces, you can also pick Expected output to use the original expected output from the experiment’s source dataset.
   • From annotation label: use the values the annotators applied. Pick one or more labels. The record’s expected_output is built from your selection.
6. Choose an existing dataset, or create a dataset.

When expected output is built from annotation labels, the exported value is a JSON object keyed by label name, for example { "is_harmful": false, "tone": ["neutral"], "topics": ["safety", "policy"] }. The same shape applies whether you select one label or multiple labels. Categorical labels are always exported as arrays of selected options, whether the label is single-select or multi-select.

Labels not selected as expected output are also included with each trace as metadata.

See Datasets for more information about using datasets in experiments.

Deleting queues

To delete a queue:

1. Navigate to AI Observability > Experiment > Annotations.
2. Open the queue.
3. Click Delete in the Details panel.

Using the API

You can manage annotation queues programmatically. The following endpoints are available in the Agent Observability API reference:

Data retention

Example workflows

Further Reading

Additional helpful documentation, links, and articles:

Learn about evaluation typesDOCUMENTATION Route traces into queues automatically with Automation RulesDOCUMENTATION Run experiments to test improvementsDOCUMENTATION Annotate traces to improve LLM quality with Datadog LLM ObservabilityBLOG Agent Observability API referenceAPI