---
title: Annotation Queues
description: >-
  Enable systematic human review of LLM traces to identify failure modes,
  validate automated evaluations, and build golden datasets.
breadcrumbs: Docs > Agent Observability > Investigate > Annotation Queues
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Annotation Queues

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Annotation Queues provide a structured workflow for human review of LLM traces. Use annotation queues to:

- Review traces with complete context including spans, metadata, tool calls, inputs, outputs, and evaluation results
- Apply structured labels and free-form observations to traces
- Identify and categorize failure patterns
- Validate LLM-as-a-Judge evaluator accuracy
- Build golden datasets with human-verified labels for testing and validation

## Creating an annotation queue{% #creating-an-annotation-queue %}

### Step 1: Configure queue settings{% #step-1-configure-queue-settings %}

1. Navigate to [AI Observability > Annotations](https://app.datadoghq.com/llm/annotations/queues) and select your project.

1. Click Create Queue.

1. On the About tab, configure:

   - Name: Descriptive name reflecting the queue's purpose (for example, "Failed Evaluations Review - Q1 2026")
   - Project: Agent Observability project this queue belongs to
   - Description (optional): Explain the queue's purpose and any special instructions for annotators

1. Then click Next.

1. On the Schema tab, define your new queue's label schema. Use the Preview pane to see how labels appear to annotators as you configure them. Each label can be marked as required and can optionally include:

   - Assessment criteria: Allow annotators to indicate pass/fail for that label value
   - Reasoning: Allow annotators to add a short explanation

1. Review your queue configuration and click Create to create the queue.

   {% image
      source="https://docs.dd-static.net/images/llm_observability/evaluations/annotation_queues/schema_edit.a609351cbdde06a053e7e813868f061f.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/evaluations/annotation_queues/schema_edit.a609351cbdde06a053e7e813868f061f.png?auto=format&fit=max&w=850&dpr=2 2x"
      alt="The Edit Queue modal showing the Schema tab with label configuration on the left and a preview pane on the right. The left panel displays fields for configuring a categorical label named failure_type with three categories: hallucination, formatting_error, and refusal. Checkboxes enable Assessment Criteria and Reasoning options. The right preview pane shows how the label appears to annotators with checkboxes for each category, Pass/Fail assessment buttons, and a reasoning text field." /%}

#### Importing evaluation labels{% #importing-evaluation-labels %}

When you create a queue from a completed experiment, Datadog automatically adds the experiment's evaluator scorers as blank, editable labels. Existing evaluator result values are not copied into the queue. Review the imported labels and edit or remove them as needed before creating the queue.

For other queue creation flows, use Add Evaluation as Label on the Schema tab to add available evaluation labels manually.

### Step 2: Select traces for annotation{% #step-2-select-traces-for-annotation %}

You can create add traces to queue manually from Trace Explorer or populate queues automatically using Automation Rules.

{% tab title="Manually from Trace Explorer" %}
Add traces to a queue manually from the Trace Explorer:

1. Navigate to [AI Observability > Traces](https://app.datadoghq.com/llm/traces)
1. Filter traces using available facets (evaluation results, error status, application, time range)
1. Select individual traces or bulk select multiple traces
1. Click Flag for Annotation
1. Choose Create New Queue or select an existing queue

{% /tab %}

{% tab title="Using Automation Rules" %}
Instead of manually selecting traces, use Automation Rules to route traces into annotation queues automatically based on filters and sampling criteria. This enables continuous, hands-off queue population without requiring manual trace selection. See [Automation Rules](https://docs.datadoghq.com/llm_observability/configure/automation_rules.md) for the full feature reference, including supported filter fields and limits.

{% alert level="info" %}
Automations apply going forward: new traces matching your rule are routed to the queue as they arrive. Existing traces matching the filter are not added retroactively.
{% /alert %}

To add an annotation queue action to an Automation Rule:

1. Navigate to [AI Observability > Traces](https://app.datadoghq.com/llm/traces)
1. Apply filters to identify traces you want to route (evaluation failures, latency thresholds, specific applications). See [Automation Rules > Supported filter fields](https://docs.datadoghq.com/llm_observability/configure/automation_rules.md#supported-filter-fields) for what's allowed.
1. Click Automate Query
1. Configure sampling rate (up to 5% for annotation queues; for example, 2% of matching traces).
1. Under Actions, select Add to Annotation Queue.
1. Choose the target queue.
1. Save the rule.

Traces matching the rule's filters are added to the queue as they arrive. Annotation queues hold up to 1,000 records; the automation pauses when the queue reaches that limit.
{% /tab %}

## Annotating traces{% #annotating-traces %}

### Accessing your queues{% #accessing-your-queues %}

Navigate to [AI Observability > Annotations](https://app.datadoghq.com/llm/annotations/queues) to see all available annotation queues. Click on a queue to see the trace list, then click Review to begin annotating.

**Note**: When an annotation task is newly assigned to you, Datadog sends an email with the queue name, the user who assigned it, and a direct link to open the assigned interaction. To opt out, disable Agent Observability annotation assignment emails in [Personal Settings email subscriptions](https://docs.datadoghq.com/account_management.md#email-subscriptions).

Review Mode displays:

- Full trace context (right panel):

  - Complete span tree with inputs, outputs, metadata
  - Tool calls and intermediate reasoning steps
  - Evaluation results on trace and individual spans

- Annotation controls (left panel):

  - Configured labels for this queue
  - Progress indicator showing position in queue
  - Navigation controls (Previous, Next)

  {% image
     source="https://docs.dd-static.net/images/llm_observability/evaluations/annotation_queues/review.5d52d7b14a54e6ba64e36cefaf677a30.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/evaluations/annotation_queues/review.5d52d7b14a54e6ba64e36cefaf677a30.png?auto=format&fit=max&w=850&dpr=2 2x"
     alt="The annotation review interface showing the annotation panel on the left and trace details on the right. The left panel displays label controls including failure_type checkboxes for hallucination, formatting_error, and refusal, plus a requires_escalation assessment with Pass and Fail buttons and a Save button at the bottom. The right panel shows the trace details for citizen_agent with a span tree, evaluation results, and expandable sections for Input and Output displaying JSON-formatted data about a weather information query." /%}

### Applying labels{% #applying-labels %}

For each trace:

1. **Review the full trace context**: Expand spans as needed to understand inputs, outputs, tool calls, and evaluation results.
1. **Apply labels**: Fill in the configured labels based on your assessment.
1. Annotations are autosaved.

### Adding comments{% #adding-comments %}

Use the Comments panel to discuss the interaction with other reviewers while you annotate it. Comments are associated with the trace being reviewed. For span interactions, the comment is associated with the corresponding span.

To add a comment:

1. Expand the Comments panel.
1. Click Add comment.
1. Enter your comment and click Save.

You can edit or delete comments that you created.

### Best practices for annotation{% #best-practices-for-annotation %}

**Be consistent**:

- Review the queue description and label definitions before starting.
- When multiple annotators work on the same queue, establish shared understanding of criteria.
- Document reasoning in notes for borderline cases.

**Provide reasoning**:

- Use free-form notes to document why you applied specific labels.
- Note patterns you observe across multiple traces.
- Reasoning helps refine evaluation criteria and understand failure modes.

## Annotation views{% #annotation-views %}

Annotation views control how the annotation workspace displays the interaction you are reviewing. Choose a built-in view for conversations, structured data, or generated web pages. Build a custom view with [Datadog Apps](https://docs.datadoghq.com/actions/datadog_apps.md) when your review workflow needs a different presentation.

Use the same interaction view while applying individual annotations or reviewing final scores.

### Use a built-in view{% #use-a-built-in-view %}

1. Open an interaction in an annotation queue.
1. Open the view menu.
1. Under Default Views, select a view:
| View               | Use it to                                                        |
| ------------------ | ---------------------------------------------------------------- |
| Trace Conversation | Review messages and tool calls as a conversation.                |
| Data Explorer      | Examine JSON objects and arrays from the selected span's output. |
| Web Page Renderer  | Preview HTML and CSS from the selected span's output.            |

Select Full Interaction to return to the standard trace or session view.

#### Review a trace conversation{% #review-a-trace-conversation %}

Use Trace Conversation to follow messages and tool calls across a trace:

- Search message content, span names, and services.
- Enable System messages to include system and context messages.
- Select a message or tool call to select its associated span.
- Switch between Chat and Span to review the conversation or the selected span's input and output.
- Copy individual messages, tool calls, inputs, or outputs for further review.

A preview trace can contain truncated span content. Check the preview notice when interpreting a conversation.

#### Explore structured output{% #explore-structured-output %}

Use Data Explorer when the selected span returns a JSON object or array:

- Search for keys or values.
- Use Structured to explore nested fields and expand or collapse objects and arrays.
- Use JSON to inspect the formatted JSON output.

Selecting another span updates the output and resets the search and expansion state. If the span has no supported structured output, select another span or use Full Interaction.

#### Preview web page output{% #preview-web-page-output %}

Use Web Page Renderer to review generated page output. Supported formats include raw or fenced HTML and JSON with `document`, `html`, or `code` fields. JSON can also include separate `css` and `javascript` or `js` fields.

The preview renders HTML and CSS with scripts disabled. Content that depends on JavaScript or external resources may appear incomplete. Select a span with page output to display its preview.

### Use a published custom view{% #use-a-published-custom-view %}

1. Open the view menu in an annotation queue interaction.
1. Search for a view under My Custom Views or All Custom Views.
1. Select the view to display the interaction with that App.

Select the star next to a custom view to keep it at the top of its group. Custom views follow [App permissions](https://docs.datadoghq.com/actions/app_builder/access_and_auth.md#app-permissions).

The workspace remembers the last view that loaded successfully for your user and organization in the same browser. Copy the interaction URL to share the selected view. Recipients need access to the queue and, for custom views, the App.

### Create a custom view{% #create-a-custom-view %}

To create a custom view, you need:

- Access to Create a custom view in the view menu.
- A local coding agent and Node.js version 20.12.0 or later. See the [Datadog Apps prerequisites](https://docs.datadoghq.com/actions/datadog_apps.md#prerequisites).
- Permission to create and publish Datadog Apps in the organization where you review annotations. See [App permissions](https://docs.datadoghq.com/actions/app_builder/access_and_auth.md#app-permissions).

Open an interaction in an annotation queue. In the view menu, select Create a custom view.

#### Step 1: Install the skill{% #step-1-install-the-skill %}

On the Install skill tab, copy the installation command and run it in your project's terminal:

```shell
npx skills add datadog-labs/agent-skills --skill datadog-app --full-depth -y
```

If the skill does not appear in your agent, restart the agent. If you already installed the skill, continue to the next step.

#### Step 2: Build and publish the view{% #step-2-build-and-publish-the-view %}

1. On the Copy prompt tab, enter a description in What would you like to see?. For example:

   ```text
   Show tool calls as a timeline, highlight failed steps, and display the final response alongside the original input.
   ```

1. Copy the generated prompt and paste it into your local coding agent. The prompt includes the target organization and Datadog site. It also includes instructions for reading the selected interaction and the tag that identifies the App as an annotation view.

1. Complete browser authorization when prompted. Select the organization shown in the setup dialog.

1. Work with your agent to build and preview the view locally. Review the result and approve uploading and publishing it to the target organization. For details, see [Build and upload your App](https://docs.datadoghq.com/actions/datadog_apps.md#build-and-upload-your-app) and [Publish and manage your Apps](https://docs.datadoghq.com/actions/datadog_apps.md#publish-and-manage-your-apps).

1. Return to the setup dialog and open the Use your view tab.

#### Alternative: use API keys instead of browser authorization{% #alternative-use-api-keys-instead-of-browser-authorization %}

For CI or environments where browser sign-in is unavailable:

1. Enable Use API keys instead on the Copy prompt tab.
1. Select or create an API key and enter an application key. The application key requires [Actions API Access](https://docs.datadoghq.com/account_management/api-app-keys.md#actions-api-access) and permission to create and publish Apps.
1. Copy the generated commands and run them in your terminal to set `DD_SITE`, `DD_API_KEY`, and `DD_APP_KEY` before starting your agent.
1. Copy the updated prompt into your agent and continue building the view.

Keep credentials in your terminal environment. Do not add them to source code or frontend bundles.

#### Step 3: Preview and apply the view{% #step-3-preview-and-apply-the-view %}

1. On the Use your view tab, select Refresh to load published Apps.
1. Search the Published App dropdown and select your view. If there is only one matching published App, it is selected automatically.
1. Review the preview, which displays the selected interaction using your App.
1. Select Use this view to apply it to the annotation workspace.

The workspace switches to the selected view after you select Use this view. Closing the dialog without applying a view keeps your existing selection.

### Update a custom view{% #update-a-custom-view %}

1. Work with your local agent to change the App and preview the changes locally.
1. Upload and publish an updated version of the same App.
1. Open Create a custom view from the annotation workspace's view menu and go to Use your view.
1. Select Refresh, select the App, and review the updated preview.
1. Select Use this view to apply it.

### Troubleshoot annotation views{% #troubleshoot-annotation-views %}

#### The published App does not appear{% #the-published-app-does-not-appear %}

- Check that you published the App to the same organization and Datadog site as the annotation workspace.
- Keep the annotation-view tag from the generated prompt in your App's configuration.
- Check that you have permission to view the App.
- Select Refresh after publishing. If there are no published Apps, select Back to prompt to continue working with your agent.

#### Published Apps fail to load{% #published-apps-fail-to-load %}

Select Refresh to retry loading the list.

## Final scores{% #final-scores %}

Use final scores to record a reviewed set of label values for an interaction in the queue. Compare individual annotator responses, resolve disagreements, and save the values to use as ground truth for evaluation.

Each final value follows its label's type: Boolean, categorical, score, or text. You can also record a pass/fail assessment where the label supports it and add reasoning. Individual annotations remain available after you save a final score.

### Final score access requirements{% #final-score-access-requirements %}

To review and save final scores, you need:

- An annotation queue with a label schema and an interaction to review.
- Access to Final Score in the annotation workspace.
- Permission to manage the queue to save or update final scores. Other users with access to the queue can view final scores.

### Review and save a final score{% #review-and-save-a-final-score %}

1. Open an annotation queue.
1. Select the Final Scores table tab, then open an interaction. From an interaction that is already open, select Finalized Score next to Annotate.
1. For each label, review the response summary. Select the response count to examine individual reviewers' values. A Disagreement indicator highlights labels with different responses.
1. Review the suggested value, assessment, and reasoning. Accept or reject each suggestion, or edit the field to provide your reviewed result. Editing a suggested field accepts that suggestion.
1. Optionally, use Accept all or Reject all to resolve all pending suggestions.
1. Enter any additional final values and reasoning, then click Save.

Resolve all pending suggestions before saving. Final scores require an explicit save. To update a saved final score, edit its fields and click Save again.

Use built-in or custom annotation views to inspect the interaction while reviewing its final scores.

### Interpret response summaries and suggestions{% #interpret-response-summaries-and-suggestions %}

Response summaries show the annotators' submitted values, including distributions and disagreements. Numeric score summaries also show statistics such as the mean and median.

When an interaction has no saved final score, the workspace prepares suggested values from the available responses:

| Label type  | Suggested final value                                                                                   |
| ----------- | ------------------------------------------------------------------------------------------------------- |
| Boolean     | The most frequent value. For pass/fail labels, the summary uses the majority assessment when available. |
| Categorical | Options selected by more than half of the reviewers who provided a value for that label.                |
| Score       | The most frequent numeric value.                                                                        |
| Text        | Enter the final text value after reviewing the individual responses.                                    |

Boolean and numeric suggestions remain empty when multiple values tie for the highest count. Suggested pass/fail assessments remain empty when pass and fail counts tie.

Suggested reasoning collects reviewers' explanations with attribution. For text labels, it also includes their text responses. Review and edit this content before accepting it.

The suggested final values require your review before saving. Dataset exports that aggregate individual annotations use a separate aggregation method.

### Compare final scores with evaluations{% #compare-final-scores-with-evaluations %}

Use Compare evaluations, when available, to compare reviewed final values with automated evaluation results:

1. Open the queue's Final Scores table tab.
1. Enable Compare evaluations.
1. Review the final-score and evaluation columns for each label. Hover over an evaluation result to see whether it agrees with the final value.

The comparison pairs results by label name and value type. If there is no matching evaluation, no final value, or no available evaluation result, the table explains why the comparison is missing. Session-level snapshot evaluations are unavailable for this comparison.

For export options, see Exporting annotated data and Adding to datasets.

### Troubleshoot final scores{% #troubleshoot-final-scores %}

#### Save is unavailable{% #save-is-unavailable %}

- Accept or reject all pending value, assessment, and reasoning suggestions.
- Check that you have permission to manage the queue.
- Change a field or accept a suggestion to create a result to save.

#### A label has no suggested final value{% #a-label-has-no-suggested-final-value %}

Check whether reviewers have submitted a value or whether their responses tie. Enter the final value after reviewing the responses. Text labels require you to enter the final text.

## Managing queues{% #managing-queues %}

### Tracking queue progress{% #tracking-queue-progress %}

The Annotations list page displays a progress bar for each queue showing the ratio of reviewed interactions to total interactions. Use this to monitor annotation completion across queues at a glance.

### Managing queue access{% #managing-queue-access %}

Queue owners manage the reviewer list, access settings, and assignments from the queue details. Only the queue owner can change the reviewer list, access settings, or assignments.

Access restrictions apply independently, so you can enable either restriction or both:

- Reviewer restriction limits unassigned interactions to designated reviewers.
- Assignee restriction limits assigned interactions to their assignees.
- When both restrictions are enabled, reviewers can annotate unassigned interactions and assignees can annotate their assigned interactions.

The queue owner retains access and can annotate all interactions.

These settings control who can annotate a queue, not who can see it. To control which teams and roles can see a queue at all, restrict its project with [Data Access Control](https://docs.datadoghq.com/llm_observability/data_governance/access_control.md).

### Filtering traces by annotation labels{% #filtering-traces-by-annotation-labels %}

use the Annotation Labels facet to filter traces by labels applied in annotation queues. This allows you to:

- Find all traces tagged with a specific failure mode (for example, `failure_type: hallucination`)
- Build targeted samples for downstream review, dataset creation, or CSV export for data analysis

### Editing queue schema{% #editing-queue-schema %}

You can modify a queue's label schema after creation:

1. Navigate to [AI Observability > Annotations](https://app.datadoghq.com/llm/annotations/queues).
1. Open the queue.
1. If the Details panel is hidden, click View Details.
1. Click Edit.
1. Add, remove, or modify labels.
1. Click Save Changes.

{% alert level="info" %}
Changing the schema doesn't affect already-applied labels, but annotators will see the updated schema going forward.
{% /alert %}

### Exporting annotated data{% #exporting-annotated-data %}

Export annotated traces for analysis or use in other workflows:

1. Navigate to [AI Observability > Annotations](https://app.datadoghq.com/llm/annotations/queues).
1. Open the queue.
1. Select traces (or select all).
1. Click Export.

The file downloads as `annotations_<queue-id>.csv`. You can also retrieve span data programmatically using the [Export API](https://docs.datadoghq.com/llm_observability/investigate/export_api.md?tab=model#api-standards). When final scoring is available, CSV downloads also include saved final scores.

{% collapsible-section #csv-format %}
#### CSV format

Each row represents one annotated interaction. The file begins with these fixed columns:

| Column            | Description                                                                                |
| ----------------- | ------------------------------------------------------------------------------------------ |
| `Content ID`      | ID of the annotated content (for example, a trace ID or session ID)                        |
| `Type`            | Interaction type: `trace`, `experiment_trace`, or `session`                                |
| `Input`           | Input summary (empty for session interactions)                                             |
| `Output`          | Output summary (empty for session interactions)                                            |
| `Expected Output` | Only present when Include Expected Output is enabled; populated for experiment traces only |

After the fixed columns, there is one set of columns per reviewer per label. Reviewers are sorted alphabetically by display name (spaces replaced with underscores). Labels follow the order defined in the queue schema:

| Column                          | Description                                                        |
| ------------------------------- | ------------------------------------------------------------------ |
| `{reviewer}_{label}`            | Label value (string, number, boolean, or JSON array)               |
| `{reviewer}_{label}_assessment` | `pass` or `fail`, if assessment criteria is enabled for that label |
| `{reviewer}_{label}_reasoning`  | Free-text reasoning, if reasoning is enabled for that label        |

If a reviewer has not annotated a given row, those cells are empty.

**Example**: A queue with reviewers Alice Johnson and Bob Smith and labels `quality` (score) and `failure_type` (categorical) produces these column headers:

```
Content ID,Type,Input,Output,Alice_Johnson_quality,Alice_Johnson_quality_assessment,Alice_Johnson_quality_reasoning,Alice_Johnson_failure_type,Alice_Johnson_failure_type_assessment,Alice_Johnson_failure_type_reasoning,Bob_Smith_quality,...
```

{% /collapsible-section %}

#### Retrieve spans by trace ID or session ID{% #retrieve-spans-by-trace-id-or-session-id %}

After exporting annotation data, use the [Export API](https://docs.datadoghq.com/llm_observability/investigate/export_api.md?tab=model#api-standards) to retrieve the full span data for traces or sessions in the CSV and join it with your annotation labels.

**By trace ID**:

```bash
curl -G "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events" \
  -H "DD-API-KEY: <YOUR_DATADOG_API_KEY>" \
  -H "DD-APPLICATION-KEY: <YOUR_DATADOG_APPLICATION_KEY>" \
  --data-urlencode "filter[trace_id]=<TRACE_ID>"
```

**By session ID**:

```bash
curl -G "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events" \
  -H "DD-API-KEY: <YOUR_DATADOG_API_KEY>" \
  -H "DD-APPLICATION-KEY: <YOUR_DATADOG_APPLICATION_KEY>" \
  --data-urlencode "filter[query]=@session_id:<SESSION_ID>"
```

### Adding to datasets{% #adding-to-datasets %}

Transfer annotated traces to datasets for experiment evaluation:

1. Navigate to [AI Observability > Annotations](https://app.datadoghq.com/llm/annotations/queues).
1. Open the queue.
1. Select traces to transfer.
1. Click Add to Dataset.
1. Set the dataset's expected output:
   - From interaction: use each trace's actual output. For experiment traces, you can also pick Expected output to use the original expected output from the experiment's source dataset.
   - From annotation label: use the values the annotators applied. Pick one or more labels. The record's `expected_output` is built from your selection.
   - From final score: use the final score recorded for each label instead of values aggregated across annotators. Pick one or more labels. The record's `expected_output` is a JSON object keyed by the selected label names. Traces without a final score for a selected label are still exported, with `null` for that label.
1. Choose an existing dataset, or create a dataset.

When **expected output** is built from annotation labels, the exported value is a JSON object keyed by label name, for example `{ "is_harmful": false, "tone": ["neutral"], "topics": ["safety", "policy"] }`. The same shape applies whether you select one label or multiple labels. Categorical labels are always exported as arrays of selected options, whether the label is single-select or multi-select.

{% collapsible-section #annotation-aggregation %}
#### How annotation values are aggregated across annotators

When multiple annotators have annotated the same trace, the value for each label is aggregated across them by consensus:

| Label type  | Aggregation                                                           |
| ----------- | --------------------------------------------------------------------- |
| Boolean     | Majority vote (ties break in favor of `true`)                         |
| Categorical | Intersection: the sorted set of options that every annotator selected |
| Score       | Average                                                               |
| Text        | List of responses                                                     |

For categorical labels (single-select or multi-select), the aggregated value is the sorted array of options that *every* annotator selected. If any annotator's selection differs, the value is an empty array. The result is always an array, even when only one annotator has annotated the trace.

**Example: categorical (consensus).** Three annotators rate `tone` and all agree:

- Annotator A: `polite`
- Annotator B: `polite`
- Annotator C: `polite`

Aggregated: `["polite"]`.

**Example: categorical (disagreement).** Three annotators rate `tone` and one differs:

- Annotator A: `polite`
- Annotator B: `rude`
- Annotator C: `polite`

Aggregated: `[]`. The intersection is empty because `rude` is not in every annotator's set.

**Example: categorical (multi-select).** Three annotators tag `topics` (each can pick multiple options):

- Annotator A: `["safety", "policy"]`
- Annotator B: `["safety", "billing"]`
- Annotator C: `["safety", "policy"]`

Aggregated: `["safety"]`. Only `safety` appears in every annotator's set; `policy` is missing from B's selection and `billing` is missing from A's and C's.

**Example: text.** Two annotators leave notes:

- Annotator A: `"Confusing phrasing"`
- Annotator B: `"Tone too casual"`

Aggregated: `["Confusing phrasing", "Tone too casual"]`. Every annotator's value is preserved.

Raw per-annotator values are preserved in each record's metadata, along with annotator identity. If the default consensus doesn't fit your workflow, you can recompute with a different strategy (for example, median, weighted vote, or reviewer pick).
{% /collapsible-section %}

Labels not selected as expected output are also included with each trace as metadata, along with any final scores and their assessment and reasoning.

See [Datasets](https://docs.datadoghq.com/llm_observability/improve/datasets.md) for more information about using datasets in offline experiments.

### Deleting queues{% #deleting-queues %}

To delete a queue:

1. Navigate to [AI Observability > Annotations](https://app.datadoghq.com/llm/annotations/queues).
1. Open the queue.
1. Click Delete in the Details panel.

{% alert level="info" %}
Deleting a queue removes the queue and label associations, but does not delete the underlying traces from Agent Observability. Traces remain accessible in Trace Explorer.
{% /alert %}

## Using the API{% #using-the-api %}

You can manage annotation queues programmatically. The following endpoints are available in the [Agent Observability API reference](https://docs.datadoghq.com/api/latest/agent-observability.md):

| Endpoint                                                                                                                                  | Description                                                                                                                                  |
| ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| [List annotation queues](https://docs.datadoghq.com/api/latest/agent-observability.md#list-agent-observability-annotation-queues)         | List all annotation queues in your organization.                                                                                             |
| [Create an annotation queue](https://docs.datadoghq.com/api/latest/agent-observability.md#create-an-agent-observability-annotation-queue) | Create an annotation queue. `name` and `project_id` are required. Include an optional `annotation_schema` to define labels at creation time. |
| [Update an annotation queue](https://docs.datadoghq.com/api/latest/agent-observability.md#update-an-agent-observability-annotation-queue) | Partially update a queue's `name`, `description`, or `annotation_schema`.                                                                    |
| [Delete an annotation queue](https://docs.datadoghq.com/api/latest/agent-observability.md#delete-an-agent-observability-annotation-queue) | Delete an annotation queue by ID.                                                                                                            |
| [Add interactions to a queue](https://docs.datadoghq.com/api/latest/agent-observability.md#add-annotation-queue-interactions)             | Add one or more traces to an annotation queue for review.                                                                                    |
| [Delete interactions from a queue](https://docs.datadoghq.com/api/latest/agent-observability.md#delete-annotation-queue-interactions)     | Remove specific interactions from a queue by interaction ID.                                                                                 |
| [Get annotated interactions](https://docs.datadoghq.com/api/latest/agent-observability.md#get-annotated-queue-interactions)               | Retrieve all interactions and their applied annotation labels for a queue.                                                                   |
| [Get label schema](https://docs.datadoghq.com/api/latest/agent-observability.md#get-annotation-queue-label-schema)                        | Retrieve the label schema configured for a queue.                                                                                            |
| [Update label schema](https://docs.datadoghq.com/api/latest/agent-observability.md#update-annotation-queue-label-schema)                  | Create or replace the label schema for a queue.                                                                                              |

## Data retention{% #data-retention %}

| Data              | Retention period                                                                     |
| ----------------- | ------------------------------------------------------------------------------------ |
| Traces in queues  | Not retained beyond your organization's span retention period, unless annotated      |
| Annotated traces  | 90 days from the time of annotation, or your span retention period if that is longer |
| Annotation labels | The same period as the trace they annotate                                           |

Annotating a trace extends its retention at no additional charge: a trace that would otherwise expire under a shorter span retention period is retained for 90 days from the time you annotate it. For details, see [Data Governance](https://docs.datadoghq.com/llm_observability/data_governance.md).

## Example workflows{% #example-workflows %}

{% collapsible-section
   open=null
   #example-error-analysis-and-failure-mode-discovery %}
### Error analysis and failure mode discovery

Review failed traces to identify recurring patterns and categorize how your application fails in production.

1. Filter traces in Trace Explorer for failed evaluations or specific error patterns
1. Manually select traces and add to an annotation queue
1. Annotators review traces and document failure types in free-form notes
1. Common patterns emerge: hallucinations in specific contexts, formatting issues, inappropriate refusals
1. Create categorical labels for identified failure modes and re-code traces
1. Use failure mode distribution to prioritize fixes

#### Queue configuration{% #queue-configuration %}

- **Labels**: Free-form notes, categorical `failure_type` label, pass/fail rating

- **Annotators**: Product managers, engineers, domain experts

{% /collapsible-section %}

{% collapsible-section
   open=null
   #example-validating-llm-as-a-judge-evaluations %}
### Validating LLM-as-a-Judge evaluators

Find traces where automated evaluators may be uncertain or incorrect, then have humans provide ground truth.

1. Sample evaluation results: all results, or a given score/threshold
1. Add selected traces to an annotation queue
1. Annotators review traces and provide human scores for the same criteria
1. Compare human labels to automated evaluation scores
1. Identify systematic disagreements (judge too strict, too lenient, or misunderstanding criteria)
1. Refine evaluator prompts based on disagreements

#### Queue configuration{% #queue-configuration-1 %}

- **Labels**: Numeric scores matching evaluation criteria (0-10), categorical `judge_accuracy` label, reasoning notes

- **Annotators**: Subject matter experts who understand evaluation criteria

{% /collapsible-section %}

{% collapsible-section open=null #example-golden-dataset-creation %}
### Golden dataset creation

Build benchmark datasets with human-verified labels for regression testing and continuous validation.

1. Sample diverse production traces from Trace Explorer (both good and bad examples)
1. Add traces to annotation queue
1. Annotators review and label traces across multiple quality dimensions
1. Add high-confidence, well-labeled examples to golden dataset
1. Use dataset for CI/CD regression testing of prompt changes
1. Continuously expand dataset with new edge cases

#### Queue configuration{% #queue-configuration-2 %}

- **Labels**: Multiple categorical labels covering quality dimensions, numeric scores, pass/fail rating, notes

- **Annotators**: Team of domain experts for consistency

{% /collapsible-section %}

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Learn about evaluation types](https://docs.datadoghq.com/llm_observability/investigate/evaluations.md)
- [Route traces into queues automatically with Automation Rules](https://docs.datadoghq.com/llm_observability/configure/automation_rules.md)
- [Run offline experiments to test improvements](https://docs.datadoghq.com/llm_observability/improve/offline_experiments.md)
- [Restrict a queue to specific teams or roles](https://docs.datadoghq.com/llm_observability/data_governance/access_control.md)
- [Annotate traces to improve LLM quality with Datadog LLM Observability](https://www.datadoghq.com/blog/automations-annotation-queues)
- [Agent Observability API reference](https://docs.datadoghq.com/api/latest/agent-observability.md)
