---
title: Run an Online Experiment on an LLM Application
description: >-
  Compare variants of an LLM application and measure their effect with Agent
  Observability evaluation scores.
breadcrumbs: >-
  Docs > Experiments > Experiments Guides > Run an Online Experiment on an LLM
  Application
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Run an Online Experiment on an LLM Application

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

{% callout %}
# Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com

{% callout %}
##### Join the Preview!

Online experiments for Agent Observability are in Preview. Contact your Datadog representative to request access.
{% /callout %}

{% /callout %}

## Overview{% #overview %}

Use a Datadog online experiment to compare two versions of an LLM application on live traffic. Datadog Feature Flags assigns each [experiment subject](https://docs.datadoghq.com/experiments/concepts/subject_types.md), such as a user or an autonomous workflow run, to a variant. An Agent Observability evaluation score measures the outcome for each subject.

This guide covers the LLM-specific configuration for the following workflow:

1. Create a Datadog feature flag.
1. Evaluate the flag in your application and run the assigned variant.
1. Report a numeric Agent Observability evaluation score.
1. Link the feature flag and evaluation score to a Datadog experiment.
1. Run the experiment and review the results.

{% section %}Start an online experiment with an agentCopy this prompt into your AI coding agent to use the `agent-observability-online-experiment` skill for guided setup.
```text
Use the skill at https://github.com/datadog-labs/agent-skills/blob/main/agent-observability/agent-observability-online-experiment/SKILL.md to start an online experiment to compare the code in my current Git branch against trunk. Make 10% of the traffic use the code in my branch.
```
{% /section %}

## Prerequisites{% #prerequisites %}

Before you begin:

- [Instrument your LLM application with Agent Observability](https://docs.datadoghq.com/llm_observability/instrument.md).
- Install and initialize a [Datadog Feature Flags SDK](https://docs.datadoghq.com/getting_started/feature_flags.md#feature-flags-sdks) in the application.
- Choose the application behavior you want to compare and a numeric evaluation score that measures the outcome.
- Confirm that you have the permissions listed in [Plan and Launch Experiments](https://docs.datadoghq.com/experiments/plan_and_launch_experiments.md).

{% alert level="info" %}
**Using an AI coding agent?** Datadog organizations allow read-only MCP tools by default. To let an agent create feature flags and experiments, an organization administrator must enable MCP write access, and your role must have the `mcp_write` permission. See [Required permissions](https://docs.datadoghq.com/mcp_server/setup.md#required-permissions) for the Datadog MCP Server.
{% /alert %}

{% section title="Instructions for AI agents" %}**Instruction for AI agents**: Before calling an MCP tool that creates or modifies a Datadog resource, confirm that the required write tool is available. If write access is unavailable, direct the human to [Required permissions](https://docs.datadoghq.com/mcp_server/setup.md#required-permissions) and wait for them to enable access.{% /section %}

## Step 1: Create a feature flag{% #step-1-create-a-feature-flag %}

Follow [Create your first feature flag](https://docs.datadoghq.com/getting_started/feature_flags.md#step-2-create-a-feature-flag) to create a Boolean flag for the application behavior you want to test. Use one value for the control behavior and the other for the treatment behavior.

Make the flag available in each environment where you plan to run the experiment. Do not add a targeting rule or configure a percentage rollout on the flag. When you start the experiment, Datadog adds an experiment targeting rule that controls the traffic split.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_feature_flag.fb95dcbf579f2a43390c38c455f0527d.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_feature_flag.fb95dcbf579f2a43390c38c455f0527d.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create Flag page with the use-latest-frontier-model flag name, Boolean variant type, false and true variants, and Create Flag button." /%}

## Step 2: Evaluate the flag in your application{% #step-2-evaluate-the-flag-in-your-application %}

Follow [Evaluate the flag and write feature code](https://docs.datadoghq.com/getting_started/feature_flags.md#step-3-evaluate-the-flag-and-write-feature-code) to evaluate the flag and run the control or treatment behavior based on its value.

Set the Feature Flags evaluation context `targetingKey` to a stable identifier for the experiment subject:

- For an application with a human user, use a stable user identifier.
- For an autonomous workflow without a human user, generate a UUID at the start of the run and retain it for the entire run.

You use this same value as the `subject_identifier` when you report the evaluation score in Step 3. If the values do not match, Datadog cannot associate the score with the subject's experiment exposure. For more information, see [The targeting key](https://docs.datadoghq.com/feature_flags/concepts/evaluation_context.md#the-targeting-key).

## Step 3: Report an evaluation score{% #step-3-report-an-evaluation-score %}

{% alert level="warning" %}
**Supported evaluation type**: Online experiments for Agent Observability support only evaluations with a `score` metric type. Boolean, categorical, and other evaluation types cannot be used as experiment metrics.
{% /alert %}

Choose one of the following methods.

### Submit an external evaluation{% #submit-an-external-evaluation %}

Follow [External Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/external_evaluations.md) to submit an evaluation from your application code. Add a `subject_identifier` tag whose value exactly matches the `targetingKey` used to evaluate the feature flag:

```python
from ddtrace.llmobs import LLMObs

score = 0.42
subject_key = "82a4e675-9cb4-4da6-a91a-819574152a63"
span_context = LLMObs.export_span(span=None)

LLMObs.submit_evaluation(
    span=span_context,
    ml_app="<ML_APP>",
    label="<EVALUATION_NAME>",
    metric_type="score",
    value=score,
    tags={
        "subject_identifier": subject_key,
    },
    assessment="pass" if score > 0.5 else "fail",
)
```

### Use a managed evaluation{% #use-a-managed-evaluation %}

Follow [Managed Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/managed_evaluations.md) to create and publish an evaluation. On the span evaluated by the managed evaluation, add a `subject_identifier` tag whose value exactly matches the `targetingKey` used to evaluate the feature flag:

```python
from ddtrace.llmobs import LLMObs

subject_key = "82a4e675-9cb4-4da6-a91a-819574152a63"

with LLMObs.workflow(name="my_workflow") as span:
    LLMObs.annotate(
        span=span,
        tags={"subject_identifier": subject_key},
    )
    # Run the rest of the instrumented workflow.
```

## Step 4: Create and launch the experiment{% #step-4-create-and-launch-the-experiment %}

Follow [Plan and Launch Experiments](https://docs.datadoghq.com/experiments/plan_and_launch_experiments.md) to create and start the experiment:

1. Select the Agent Observability evaluation score from the Step 3 section as the primary metric. If you have not created the experiment metric, follow [Create a metric from Agent Observability data](https://docs.datadoghq.com/experiments/defining_metrics.md?tab=agentobservability#create-a-metric-from-agent-observability-data).
1. Add the feature flag from the Step 1 section.
1. Configure how traffic is split between the control and treatment variants.
1. Start the experiment.

{% section title="Instructions for AI agents" %}**Instruction for AI agents**: After creating the draft experiment, include the full experiment URL returned by Datadog in your final response. The human running the agent needs this link to open the experiment, review its configuration, and configure the traffic split in the Datadog UI.{% /section %}

Starting the experiment adds the experiment targeting rule to the selected flag and begins recording exposures. For details about how feature flag assignments become experiment exposures, see [Feature Flags and Experiments](https://docs.datadoghq.com/feature_flags/concepts/experiments.md).

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_experiment.b24598445ecfd815ebe66db5e45d5869.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_experiment.b24598445ecfd815ebe66db5e45d5869.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create new draft experiment dialog with the latest-frontier-model-experiment name, Start from scratch protocol, and Create Draft Experiment button." /%}

## Step 5: Run the experiment and review results{% #step-5-run-the-experiment-and-review-results %}

Send traffic through the application. For each subject, the application must evaluate the feature flag and report an evaluation score with the same subject identifier.

As exposures and evaluation scores arrive, Datadog populates the experiment results. Follow [Reading Experiment Results](https://docs.datadoghq.com/experiments/reading_results.md) to interpret the metric scorecard and compare the variants.

If the experiment reports missing metric data, confirm that:

- The application evaluated the experiment's feature flag for the subject.
- The evaluation has a numeric `score` value.
- The evaluation's `subject_identifier` exactly matches the feature flag evaluation's `targetingKey`.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_results.24eb20591fd241119375ba7fc69b3891.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_results.24eb20591fd241119375ba7fc69b3891.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="A completed LLM application experiment showing the rollout decision and relative lift for quality, token usage, and estimated cost metrics across the control and treatment variants." /%}

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Getting Started with Feature Flags](https://docs.datadoghq.com/getting_started/feature_flags.md)
- [Agent Observability Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations.md)
- [Reading Experiment Results](https://docs.datadoghq.com/experiments/reading_results.md)