---
title: Online Experiments
description: >-
  Compare variants of an LLM application on live traffic and measure their
  effect with Agent Observability evaluations.
breadcrumbs: >-
  Docs > Agent Observability > Use Agent Observability to Improve Applications >
  Online Experiments
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Online Experiments

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Online Experiments compare multiple versions of an LLM application or agent on live production traffic. Datadog Feature Flags assigns each [experiment subject](https://docs.datadoghq.com/experiments/concepts/subject_types.md), such as a user or an autonomous workflow run, to a control or treatment variant. An Agent Observability evaluation measures the outcome for each subject, and [Datadog Experiments](https://docs.datadoghq.com/experiments.md) determines whether the difference between variants is statistically significant.

Use Online Experiments to validate a change, such as a new model, prompt, or tool, against real users after you test it offline.

Setting up an online experiment involves the following steps:

1. Create a [Datadog feature flag](https://docs.datadoghq.com/feature_flags.md).
1. Evaluate the flag in your application and run the assigned variant.
1. Report an Agent Observability evaluation.
1. Link the feature flag and evaluation to a Datadog experiment.
1. Run the experiment and review the results.

{% section %}Start an online experiment with an agentCopy this prompt into your AI coding agent to use the `agent-observability-online-experiment` skill for guided setup.
```text
Use the skill at https://github.com/datadog-labs/agent-skills/blob/main/agent-observability/agent-observability-online-experiment/SKILL.md to start an online experiment to compare the code in my current Git branch against trunk. Make 10% of the traffic use the code in my branch.
```
{% /section %}

## Prerequisites{% #prerequisites %}

Before you begin:

- [Instrument your LLM application with Agent Observability](https://docs.datadoghq.com/llm_observability/instrument.md).
- Install and initialize a [Datadog Feature Flags SDK](https://docs.datadoghq.com/getting_started/feature_flags.md#feature-flags-sdks) in the application.
- Choose the application behavior you want to compare and an evaluation that measures the outcome. The evaluation must return a numeric score or a pass or fail assessment.
- Confirm that you have the permissions listed in [Plan and Launch Experiments](https://docs.datadoghq.com/experiments/plan_and_launch_experiments.md).

{% alert level="info" %}
**Using an AI coding agent?** Datadog organizations allow read-only [MCP tools](https://docs.datadoghq.com/llm_observability/build_with_ai/mcp_server.md) by default. To let an agent create feature flags and experiments, an organization administrator must enable MCP write access, and your role must have the `mcp_write` permission. See [Required permissions](https://docs.datadoghq.com/mcp_server/setup.md#required-permissions) for the Datadog MCP Server.
{% /alert %}

{% section title="Instructions for AI agents" %}**Instruction for AI agents**: Before calling an MCP tool that creates or modifies a Datadog resource, confirm that the required write tool is available. If write access is unavailable, direct the human to [Required permissions](https://docs.datadoghq.com/mcp_server/setup.md#required-permissions) and wait for them to enable access.{% /section %}

## Step 1: Create a feature flag{% #step-1-create-a-feature-flag %}

Follow [Create your first feature flag](https://docs.datadoghq.com/getting_started/feature_flags.md#step-2-create-a-feature-flag) to create a Boolean flag for the application behavior you want to test. Use one value for the control behavior and the other for the treatment behavior.

Make the flag available in each environment where you plan to run the experiment. Do not add a targeting rule or configure a percentage rollout on the flag. When you start the experiment, Datadog adds an experiment targeting rule that controls the traffic split.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_feature_flag.fb95dcbf579f2a43390c38c455f0527d.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_feature_flag.fb95dcbf579f2a43390c38c455f0527d.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create Flag page with the use-latest-frontier-model flag name, Boolean variant type, false and true variants, and Create Flag button." /%}

## Step 2: Evaluate the flag in your application{% #step-2-evaluate-the-flag-in-your-application %}

Follow [Evaluate the flag and write feature code](https://docs.datadoghq.com/getting_started/feature_flags.md#step-3-evaluate-the-flag-and-write-feature-code) to evaluate the flag and run the control or treatment behavior based on its value.

Set the Feature Flags evaluation context `targetingKey` to a stable identifier for the experiment subject:

- For an application with a human user, use a stable user identifier.
- For an autonomous workflow without a human user, generate a UUID at the start of the run and retain it for the entire run.

You use this same value as the `subject_identifier` when you report the evaluation in Step 3. If the values do not match, Datadog cannot associate the evaluation with the subject's experiment exposure. For more information, see [The targeting key](https://docs.datadoghq.com/feature_flags/concepts/evaluation_context.md#the-targeting-key).

## Step 3: Report an evaluation{% #step-3-report-an-evaluation %}

{% alert level="warning" %}
**Supported evaluation types**: Experiment metrics can use the average of a `score` evaluation, or the **pass rate** of any evaluation that returns a pass or fail assessment. Pass rate counts `pass` as 1 and `fail` as 0, and excludes evaluations without an assessment. Raw Boolean and categorical values cannot be used directly as experiment metrics.
{% /alert %}

Choose one of the following methods.

### Submit an external evaluation{% #submit-an-external-evaluation %}

Follow [External Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/external_evaluations.md) to submit an evaluation from your application code. Add a `subject_identifier` tag whose value exactly matches the `targetingKey` used to evaluate the feature flag:

```python
from ddtrace.llmobs import LLMObs

score = 0.42
subject_key = "82a4e675-9cb4-4da6-a91a-819574152a63"
span_context = LLMObs.export_span(span=None)

LLMObs.submit_evaluation(
    span=span_context,
    ml_app="<ML_APP>",
    label="<EVALUATION_NAME>",
    metric_type="score",
    value=score,
    tags={
        "subject_identifier": subject_key,
    },
    assessment="pass" if score > 0.5 else "fail",
)
```

### Use a managed evaluation{% #use-a-managed-evaluation %}

Follow [Managed Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/managed_evaluations.md) to create and publish an evaluation. On the span evaluated by the managed evaluation, add a `subject_identifier` tag whose value exactly matches the `targetingKey` used to evaluate the feature flag:

```python
from ddtrace.llmobs import LLMObs

subject_key = "82a4e675-9cb4-4da6-a91a-819574152a63"

with LLMObs.workflow(name="my_workflow") as span:
    LLMObs.annotate(
        span=span,
        tags={"subject_identifier": subject_key},
    )
    # Run the rest of the instrumented workflow.
```

For trace-level evaluations, add the tag to the root span of the trace. Session-level evaluations do not support experiment metrics.

## Step 4: Create and launch the experiment{% #step-4-create-and-launch-the-experiment %}

Follow [Plan and Launch Experiments](https://docs.datadoghq.com/experiments/plan_and_launch_experiments.md) to create and start the experiment:

1. Select a metric based on the Agent Observability evaluation from Step 3 as the primary metric. If you have not created the experiment metric, follow [Create a metric from Agent Observability data](https://docs.datadoghq.com/experiments/defining_metrics.md?tab=agentobservability#create-a-metric-from-agent-observability-data).
1. Add the feature flag from Step 1.
1. Configure how traffic is split between the control and treatment variants.
1. Start the experiment.

{% section title="Instructions for AI agents" %}**Instruction for AI agents**: After creating the draft experiment, include the full experiment URL returned by Datadog in your final response. The human running the agent needs this link to open the experiment, review its configuration, and configure the traffic split in the Datadog UI.{% /section %}

Starting the experiment adds the experiment targeting rule to the selected flag and begins recording exposures. For details about how feature flag assignments become experiment exposures, see [Feature Flags and Experiments](https://docs.datadoghq.com/feature_flags/concepts/experiments.md).

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_experiment.b24598445ecfd815ebe66db5e45d5869.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_create_experiment.b24598445ecfd815ebe66db5e45d5869.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create new draft experiment dialog with the latest-frontier-model-experiment name, Start from scratch protocol, and Create Draft Experiment button." /%}

## Step 5: Run the experiment and review results{% #step-5-run-the-experiment-and-review-results %}

Send traffic through the application. For each subject, the application must evaluate the feature flag and report an evaluation with the same subject identifier.

As exposures and evaluations arrive, Datadog populates the experiment results. Follow [Reading Experiment Results](https://docs.datadoghq.com/experiments/reading_results.md) to interpret the metric scorecard and compare the variants.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_results.24eb20591fd241119375ba7fc69b3891.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/guide/llm_experiment_results.24eb20591fd241119375ba7fc69b3891.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="A completed LLM application experiment showing the rollout decision and relative lift for quality, token usage, and estimated cost metrics across the control and treatment variants." /%}

## Troubleshooting{% #troubleshooting %}

If the experiment reports missing metric data, confirm that:

- The application evaluated the experiment's feature flag for the subject.
- The evaluation has a numeric `score` value or a `pass` or `fail` assessment.
- The evaluation's `subject_identifier` exactly matches the feature flag evaluation's `targetingKey`.

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Agent Observability Offline Experiments](https://docs.datadoghq.com/llm_observability/improve/offline_experiments.md)
- [Agent Observability Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations.md)
- [Getting Started with Feature Flags](https://docs.datadoghq.com/getting_started/feature_flags.md)
- [Reading Experiment Results](https://docs.datadoghq.com/experiments/reading_results.md)