---
title: Prompt Experimentation
description: >-
  Compare managed prompt versions with A/B tests and deploy versions with
  Guarded Rollouts.
breadcrumbs: >-
  Docs > Agent Observability > Configure Agent Observability > Prompt
  Experimentation
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Prompt Experimentation

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Test a prompt change on live traffic before making it the default. Use an **A/B test** to compare versions against an outcome, such as checkout conversion. Use a **Guarded Rollout** to introduce a version gradually and pause if Datadog detects a regression.

{% alert level="info" %}
**Preview:** Prompt Experimentation is available in Preview. To request access, contact [Datadog Support](https://www.datadoghq.com/support/) or your Customer Success Manager. You can request access to A/B testing and Guarded Rollouts independently.
{% /alert %}

- **Which prompt performs better?** Run an A/B test.
- **How do I introduce a change gradually?** Use a Guarded Rollout.

## Before you begin{% #before-you-begin %}

You need a managed prompt with at least two versions, a version deployed to the environment you want to use, and an [Experiment metric](https://docs.datadoghq.com/experiments/defining_metrics.md) that measures your outcome. Check the permissions for your workflow.

**Permissions:** A/B testing requires permissions in Product Analytics and Feature Flags, in addition to LLM Observability. Preview access does not grant these permissions. Ask your Datadog administrator to check your role before starting.

**First time?** Complete the one-time application setup before starting either workflow. It requires Python `ddtrace[openfeature]` 4.15.0 or later, a reachable Datadog Agent, and matching user identifiers in prompt requests and outcome events. Agentless mode is not supported.

## Run an A/B test{% #run-an-ab-test %}

For example, compare two versions of a checkout assistant to see which produces a higher checkout conversion rate.

1. Open the prompt and click Set up A/B test.
1. Select an environment and click Create draft & continue. This opens the test in Product Analytics. Creating a draft does not change live traffic.
1. Choose the prompt versions, audience, traffic split, and Primary metric. Under Calculate metrics by, choose the subject type that matches your application's `targeting_key`—for example, **User** when you pass a user ID.
1. Start the test. Check Flag & Exposures to confirm that it receives assignments, then [compare the results](https://docs.datadoghq.com/experiments/reading_results.md).

Return to A/B tests using this prompt on the prompt page to open the results or finish a draft. Configure and conclude the test in Product Analytics; see [Plan and Launch Experiments](https://docs.datadoghq.com/experiments/plan_and_launch_experiments.md) for detailed instructions.

{% image
   source="https://docs.dd-static.net/images/llm_observability/monitoring/prompt-ab-tests.a0d0b7201863adc7d80da8864b5a238c.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/monitoring/prompt-ab-tests.a0d0b7201863adc7d80da8864b5a238c.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="A/B tests using a prompt, showing a running test comparing concise and detailed answers, its versions, primary metric and audience, and a draft with a Finish Setup button." /%}

**Choosing a winner does not deploy it automatically.** Conclude the test, then return to the prompt and deploy the version you choose. Users outside the test's audience continue to follow the environment's other serving rules.

## Use a Guarded Rollout{% #use-a-guarded-rollout %}

For example, introduce an updated checkout assistant while watching checkout conversion. The deployed version is the **baseline**; the new version is the **candidate**. If the environment has no deployed version, deploy one immediately first to establish a baseline.

1. Open the candidate version and click Deploy Version.
1. Select the environment in Deploy to.
1. Expand Deployment strategy, choose Guarded rollout, and select a Guardrail metric.
1. Use Inspect metric to check its definition and confirm whether higher or lower values are better.
1. Review the proposed change and click Deploy Version to start the rollout.

{% image
   source="https://docs.dd-static.net/images/llm_observability/monitoring/prompt-guarded-rollout-setup.1b392f0b1ef432c156972f27c23a50f0.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/monitoring/prompt-guarded-rollout-setup.1b392f0b1ef432c156972f27c23a50f0.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Deploy version 3 dialog with Production selected, Guarded rollout enabled, and Checkout completion rate selected as the guardrail metric, with an Inspect metric link." /%}

### Follow its progress{% #follow-its-progress %}

Open Active rollouts on the prompt page to follow the candidate's exposure. Exposure increases automatically while the rollout is running. Use Pause to hold the current split and Resume to continue.

If a guardrail detects a regression, the rollout pauses under Needs attention. Open the metric to investigate, then choose:

- Resume anyway to continue exposing the candidate.
- Stop to end the rollout and restore its eligible traffic to the baseline.

**A pause does not roll back the candidate:** the current traffic split stays in place until you act. Stopping affects only that environment's rollout, not other environments or A/B tests. To start again after stopping, deploy a version again.

{% image
   source="https://docs.dd-static.net/images/llm_observability/monitoring/prompt-guarded-rollout-paused.7b7956df5038357c9b6f1b9564588586.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/monitoring/prompt-guarded-rollout-paused.7b7956df5038357c9b6f1b9564588586.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="A prompt page showing a Staging rollout under Needs Attention, paused at 25 percent exposure to version 2, with the Checkout Completion Rate metric and Resume Anyway and Stop actions." /%}

When the rollout finishes, the candidate receives all traffic eligible for that rollout and disappears from Active rollouts. Other targeting rules may still serve different versions.

{% alert level="warning" %}
Guardrails depend on available outcome data. Sparse, missing, or unmatched data may not trigger a pause. Reaching full exposure does not prove that the candidate is better or safe.
{% /alert %}

## Set up your application{% #set-up-your-application %}

Complete this setup once for either workflow. If your application already sends prompt assignments and matching outcomes, go directly to A/B testing or Guarded Rollouts.

### Connect the application{% #connect-the-application %}

1. Complete the Prompt Management [prerequisites](https://docs.datadoghq.com/llm_observability/configure/prompt_management.md#prerequisites).
1. Install the supported Python SDK: `pip install --upgrade "ddtrace[openfeature]>=4.15.0"`.
1. Set `DD_ENV` to the environment you want to use.
1. Ensure the application can reach a [Datadog Agent](https://docs.datadoghq.com/getting_started/agent.md). Prompt Experimentation sends assignment data through the Agent; do not enable Agentless mode (`DD_LLMOBS_AGENTLESS_ENABLED=1`).

### Match prompt requests to outcomes{% #match-prompt-requests-to-outcomes %}

Use the same user ID when fetching the prompt and recording the outcome. This lets Datadog connect a user's outcome to the prompt version they received.

```python
from ddtrace.llmobs import LLMObs

prompt = LLMObs.get_prompt(
    "checkout-assistant",
    targeting_key=str(current_user.id),
    fallback="You are a helpful checkout assistant.",
)
```

Use a stable, non-empty string for `targeting_key`. Do not pass an exact `version`: that bypasses environment assignment and does not record an experiment exposure.

For example, if your checkout metric uses RUM events and the **User** subject type, identify the browser user with the same ID:

```javascript
datadogRum.setUser({
  id: String(currentUser.id),
})
```

RUM is only needed when it supplies your chosen outcome. For other outcome sources, or to compare organizations rather than users, configure the matching [subject type](https://docs.datadoghq.com/experiments/concepts/subject_types.md). Outcomes must occur after the user's first exposure to the prompt version to be attributed to it.

### Choose an outcome metric{% #choose-an-outcome-metric %}

Create or select an [Experiment metric](https://docs.datadoghq.com/experiments/defining_metrics.md) under [Product Analytics > Experiments > Metrics](https://app.datadoghq.com/product-analytics/experimentation-metrics). Metrics Explorer metrics and monitors are not selectable here.

For example, for a checkout conversion metric, set Desired metric direction to Higher is better. Ensure the outcome events use the same user ID as the prompt request. Confirm that outcome events are arriving before relying on the metric to compare versions or pause a rollout.

{% section title="Instructions for AI agents" %}**Note for AI agents:** Use the application's existing stable subject identifier as a string for `targeting_key`, matching the Experiment subject. Ask if the identity or outcome source is unclear. Do not install RUM unless the outcome uses RUM. Preserve the existing fallback, prompt variables, provider, model, and application behavior. Do not request an exact prompt version. Creating or editing metrics, starting experiments, deploying versions, and controlling rollouts require an explicit user request; when asked only to prepare code, report the remaining Datadog UI steps.{% /section %}

## Permissions{% #permissions %}

In addition to the [Prompt Management prerequisites](https://docs.datadoghq.com/llm_observability/configure/prompt_management.md#prerequisites), your role needs:

| Action                            | Required permission                                               |
| --------------------------------- | ----------------------------------------------------------------- |
| Set up an A/B test from a prompt  | Product Analytics Experiments Write                               |
| Start a Guarded Rollout           | LLM Observability Write and Feature Flags Environment Config Read |
| Pause or resume a Guarded Rollout | Feature Flags Config Write                                        |
| Stop a Guarded Rollout            | LLM Observability Read and Write                                  |

To configure and launch an A/B test in Product Analytics, see [Product Analytics permissions](https://docs.datadoghq.com/account_management/rbac/permissions.md#product-analytics) and [Feature Flags permissions](https://docs.datadoghq.com/account_management/rbac/permissions.md#feature-flags). If your environment requires approvals, request approval in Feature Flags before changing live traffic.

## Troubleshooting{% #troubleshooting %}

### I cannot start a test or rollout{% #i-cannot-start-a-test-or-rollout %}

- **Controls are missing or disabled:** confirm access to the relevant Preview and the required permissions. A/B setup also requires two prompt versions and an enabled environment whose serving configuration can be loaded.
- **A guarded deployment is rejected:** establish a baseline first. Resolve the conflict shown for the environment, such as an active all-user experiment or a forced version override, before trying again.
- **You can create a draft but cannot configure or launch the test:** check your [Product Analytics permissions](https://docs.datadoghq.com/account_management/rbac/permissions.md#product-analytics) and [Feature Flags permissions](https://docs.datadoghq.com/account_management/rbac/permissions.md#feature-flags) with your Datadog administrator, along with any required environment approvals. Confirm that the test has a primary metric selected.

### I do not see assignments or results{% #i-do-not-see-assignments-or-results %}

- **Only the fallback is returned, or assignments are missing:** check the SDK installation, `DD_ENV`, Agent connectivity, and that retrieval does not specify `version`.
- **Outcomes are missing:** check matching subject IDs, metric scope, and that events occur after exposure. Allow enough data to accumulate; see [Experiment Diagnostics](https://docs.datadoghq.com/experiments/diagnostics.md).

### I cannot control or inspect a rollout{% #i-cannot-control-or-inspect-a-rollout %}

- **Pause, Resume, or Stop is unavailable:** check the permissions. Stop requires different permissions from Pause and Resume.
- **Status cannot be loaded:** refresh, then check the reported error or contact Datadog Support. An unavailable status does not mean the rollout stopped.

### I am deploying again or to several environments{% #i-am-deploying-again-or-to-several-environments %}

Review each environment's deployment strategy: saved guarded settings may already be selected. Immediate environments switch without a rollout. Deploying another version can replace an active rollout; review the replacement warning before confirming.

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Prompt Management](https://docs.datadoghq.com/llm_observability/configure/prompt_management.md)
- [Create Experiment Metrics](https://docs.datadoghq.com/experiments/defining_metrics.md)
- [Read Experiment Results](https://docs.datadoghq.com/experiments/reading_results.md)