---
title: Progressive Rollouts and Canaries
description: >-
  Gradually roll out feature flags on a schedule with optional guardrail metrics
  for canary releases.
breadcrumbs: Docs > Feature Flags > Use Cases > Progressive Rollouts and Canaries
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Progressive Rollouts and Canaries

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

**Progressive rollouts** release new functionality gradually by incrementing the percentage of subjects exposed to the feature over time. **Canaries** are progressive rollouts that monitor guardrail metrics and automatically pause or stop when they detect regressions.

## Progressive rollouts{% #progressive-rollouts %}

### Configure a progressive rollout{% #configure-a-progressive-rollout %}

1. Navigate to your feature flag and open **Targeting Rules & Rollouts** for the target environment.
1. Click **Add Targeting Rule** and select **Add Rollout Steps** to create a multistep rollout.
1. Configure rollout steps:
   - Customize the percentage for each step, and add or delete steps as needed.
   - Change the delay between steps for a slower or faster rollout.
   - Click **Split Traffic** to roll out multiple variants at the same time.

{% image
   source="https://docs.dd-static.net/images/feature_flags/create-progressive-rollout.5b396e4fa5435b68f51400b545d24966.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/feature_flags/create-progressive-rollout.5b396e4fa5435b68f51400b545d24966.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Multistep progressive rollout configuration." /%}

### Start and control the rollout{% #start-and-control-the-rollout %}

1. **Enable** the flag in the environment so SDKs evaluate your targeting rules.
1. Click **Start Rollout** to kick off the progressive rollout.

{% image
   source="https://docs.dd-static.net/images/feature_flags/start-progressive-rollout.fa5d42b546dd9967d0ff7ae4a0c12ffa.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/feature_flags/start-progressive-rollout.fa5d42b546dd9967d0ff7ae4a0c12ffa.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Multistep progressive rollout display." /%}

To start the rollout automatically at a future date and time instead of clicking **Start Rollout**, [schedule a start time](https://docs.datadoghq.com/feature_flags/concepts/scheduled_rollouts.md) when you configure the rule.

After the rollout starts:

- Click **Pause Rollout** to stop progress temporarily.
- Click **Stop Rollout** to revert all progress on the rollout.

Monitor progress with evaluation tracking and configure notifications for rollout events.

## Canaries{% #canaries %}

A canary is a progressive rollout that includes **guardrail metrics**. Guardrail metrics measure key performance indicators (KPIs), such as error rate, latency, and long task count.

### How canaries work{% #how-canaries-work %}

When guardrail metrics are configured, the rollout monitors metrics in both groups:

- **Treatment**: Subjects receiving the variant you are rolling out
- **Control**: Subjects not receiving the treatment variant

When the canary detects a statistically significant adverse change in a guardrail metric, it automatically **pauses** or **stops** the rollout.

### Use APM guardrail metrics{% #use-apm-guardrail-metrics %}

{% alert level="info" %}
APM guardrail metrics for canary rollouts are in Preview. Contact your Datadog representative to request access.
{% /alert %}

APM guardrail metrics use retained spans to compare application performance between the control and treatment groups. You can monitor:

- Mean span duration
- Error rate
- P90 span duration
- Mean of a numeric span attribute

[APM trace enrichment](https://docs.datadoghq.com/feature_flags/guide/apm_trace_enrichment.md) is required before you add an APM metric to a canary. Enrichment records the flag allocation, variant, and subject on each trace so the canary can associate APM data with the correct group.

After you enable enrichment:

1. Verify that the application evaluates the feature flag during traced requests.
1. In Trace Explorer, confirm that root spans contain the decoded `@feature_flags.<flag_key>` attribute.
1. Create an APM metric with a span query that matches the requests you want to monitor.
1. Select the APM metric as a guardrail when you configure the canary.

Canary analysis uses the retained sample of matching traces. If the retained-trace volume is low, the canary waits until it has enough sampled subjects and events to make a decision. To increase the random retained sample, configure temporary trace retention for the canary. Temporary retention does not recover traces dropped before they reach Datadog.

{% image
   source="https://docs.dd-static.net/images/feature_flags/apm_canaries/apm-metric-create-flow-1.8076b3b487c2be569ebd49355f115d0b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/feature_flags/apm_canaries/apm-metric-create-flow-1.8076b3b487c2be569ebd49355f115d0b.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Metric creation flow with APM Spans selected and supported APM guardrail metric types, including P90 span duration." /%}

{% image
   source="https://docs.dd-static.net/images/feature_flags/apm_canaries/apm-metric-create-flow-2.d1db356308ae487824c42a546c1c8067.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/feature_flags/apm_canaries/apm-metric-create-flow-2.d1db356308ae487824c42a546c1c8067.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="APM metric creation flow with an APM span query and matching span preview." /%}

### Configure a canary rollout{% #configure-a-canary-rollout %}

1. Create a progressive rollout targeting rule as described in the Configure a progressive rollout section.
1. Add guardrail metrics to the rollout configuration.
1. For an APM guardrail, configure temporary trace retention if you want to increase the retained sample.
1. Choose whether guardrail failures should pause or stop the rollout.

{% image
   source="https://docs.dd-static.net/images/feature_flags/canary-rollout-config.28e4b9781c2cf30b9cc4d0b48f146667.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/feature_flags/canary-rollout-config.28e4b9781c2cf30b9cc4d0b48f146667.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Canary rollout configuration showing rollout steps with guardrail metrics and a control variant." /%}

{% image
   source="https://docs.dd-static.net/images/feature_flags/apm_canaries/apm-canary-metric-monitored-with-retention.1985face592cbdf6155163d3236a3490.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/feature_flags/apm_canaries/apm-canary-metric-monitored-with-retention.1985face592cbdf6155163d3236a3490.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Canary configuration with a P90 span duration guardrail, abort action, temporary APM trace retention, and a control variant." /%}

## Best practices{% #best-practices %}

- Configure notifications on the flag to be alerted when the rollout starts, pauses, or stops.
- Configure canary notifications before starting the rollout. Notifications alert you when a guardrail metric pauses or stops the rollout.
- Use evaluation tracking to monitor how many subjects are receiving each variant as the rollout progresses.
- For APM guardrails, confirm that the retained sample contains enough sampled subjects and events before increasing rollout exposure.