For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/tests/flaky_tests/auto_test_retries.md. A documentation index is available at /llms.txt.
This product is not supported for your selected Datadog site. ().

Overview

Test Optimization’s Auto Test Retries feature allows retrying failing tests up to N times to avoid failing your build due to flaky tests: a failing test case is retried either until it passes successfully or until there are no more retry attempts left (in which case the build fails).

Retries added by Auto Test Retries are not billed as test spans. For details, see CI/CD Optimization Billing.

Setup

Ensure Test Optimization is configured for your test runs.

Compatibility

dd-trace-java >= 1.34.0

The test framework compatibility is the same as Test Optimization Compatibility, with the exception of Scala Weaver.

Configuration

After you set up Test Optimization, configure Auto Test Retries in CI/CD Optimization settings. You can apply the setting at the organization, repository, or test service level.

Auto Test Retries toggle in CI/CD Settings.

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

  • DD_CIVISIBILITY_FLAKY_RETRY_ONLY_KNOWN_FLAKES - if this environment variable is set to true, only the test cases that Test Optimization considers flaky are retried.
  • DD_CIVISIBILITY_FLAKY_RETRY_COUNT - can be set to any non-negative number to change the maximum number of retries per test case.

Compatibility

dd-trace-js >= v5.19.0

Configuration

After you set up Test Optimization, configure Auto Test Retries in CI/CD Optimization settings. You can apply the setting at the organization, repository, or test service level.

Auto Test Retries toggle in CI/CD Settings.

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

  • DD_CIVISIBILITY_FLAKY_RETRY_ENABLED - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true).
  • DD_CIVISIBILITY_FLAKY_RETRY_COUNT - a non-negative number to change the maximum number of retries per test case (default: 5).

Compatibility

datadog-ci-rb >= 1.4.0

Configuration

After you set up Test Optimization, configure Auto Test Retries in CI/CD Optimization settings. You can apply the setting at the organization, repository, or test service level.

Auto Test Retries toggle in CI/CD Settings.

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

  • DD_CIVISIBILITY_FLAKY_RETRY_ENABLED - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true)
  • DD_CIVISIBILITY_FLAKY_RETRY_COUNT - a non-negative number to change the maximum number of retries per test case (default: 5).
  • DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

Compatibility

dd-trace-dotnet >= 3.4.0

Configuration

After you set up Test Optimization, configure Auto Test Retries in CI/CD Optimization settings. You can apply the setting at the organization, repository, or test service level.

Auto Test Retries toggle in CI/CD Settings.

By default, the feature retries any failing test case up to 5 times. Customize the Auto Test Retries with the following environment variables:

  • DD_CIVISIBILITY_FLAKY_RETRY_ENABLED - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true)
  • DD_CIVISIBILITY_FLAKY_RETRY_COUNT - a non-negative number to change the maximum number of retries per test case (default: 5).
  • DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

Compatibility

ddtest >= 1.14.0

Configuration

After you set up Test Optimization, configure Auto Test Retries in CI/CD Optimization settings. You can apply the setting at the organization, repository, or test service level.

Auto Test Retries toggle in CI/CD Settings.

By default, the feature retries each failing test case up to 5 times. Customize the Auto Test Retries with the following environment variables:

  • DD_CIVISIBILITY_FLAKY_RETRY_ENABLED - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true)
  • DD_CIVISIBILITY_FLAKY_RETRY_COUNT - a non-negative number to change the maximum number of retries per test case (default: 5).
  • DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

Compatibility

dd-trace-py >= 3.0.0 (pytest >= 7.2.0)

Configuration

After you set up Test Optimization, configure Auto Test Retries in CI/CD Settings. You can apply the setting at the organization, repository, or test service level.

Auto Test Retries toggle in CI/CD Settings.

The default behavior of the feature is to retry any failing test case up to five times. Tests that originally fail either the original setup, teardown, or fixtures in Pytest, are not retried.

You can fine tune this behavior with the following environment variables:

  • DD_CIVISIBILITY_FLAKY_RETRY_ENABLED - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true)
  • DD_CIVISIBILITY_FLAKY_RETRY_COUNT - a non-negative number to change the maximum number of retries per test case (default: 5).
  • DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

Dynamic Auto Test Retries

dd-trace-py >= 4.15.0

By default, Auto Test Retries applies the same retry limit to every failing test. Dynamic Auto Test Retries (Dynamic ATR) bases the number of retries on how long the test takes to run on its first attempt. Faster tests receive more retries and slower tests receive fewer.

The retry budget for a test is determined once, from the duration of its initial attempt, and applies to all of its retries. A test stops being retried as soon as a retry passes.

The default duration buckets are:

First attempt durationDefault retries
5 seconds or less10
More than 5 seconds, up to 10 seconds5
More than 10 seconds, up to 30 seconds3
More than 30 seconds, up to 5 minutes2
More than 5 minutes1

Every failing test is retried at least once, regardless of its duration.

To enable Dynamic Auto Test Retries, set the following environment variables:

  • DD_CIVISIBILITY_DYNAMIC_ATR_ENABLED - set to true to base the number of retries on the test’s first attempt duration instead of the flat DD_CIVISIBILITY_FLAKY_RETRY_COUNT limit (default: false) - the flat limit is ignored while Dynamic ATR is enabled. Auto Test Retries must be enabled in CI/CD Settings.
  • DD_CIVISIBILITY_DYNAMIC_ATR_BUCKETS - five comma-separated integers between 1 and 20 that override the default budgets in the previous table, ordered from the fastest to the slowest duration bucket (for example, 10,4,1,1,1). If this variable is unset or empty, the default budgets in the previous table are used. If DD_CIVISIBILITY_DYNAMIC_ATR_BUCKETS contains an invalid value, the library ignores it, logs a warning, and uses the default budgets.

Note: The session-level limit DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT still applies when Dynamic ATR is enabled.

Compatibility

dd-sdk-swift-testing >= 2.5.2

Configuration

After you set up Test Optimization, configure Auto Test Retries in CI/CD Optimization settings. You can apply the setting at the organization, repository, or test service level.

Auto Test Retries toggle in CI/CD Settings.

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

  • DD_CIVISIBILITY_FLAKY_RETRY_ENABLED - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true)
  • DD_CIVISIBILITY_FLAKY_RETRY_COUNT - a non-negative number to change the maximum number of retries per test case (default: 5).
  • DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

Failed Test Replay

Failed Test Replay is only supported for Java, JavaScript, and .NET.

In addition to automatically retrying failed tests, Failed Test Replay allows you to see local variable data in the topmost frame of the test error’s stack trace.

Failed Test Replay requires Auto Test Retries to be enabled, because it captures variable data from the retried test runs.

Enable Failed Test Replay in CI/CD Optimization settings under Mitigation > Failed Test Replay. You can apply the setting at the organization, repository, or test service level.

Create a logs index

Failed Test Replay creates logs that are sent to Datadog and appear alongside your regular application logs.

If you use Exclusion filters, ensure Failed Test Replay logs are not filtered:

  1. Create a logs index and configure it to the desired retention with no sampling.
  2. Set the filter to match on the source:dd_debugger tag. All Failed Test Replay logs have this source.
  3. Ensure that the new index takes precedence over any other index with filters that match that tag, because the first match wins.

After you enable this feature, you can see local variable data in failed tests:

Failed Test Replay.

Known limitations

jest-image-snapshot is incompatible with jest.retryTimes unless customSnapshotIdentifier is passed (see jest-image-snapshot docs) to toMatchImageSnapshot. Therefore, auto test retries do not work unless customSnapshotIdentifier is used.

Explore results in the Test Optimization Explorer

You can query the retried tests in the Test Optimization Explorer: they have the @test.is_retry tag set to true (some of them may also have the @test.is_new set to true, which indicates they have been retried by the Early Flakiness Detection feature).

Troubleshooting

If you suspect there are any issues with Auto Test Retries, open CI/CD Optimization settings, find your repository or service, and toggle Auto Test Retries off.

Exclude individual tests from retries

Use a code annotation to exclude individual tests from Auto Test Retries, Early Flake Detection, and Attempt to Fix retries. Explicit retries configured in the test framework remain available. See Control Test Automation with Code Annotations for examples and version requirements.

Further Reading