---
title: Auto Test Retries
description: Retry failing test cases to avoid failing the build because of flaky tests.
breadcrumbs: >-
  Docs > Test Optimization in Datadog > Working with Flaky Tests > Auto Test
  Retries
---

# Auto Test Retries

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). ().
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Test Optimization's Auto Test Retries feature allows retrying failing tests up to N times to avoid failing your build due to flaky tests: a failing test case is retried either until it passes successfully or until there are no more retry attempts left (in which case the build fails).

## Setup{% #setup %}

Ensure [Test Optimization](https://docs.datadoghq.com/tests/setup/) is configured for your test runs.

{% tab title="Java" %}
### Compatibility{% #compatibility %}

`dd-trace-java >= 1.34.0`

The test framework compatibility is the same as [Test Optimization Compatibility](https://docs.datadoghq.com/tests/setup/java/#compatibility), with the exception of `Scala Weaver`.

### Configuration{% #configuration %}

After you have set up Test Optimization, you can configure Auto Test Retries from the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization).

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/auto_test_retries_test_settings-2.34116b38e90672dcfa53d96c4a810aa7.png?auto=format"
   alt="Auto Test Retries in Test Service Settings." /%}

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

- `DD_CIVISIBILITY_FLAKY_RETRY_ONLY_KNOWN_FLAKES` - if this environment variable is set to `true`, only the test cases that Test Optimization considers [flaky](https://docs.datadoghq.com/tests/flaky_test_management/) are retried.
- `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - can be set to any non-negative number to change the maximum number of retries per test case.

{% /tab %}

{% tab title="JavaScript" %}
### Compatibility{% #compatibility %}

`dd-trace-js >= v5.19.0`

### Configuration{% #configuration %}

After you have set up Test Optimization, you can configure Auto Test Retries from the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization).

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/auto_test_retries_test_settings-2.34116b38e90672dcfa53d96c4a810aa7.png?auto=format"
   alt="Auto Test Retries in Test Service Settings." /%}

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

- `DD_CIVISIBILITY_FLAKY_RETRY_ENABLED` - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true).
- `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - a non-negative number to change the maximum number of retries per test case (default: 5).

{% /tab %}

{% tab title="Ruby" %}
### Compatibility{% #compatibility %}

`datadog-ci-rb >= 1.4.0`

### Configuration{% #configuration %}

After you have set up Test Optimization, you can configure Auto Test Retries from the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization).

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/auto_test_retries_test_settings-2.34116b38e90672dcfa53d96c4a810aa7.png?auto=format"
   alt="Auto Test Retries in Test Service Settings." /%}

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

- `DD_CIVISIBILITY_FLAKY_RETRY_ENABLED` - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true)
- `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - a non-negative number to change the maximum number of retries per test case (default: 5).
- `DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT` - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

{% /tab %}

{% tab title=".NET" %}
### Compatibility{% #compatibility %}

`dd-trace-dotnet >= 3.4.0`

### Configuration{% #configuration %}

After you set up Test Visibility, you can configure Auto Test Retries from the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization).

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/auto_test_retries_test_settings-2.34116b38e90672dcfa53d96c4a810aa7.png?auto=format"
   alt="Auto Test Retries enabled in Test Service Settings." /%}

By default, the feature retries any failing test case up to 5 times. Customize the Auto Test Retries with the following environment variables:

- `DD_CIVISIBILITY_FLAKY_RETRY_ENABLED` - set to `0` or `false` to explicitly disable retries even if the remote setting is enabled (default: true)
- `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - a non-negative number to change the maximum number of retries per test case (default: 5).
- `DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT` - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

{% /tab %}

{% tab title="Go" %}
### Compatibility{% #compatibility %}

`orchestrion >= 0.9.4` + `dd-trace-go >= 1.69.1`

### Configuration{% #configuration %}

After you set up Test Visibility, you can configure Auto Test Retries from the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization).

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/auto_test_retries_test_settings-2.34116b38e90672dcfa53d96c4a810aa7.png?auto=format"
   alt="Auto Test Retries enabled in Test Service Settings." /%}

By default, the feature retries each failing test case up to 5 times. Customize the Auto Test Retries with the following environment variables:

- `DD_CIVISIBILITY_FLAKY_RETRY_ENABLED` - set to `0` or `false` to explicitly disable retries even if the remote setting is enabled (default: true)
- `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - a non-negative number to change the maximum number of retries per test case (default: 5).
- `DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT` - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

{% /tab %}

{% tab title="Python" %}
### Compatibility{% #compatibility %}

`dd-trace-py >= 3.0.0` (`pytest >= 7.2.0`)

### Configuration{% #configuration %}

After you have set up Test Optimization, you can configure Auto Test Retries from the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization).

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/auto_test_retries_test_settings-2.34116b38e90672dcfa53d96c4a810aa7.png?auto=format"
   alt="Auto Test Retries in Test Service Settings." /%}

The default behavior of the feature is to retry any failing test case up to five times. Tests that originally fail either the original setup, teardown, or fixtures in Pytest, are not retried.

You can fine tune this behavior with the following environment variables:

- `DD_CIVISIBILITY_FLAKY_RETRY_ENABLED` - set to `0` or `false` to explicitly disable retries even if the remote setting is enabled (default: `true`)
- `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - a non-negative number to change the maximum number of retries per test case (default: `5`).
- `DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT` - a non-negative number to set the maximum total number of failed tests to retry (default: `1000`)

{% /tab %}

{% tab title="Swift" %}
### Compatibility{% #compatibility %}

`dd-sdk-swift-testing >= 2.5.2`

### Configuration{% #configuration %}

After you have set up Test Optimization, you can configure Auto Test Retries from the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization).

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/auto_test_retries_test_settings-2.34116b38e90672dcfa53d96c4a810aa7.png?auto=format"
   alt="Auto Test Retries in Test Service Settings." /%}

The default behavior of the feature is to retry any failing test case up to 5 times. This behavior can be fine-tuned with the following environment variables:

- `DD_CIVISIBILITY_FLAKY_RETRY_ENABLED` - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true)
- `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - a non-negative number to change the maximum number of retries per test case (default: 5).
- `DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT` - a non-negative number to set the maximum total number of failed tests to retry (default: 1000)

{% /tab %}

### Failed Test Replay{% #failed-test-replay %}

{% alert level="info" %}
Failed Test Replay is only supported for Java, JavaScript, and .NET.
{% /alert %}

In addition to automatically retrying failed tests, Failed Test Replay allows you to see local variable data in the topmost frame of the test error's stack trace. Enable this feature with the **Failed Test Replay** toggle.

#### Create a logs index{% #create-a-logs-index %}

Failed Test Replay creates logs that are sent to Datadog and appear alongside your regular application logs.

If you use [Exclusion filters](https://docs.datadoghq.com/logs/log_configuration/indexes/#exclusion-filters), ensure Failed Test Replay logs are not filtered:

1. Create a logs index and [configure it](https://docs.datadoghq.com/logs/log_configuration/indexes/#add-indexes) to the desired retention with **no sampling**.
1. Set the filter to match on the `source:dd_debugger` tag. All Failed Test Replay logs have this source.
1. Ensure that the new index takes precedence over any other index with filters that match that tag, because the first match wins.

After you enable this feature, you can see local variable data in failed tests:

{% image
   source="https://datadog-docs.imgix.net/images/continuous_integration/failed_test_replay_local_variables.36cc9dd2e0e102e2cbb8fec87db002b4.png?auto=format"
   alt="Failed Test Replay." /%}

#### Known limitations{% #known-limitations %}

[jest-image-snapshot](https://www.npmjs.com/package/jest-image-snapshot) is incompatible with `jest.retryTimes` unless `customSnapshotIdentifier` is passed (see [jest-image-snapshot docs](https://github.com/americanexpress/jest-image-snapshot?tab=readme-ov-file#jestretrytimes)) to `toMatchImageSnapshot`. Therefore, auto test retries do not work unless `customSnapshotIdentifier` is used.

## Explore results in the Test Optimization Explorer{% #explore-results-in-the-test-optimization-explorer %}

You can query the retried tests in the [Test Optimization Explorer](https://docs.datadoghq.com/tests/explorer/): they have the `@test.is_retry` tag set to `true` (some of them may also have the `@test.is_new` set to `true`, which indicates they have been retried by the [Early Flakiness Detection](https://docs.datadoghq.com/tests/flaky_test_management/early_flake_detection) feature).

## Troubleshooting{% #troubleshooting %}

If you suspect there are any issues with Auto Test Retries, navigate to the [Test Service Settings page](https://app.datadoghq.com/ci/settings/test-optimization), look for your test service, and click **Configure**. Disable Auto Test Retries by clicking on the toggle.

## Further Reading{% #further-reading %}

- [Learn about Test Optimization](https://docs.datadoghq.com/tests)
- [Learn about Flaky Test Management](https://docs.datadoghq.com/tests/flaky_test_management)