---
title: Validate Optimizations
description: >-
  Validate that Test Optimization features—including Early Flake Detection, Auto
  Test Retries, and Flaky Test Management—are working correctly in your
  repository.
breadcrumbs: >-
  Docs > Test Optimization in Datadog > Test Optimization Guides > Validate
  Optimizations
---

# Validate Optimizations

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ().
{% /alert %}

{% /callout %}

This page explains how to check that the optimizations offered by Test Optimization are working as intended. The guide assumes that [Test Optimization](https://docs.datadoghq.com/tests.md) already works for the repository under validation, and it shows the steps to validate optimizations for a **single repository**.

{% alert level="warning" %}
Run these validations in a feature branch only, and do not merge them into your default or main branch.
{% /alert %}

## Prerequisites{% #prerequisites %}

These optimizations require a [supported native library](https://docs.datadoghq.com/tests.md). JUnit XML uploads are not supported.

## Prevention{% #prevention %}

Prevention is achieved through [Early Flake Detection](https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection.md) and [New Flaky Test PR Gates](https://docs.datadoghq.com/tests/guides/setup_new_flaky_pr_gate.md), which help detect new flaky tests and block them from reaching your default branch.

To validate prevention is working, follow these steps:

1. Enable Early Flake Detection in the [settings page](https://app.datadoghq.com/ci/settings/test-optimization/advanced-features).
1. Create a [New Flaky Test PR gate](https://app.datadoghq.com/ci/pr-gates/rule/create?dataSource=test_optimization) and define its scope to the repository you are validating.

{% image
   source="https://docs.dd-static.net/images/pr_gates/setup/pr_gate_scope.2f4414c836045b0d7d292a1187579ea5.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/pr_gate_scope.2f4414c836045b0d7d292a1187579ea5.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="New flaky PR gate scope" /%}
Add a new flaky test.
This test is flaky by design. **It is not intended to be committed to the default branch**. Run the validation in a feature branch, and do not merge it.

Here are some simple examples of flaky tests. The test name must include the string `flaky`:

{% tab title="JavaScript" %}

```javascript
test('flaky test', () => {
    expect(Math.random()).toBeGreaterThan(0.5);
});
```

{% /tab %}

{% tab title="Python" %}

```python
import random

def test_flaky():
    assert random.random() > 0.5
```

{% /tab %}

{% tab title="Java" %}

```java
@Test
public void flakyTest() {
    assertTrue(Math.random() > 0.5);
}
```

{% /tab %}

{% tab title="Ruby" %}

```ruby
it 'is flaky' do
  expect(rand).to be > 0.5
end
```

{% /tab %}

{% tab title=".NET" %}

```csharp
[Fact]
public void FlakyTest()
{
    Assert.True(new Random().NextDouble() > 0.5);
}
```

{% /tab %}

{% tab title="Go" %}

```go
func TestFlaky(t *testing.T) {
    if rand.Float64() <= 0.5 {
        t.Fail()
    }
}
```

{% /tab %}

{% tab title="Swift" %}

```swift
func testFlaky() {
    XCTAssertTrue(Double.random(in: 0..<1) > 0.5)
}
```

{% /tab %}
Create a new branch `validate-test-optimization-prevention`, commit the changes to add a new flaky test, and push the changes to open a pull request.
```bash
git checkout -b validate-test-optimization-prevention
git add -A
git commit -m "Validate Test Optimization's Prevention"
git push origin validate-test-optimization-prevention
```
Wait for CI to run.In the GitHub checks of your pull request, the New Flaky Test PR Gate should be failing:
{% image
   source="https://docs.dd-static.net/images/pr_gates/setup/failed_pr_gate.340f689e06e353559d009c926477ca24.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/failed_pr_gate.340f689e06e353559d009c926477ca24.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="GitHub Pull Request check failing because a new flaky test is detected" /%}
Click on the failing GitHub check:
{% image
   source="https://docs.dd-static.net/images/pr_gates/setup/pr_gate_detail.370daebed189f6dbd1e2634aaf31fd09.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/pr_gate_detail.370daebed189f6dbd1e2634aaf31fd09.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Datadog PR gate detail view" /%}

The test you added is included in the list of new flaky tests. Click it to be redirected to [Flaky Test Management](https://app.datadoghq.com/ci/test/flaky?query=%40test.name%3A%2Aflaky%2A%20first_flaked_branch%3Avalidate-test-optimization-prevention).
Additionally check that the test is detected as new flaky in [Test Runs](https://app.datadoghq.com/ci/test/runs?query=test_level%3Atest%20%40test.name%3A%2Aflaky%2A%20%40git.branch%3Avalidate-test-optimization-prevention%20%40test.test_management.is_new_flaky%3Atrue). Check that the filter parameters include `@test.name:*flaky*`, `@git.branch:validate-test-optimization-prevention` and `@test.test_management.is_new_flaky:true`.
## Mitigation{% #mitigation %}

Mitigation is achieved through [Auto Test Retries](https://docs.datadoghq.com/tests/flaky_tests/auto_test_retries.md), [Flaky Test Management](https://docs.datadoghq.com/tests/flaky_management.md), and [Flaky Test Policies](https://docs.datadoghq.com/tests/flaky_management.md#configure-policies-to-automate-the-flaky-test-lifecycle). These optimizations allow you to automatically retry flaky tests and apply policies on them, such as quarantining or disabling.

To validate mitigation is working, follow these steps:

1. Enable Auto Test Retries in the [settings page](https://app.datadoghq.com/ci/settings/test-optimization/advanced-features).
1. Enable Flaky Test Policies in the [Flaky Tests Policies settings page](https://app.datadoghq.com/ci/settings/test-optimization/flaky-test-management).
1. Create a flaky test policy that disables the flaky test if it flakes on `validate-test-optimization-mitigation`.

{% image
   source="https://docs.dd-static.net/images/pr_gates/setup/flaky_test_policy_disable.d13e1f6c806a6cb1f7889815002b3e65.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/flaky_test_policy_disable.d13e1f6c806a6cb1f7889815002b3e65.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Flaky test policy for disabling a test" /%}
Add a new flaky test.
This test is flaky by design. **It is not intended to be committed to the default branch**. Run the validation in a feature branch, and do not merge it.

Here are some simple examples of flaky tests. The test name must include the string `flaky`:

{% tab title="JavaScript" %}

```javascript
test('flaky test', () => {
    expect(Math.random()).toBeGreaterThan(0.5);
});
```

{% /tab %}

{% tab title="Python" %}

```python
import random

def test_flaky():
    assert random.random() > 0.5
```

{% /tab %}

{% tab title="Java" %}

```java
@Test
public void flakyTest() {
    assertTrue(Math.random() > 0.5);
}
```

{% /tab %}

{% tab title="Ruby" %}

```ruby
it 'is flaky' do
  expect(rand).to be > 0.5
end
```

{% /tab %}

{% tab title=".NET" %}

```csharp
[Fact]
public void FlakyTest()
{
    Assert.True(new Random().NextDouble() > 0.5);
}
```

{% /tab %}

{% tab title="Go" %}

```go
func TestFlaky(t *testing.T) {
    if rand.Float64() <= 0.5 {
        t.Fail()
    }
}
```

{% /tab %}

{% tab title="Swift" %}

```swift
func testFlaky() {
    XCTAssertTrue(Double.random(in: 0..<1) > 0.5)
}
```

{% /tab %}
Create a new branch `validate-test-optimization-mitigation`, commit the changes to add a new flaky test, and push the changes to open a pull request.
```bash
git checkout -b validate-test-optimization-mitigation
git add -A
git commit -m "Validate Test Optimization's mitigation"
git push origin validate-test-optimization-mitigation
```
Wait for CI to run.Confirm that the newly added flaky test does not cause CI to fail.Go to [Flaky Test Management](https://app.datadoghq.com/ci/test/flaky?query=%40test.name%3A%2Aflaky%2A%20first_flaked_branch%3Avalidate-test-optimization-mitigation) and check that the newly added flaky test shows up.
**Important**: Check that the filter parameters include `@test.name:*flaky*`, `first_flaked_branch:validate-test-optimization-mitigation`.

Click on the only test in the list and verify that it shows as `DISABLED`. This confirms that the flaky test policy was triggered.
Go to [Test Runs](https://app.datadoghq.com/ci/test/runs?query=test_level%3Atest%20%40test.name%3A%2Aflaky%2A%20%40git.branch%3Avalidate-test-optimization-mitigation%20%40test.is_flaky%3Atrue) and check that the newly added flaky test shows up.
**Important**: Check that the filter parameters include `@test.name:*flaky*`, `@git.branch:validate-test-optimization-mitigation` and `@test.is_flaky:true`.

## Remediation{% #remediation %}

Test Optimization helps with the remediation of test flakiness with attempt to fix and Bits AI auto fixes. This section focuses on the validation of the attempt to fix workflow.

To validate attempt to fix, follow these steps:

1. Enable Auto Test Retries in the [settings page](https://app.datadoghq.com/ci/settings/test-optimization/advanced-features).
1. Add a new flaky test.

This test is flaky by design. **It is not intended to be committed to the default branch**. Run the validation in a feature branch, and do not merge it.

Here are some simple examples of flaky tests. The test name must include the string `flaky`:

{% tab title="JavaScript" %}

```javascript
test('flaky test', () => {
    expect(Math.random()).toBeGreaterThan(0.5);
});
```

{% /tab %}

{% tab title="Python" %}

```python
import random

def test_flaky():
    assert random.random() > 0.5
```

{% /tab %}

{% tab title="Java" %}

```java
@Test
public void flakyTest() {
    assertTrue(Math.random() > 0.5);
}
```

{% /tab %}

{% tab title="Ruby" %}

```ruby
it 'is flaky' do
  expect(rand).to be > 0.5
end
```

{% /tab %}

{% tab title=".NET" %}

```csharp
[Fact]
public void FlakyTest()
{
    Assert.True(new Random().NextDouble() > 0.5);
}
```

{% /tab %}

{% tab title="Go" %}

```go
func TestFlaky(t *testing.T) {
    if rand.Float64() <= 0.5 {
        t.Fail()
    }
}
```

{% /tab %}

{% tab title="Swift" %}

```swift
func testFlaky() {
    XCTAssertTrue(Double.random(in: 0..<1) > 0.5)
}
```

{% /tab %}
Create a new branch `validate-test-optimization-attempt-to-fix`, commit the changes to add a new flaky test, and push the changes to open a pull request.
```bash
git checkout -b validate-test-optimization-attempt-to-fix
git add -A
git commit -m "Validate Test Optimization's attempt to fix"
git push origin validate-test-optimization-attempt-to-fix
```
Wait for CI to run.Confirm that the newly added flaky test does not cause CI to fail.Go to [Flaky Test Management](https://app.datadoghq.com/ci/test/flaky?query=%40test.name%3A%2Aflaky%2A%20first_flaked_branch%3Avalidate-test-optimization-attempt-to-fix) and check that the newly added flaky test shows up as **Active**.
**Important**: Check that the filter parameters include `@test.name:*flaky*`, `first_flaked_branch:validate-test-optimization-attempt-to-fix`.
Fix the flaky test by removing its randomness:
{% tab title="JavaScript" %}

```javascript
test('flaky test', () => {
    expect(true).toBe(true);
});
```

{% /tab %}

{% tab title="Python" %}

```python
def test_flaky():
    assert True
```

{% /tab %}

{% tab title="Java" %}

```java
@Test
public void flakyTest() {
    assertTrue(true);
}
```

{% /tab %}

{% tab title="Ruby" %}

```ruby
it 'is flaky' do
  expect(true).to be true
end
```

{% /tab %}

{% tab title=".NET" %}

```csharp
[Fact]
public void FlakyTest()
{
    Assert.True(true);
}
```

{% /tab %}

{% tab title="Go" %}

```go
func TestFlaky(t *testing.T) {
}
```

{% /tab %}

{% tab title="Swift" %}

```swift
func testFlaky() {
    XCTAssertTrue(true)
}
```

{% /tab %}
In [Flaky Test Management](https://app.datadoghq.com/ci/test/flaky?query=%40test.name%3A%2Aflaky%2A%20first_flaked_branch%3Avalidate-test-optimization-attempt-to-fix), click the flaky test, then click on the `Actions` button and select `Link commit to Flaky Test fix`. This opens a modal that provides a test key and sample Git command:
{% image
   source="https://docs.dd-static.net/images/pr_gates/setup/attempt_to_fix_modal.5c93a1ec64a7e758e8b105f41b5839d5.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/attempt_to_fix_modal.5c93a1ec64a7e758e8b105f41b5839d5.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Attempt to fix modal" /%}

Copy the git commit command.
Commit and push the flaky test fixes:
```bash
git add -A
git commit -m "Fix flaky test DD_ABC123"
git push origin validate-test-optimization-attempt-to-fix
```
Wait for CI to finish.After CI has finished, go back to [Flaky Test Management](https://app.datadoghq.com/ci/test/flaky?query=%40test.name%3A%2Aflaky%2A%20first_flaked_branch%3Avalidate-test-optimization-attempt-to-fix). The test now shows up as `Fix In Progress`. This means that the attempt to fix has worked. The test automatically moves to `Fixed` when the PR is merged.
**Important**: Do not merge the PR, as it was just purely for validation purposes. Close the pull request without merging.

## Further reading{% #further-reading %}

- [Set up a New Flaky Test PR Gate](https://docs.datadoghq.com/tests/guides/setup_new_flaky_pr_gate.md)
- [Learn about Early Flake Detection](https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection.md)
- [Learn about Auto Test Retries](https://docs.datadoghq.com/tests/flaky_tests/auto_test_retries.md)
- [Learn about Flaky Test Management](https://docs.datadoghq.com/tests/flaky_management.md)