This page explains how to check that the optimizations offered by Test Optimization are working as intended. The guide assumes that Test Optimization already works for the repository under validation, and it shows the steps to validate optimizations for a single repository.

Prerequisites

These optimizations require a supported native library. JUnit XML uploads are not supported.

Prevention

Prevention is achieved through Early Flake Detection and New Flaky Test PR Gates, which help detect new flaky tests and block them from reaching your default branch.

To validate prevention is working, follow these steps:

1. Enable Early Flake Detection in the settings page.
2. Create a New Flaky Test PR gate and define its scope to the repository you are validating.

3. Add a new flaky test.

This test is flaky by design. It is not intended to be committed to the default branch. Run the validation in a feature branch, and do not merge it.

Here are some simple examples of flaky tests. The test name must include the string flaky:

test('flaky test', () => {
    expect(Math.random()).toBeGreaterThan(0.5);
});

import random

def test_flaky():
    assert random.random() > 0.5

@Test
public void flakyTest() {
    assertTrue(Math.random() > 0.5);
}

it 'is flaky' do
  expect(rand).to be > 0.5
end

[Fact]
public void FlakyTest()
{
    Assert.True(new Random().NextDouble() > 0.5);
}

func TestFlaky(t *testing.T) {
    if rand.Float64() <= 0.5 {
        t.Fail()
    }
}

func testFlaky() {
    XCTAssertTrue(Double.random(in: 0..<1) > 0.5)
}

4. Create a new branch validate-test-optimization-prevention, commit the changes to add a new flaky test, and push the changes to open a pull request.

git checkout -b validate-test-optimization-prevention
git add -A
git commit -m "Validate Test Optimization's Prevention"
git push origin validate-test-optimization-prevention

5. Wait for CI to run.
6. In the GitHub checks of your pull request, the New Flaky Test PR Gate should be failing:

7. Click on the failing GitHub check:

The test you added is included in the list of new flaky tests. Click it to be redirected to Flaky Test Management.

8. Additionally check that the test is detected as new flaky in Test Runs. Check that the filter parameters include @test.name:*flaky*, @git.branch:validate-test-optimization-prevention and @test.test_management.is_new_flaky:true.

Mitigation

Mitigation is achieved through Auto Test Retries, Flaky Test Management, and Flaky Test Policies. These optimizations allow you to automatically retry flaky tests and apply policies on them, such as quarantining or disabling.

To validate mitigation is working, follow these steps:

1. Enable Auto Test Retries in the settings page.
2. Enable Flaky Test Policies in the Flaky Tests Policies settings page.
3. Create a flaky test policy that disables the flaky test if it flakes on validate-test-optimization-mitigation.

4. Add a new flaky test.

This test is flaky by design. It is not intended to be committed to the default branch. Run the validation in a feature branch, and do not merge it.

Here are some simple examples of flaky tests. The test name must include the string flaky:

test('flaky test', () => {
    expect(Math.random()).toBeGreaterThan(0.5);
});

import random

def test_flaky():
    assert random.random() > 0.5

@Test
public void flakyTest() {
    assertTrue(Math.random() > 0.5);
}

it 'is flaky' do
  expect(rand).to be > 0.5
end

[Fact]
public void FlakyTest()
{
    Assert.True(new Random().NextDouble() > 0.5);
}

func TestFlaky(t *testing.T) {
    if rand.Float64() <= 0.5 {
        t.Fail()
    }
}

func testFlaky() {
    XCTAssertTrue(Double.random(in: 0..<1) > 0.5)
}

5. Create a new branch validate-test-optimization-mitigation, commit the changes to add a new flaky test, and push the changes to open a pull request.

git checkout -b validate-test-optimization-mitigation
git add -A
git commit -m "Validate Test Optimization's mitigation"
git push origin validate-test-optimization-mitigation

6. Wait for CI to run.
7. Confirm that the newly added flaky test does not cause CI to fail.
8. Go to Flaky Test Management and check that the newly added flaky test shows up.

Important: Check that the filter parameters include @test.name:*flaky*, first_flaked_branch:validate-test-optimization-mitigation.

Click on the only test in the list and verify that it shows as DISABLED. This confirms that the flaky test policy was triggered.

9. Go to Test Runs and check that the newly added flaky test shows up.

Important: Check that the filter parameters include @test.name:*flaky*, @git.branch:validate-test-optimization-mitigation and @test.is_flaky:true.

Remediation

Test Optimization helps with the remediation of test flakiness with attempt to fix and Bits AI auto fixes. This section focuses on the validation of the attempt to fix workflow.

To validate attempt to fix, follow these steps:

1. Enable Auto Test Retries in the settings page.
2. Add a new flaky test.

This test is flaky by design. It is not intended to be committed to the default branch. Run the validation in a feature branch, and do not merge it.

Here are some simple examples of flaky tests. The test name must include the string flaky:

test('flaky test', () => {
    expect(Math.random()).toBeGreaterThan(0.5);
});

import random

def test_flaky():
    assert random.random() > 0.5

@Test
public void flakyTest() {
    assertTrue(Math.random() > 0.5);
}

it 'is flaky' do
  expect(rand).to be > 0.5
end

[Fact]
public void FlakyTest()
{
    Assert.True(new Random().NextDouble() > 0.5);
}

func TestFlaky(t *testing.T) {
    if rand.Float64() <= 0.5 {
        t.Fail()
    }
}

func testFlaky() {
    XCTAssertTrue(Double.random(in: 0..<1) > 0.5)
}

3. Create a new branch validate-test-optimization-attempt-to-fix, commit the changes to add a new flaky test, and push the changes to open a pull request.

git checkout -b validate-test-optimization-attempt-to-fix
git add -A
git commit -m "Validate Test Optimization's attempt to fix"
git push origin validate-test-optimization-attempt-to-fix

4. Wait for CI to run.
5. Confirm that the newly added flaky test does not cause CI to fail.
6. Go to Flaky Test Management and check that the newly added flaky test shows up as Active.

Important: Check that the filter parameters include @test.name:*flaky*, first_flaked_branch:validate-test-optimization-attempt-to-fix.

7. Fix the flaky test by removing its randomness:

test('flaky test', () => {
    expect(true).toBe(true);
});

def test_flaky():
    assert True

@Test
public void flakyTest() {
    assertTrue(true);
}

it 'is flaky' do
  expect(true).to be true
end

[Fact]
public void FlakyTest()
{
    Assert.True(true);
}

func TestFlaky(t *testing.T) {
}

func testFlaky() {
    XCTAssertTrue(true)
}

8. In Flaky Test Management, click the flaky test, then click on the Actions button and select Link commit to Flaky Test fix. This opens a modal that provides a test key and sample Git command:

Copy the git commit command.

9. Commit and push the flaky test fixes:

git add -A
git commit -m "Fix flaky test DD_ABC123"
git push origin validate-test-optimization-attempt-to-fix

10. Wait for CI to finish.
11. After CI has finished, go back to Flaky Test Management. The test now shows up as Fix In Progress. This means that the attempt to fix has worked. The test automatically moves to Fixed when the PR is merged.

Important: Do not merge the PR, as it was just purely for validation purposes. Close the pull request without merging.

Further reading

Additional helpful documentation, links, and articles:

Set up a New Flaky Test PR GateDOCUMENTATION Learn about Early Flake DetectionDOCUMENTATION Learn about Auto Test RetriesDOCUMENTATION Learn about Flaky Test ManagementDOCUMENTATION