Validate Optimizations

Docs > Test Optimization in Datadog > Test Optimization Guides > Validate Optimizations

This page explains how to check that the optimizations offered by Test Optimization are working as intended. The guide assumes that Test Optimization already works for the repository under validation, and it shows the steps to validate optimizations for a single repository.

Run these validations in a feature branch only, and do not merge them into your default or main branch.

Prerequisites

These optimizations require a supported native library. JUnit XML uploads are not supported.

Option 1: Validate locally with a coding agent

Using the provided prompt below, ask a local coding agent (an AI assistant that can inspect and run commands in your local repository) to inspect your installed dd-trace package and run its Test Optimization validation runbook. This method checks local library compatibility and CI configuration. It also checks Early Flake Detection, Auto Test Retries, and Test Management without changing Datadog settings or sending validation results to Datadog.

The runbook is at ci/runbook.md relative to the installed dd-trace package root.

Pass this prompt to your local coding agent:

Locate the installed dd-trace package, then read and execute its ci/runbook.md.

This coding-agent method is a local check that does not exercise the entire Datadog workflow. To validate the full Prevention, Mitigation, and Remediation workflows, or to validate a language other than JavaScript or TypeScript, use Option 2, below.

Option 2: Validate the full workflow

This validation workflow checks the complete Test Optimization workflow in Datadog. Perform these validations (Prevention, Mitigation, and Remediation) in order, as they use the same branch and test.

Step 1: Set up validation

This guide walks you through making local changes and committing them for CI to run. It uses a dedicated test service and branch to minimize the validation workflow’s impact on other developers in the repository.

Configure your CI test job to set DD_SERVICE before it runs the test command:
```
export DD_SERVICE=validate-test-optimization
```

Create the validation branch:

git checkout -b validate-test-optimization

Commit the CI configuration change that sets DD_SERVICE, then push the validation branch to trigger a test execution:
```
git add -A
git commit -m "Configure Test Optimization validation service"
git push -u origin validate-test-optimization
```
Datadog detects the validate-test-optimization service when the tests report under that name.
After CI finishes, go to the CI/CD Repositories settings and select the repository you are validating.
In the upper-right corner of the slide-out panel, click Test Service.
In Test service overrides, select the validate-test-optimization service.
Configure the following service overrides:
- Enable Early Flake Detection.
- Enable Auto Test Retries.
- Disable Test Impact Analysis (so it does not skip the validation test).
Return to the repository settings. Flaky Test Policies apply to every test service in the repository, not to an individual test service. To limit the validation policy’s impact, configure it only for the validate-test-optimization branch. Under Flaky Test Policies, on the Quarantine tile, click Configure.
Enable the second auto-rule: If an Active flaky test flakes in the validate-test-optimization branch, then move to Quarantined.
Click Save.
Create a New Flaky Test PR Gate and scope it to the repository you are validating.

Step 2: Prevention

Early Flake Detection detects new flaky tests. New Flaky Test PR Gates block them from reaching your default branch.

Add a test that fails on the first attempt and passes on retries (optionally using the code provided below). The test name must contain both flaky and validation so you can identify it in Datadog.

const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');

test('flaky validation test', () => {
    const marker = path.join(os.tmpdir(), 'dd-validation-flaky');
    if (!fs.existsSync(marker)) {
        fs.writeFileSync(marker, '1');
        throw new Error('first attempt fails so Datadog can retry it');
    }
});

from pathlib import Path
from tempfile import gettempdir


def test_flaky_validation_test():
    marker = Path(gettempdir()) / "dd-validation-flaky"
    if not marker.exists():
        marker.write_text("1")
        raise AssertionError("first attempt fails so Datadog can retry it")

import static org.junit.jupiter.api.Assertions.fail;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import org.junit.jupiter.api.Test;

class ValidationFlakyTest {
    @Test
    void flakyValidationTest() throws IOException {
        Path marker = Paths.get(
            System.getProperty("java.io.tmpdir"),
            "dd-validation-flaky"
        );
        if (Files.notExists(marker)) {
            Files.write(marker, new byte[] { '1' });
            fail("first attempt fails so Datadog can retry it");
        }
    }
}

require 'tmpdir'

RSpec.describe 'validation flaky tests' do
  it 'flaky validation test' do
    marker = File.join(Dir.tmpdir, 'dd-validation-flaky')
    unless File.exist?(marker)
      File.write(marker, '1')
      raise 'first attempt fails so Datadog can retry it'
    end
  end
end

using System.IO;
using Xunit;

public class ValidationFlakyTests
{
    [Fact]
    public void FlakyValidationTest()
    {
        var marker = Path.Combine(Path.GetTempPath(), "dd-validation-flaky");
        if (!File.Exists(marker))
        {
            File.WriteAllText(marker, "1");
            throw new System.Exception("first attempt fails so Datadog can retry it");
        }
    }
}

package validation

import (
    "errors"
    "os"
    "path/filepath"
    "testing"
)

func TestFlakyValidationTest(t *testing.T) {
    marker := filepath.Join(os.TempDir(), "dd-validation-flaky")
    if _, err := os.Stat(marker); errors.Is(err, os.ErrNotExist) {
        if writeErr := os.WriteFile(marker, []byte("1"), 0600); writeErr != nil {
            t.Fatal(writeErr)
        }
        t.Fatal("first attempt fails so Datadog can retry it")
    }
}

import XCTest

final class ValidationFlakyTests: XCTestCase {
    func testFlakyValidationTest() throws {
        let marker = FileManager.default.temporaryDirectory
            .appendingPathComponent("dd-validation-flaky")
        if !FileManager.default.fileExists(atPath: marker.path) {
            try "1".write(to: marker, atomically: true, encoding: .utf8)
            XCTFail("first attempt fails so Datadog can retry it")
        }
    }
}

Commit and push the test, then open a pull request from the validation branch:

git add -A
git commit -m "Validate Test Optimization prevention"
git push origin validate-test-optimization

Wait for CI to run. Early Flake Detection retries the new test, and the New Flaky Test PR Gate evaluates the result. In the GitHub checks for your pull request, confirm that the New Flaky Test PR Gate fails:
Click the failing GitHub check and confirm that the test is included in the list of new flaky tests:
In Test Runs, confirm that Early Flake Detection retried the test and detected it as a new flaky test using this query, which uses the following filters:
- @test.name:*flaky*validation*
- @git.branch:validate-test-optimization
- @test.retry_reason:early_flake_detection
- @test.test_management.is_new_flaky:true

Step 3: Mitigation

Mitigation is achieved through Auto Test Retries, Flaky Test Management, and Flaky Test Policies. These features retry flaky tests and quarantine known flaky failures so they do not block CI.

In the same test that you added for Prevention, change the marker filename from dd-validation-flaky to dd-validation-flaky-mitigation. Do not rename the test function or test case. The new marker causes another intentional first-attempt failure. Keeping the test name unchanged lets Datadog associate the run with the flaky test detected during Prevention. No additional Datadog configuration is required; Auto Test Retries and Flaky Test Management handle the test during this run. Update the test for your language:

const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');

test('flaky validation test', () => {
    // Changed from dd-validation-flaky.
    const marker = path.join(os.tmpdir(), 'dd-validation-flaky-mitigation');
    if (!fs.existsSync(marker)) {
        fs.writeFileSync(marker, '1');
        throw new Error('first attempt fails so Datadog can retry it');
    }
});

from pathlib import Path
from tempfile import gettempdir


def test_flaky_validation_test():
    # Changed from dd-validation-flaky.
    marker = Path(gettempdir()) / "dd-validation-flaky-mitigation"
    if not marker.exists():
        marker.write_text("1")
        raise AssertionError("first attempt fails so Datadog can retry it")

import static org.junit.jupiter.api.Assertions.fail;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import org.junit.jupiter.api.Test;

class ValidationFlakyTest {
    @Test
    void flakyValidationTest() throws IOException {
        // Changed from dd-validation-flaky.
        Path marker = Paths.get(
            System.getProperty("java.io.tmpdir"),
            "dd-validation-flaky-mitigation"
        );
        if (Files.notExists(marker)) {
            Files.write(marker, new byte[] { '1' });
            fail("first attempt fails so Datadog can retry it");
        }
    }
}

require 'tmpdir'

RSpec.describe 'validation flaky tests' do
  it 'flaky validation test' do
    # Changed from dd-validation-flaky.
    marker = File.join(Dir.tmpdir, 'dd-validation-flaky-mitigation')
    unless File.exist?(marker)
      File.write(marker, '1')
      raise 'first attempt fails so Datadog can retry it'
    end
  end
end

using System.IO;
using Xunit;

public class ValidationFlakyTests
{
    [Fact]
    public void FlakyValidationTest()
    {
        // Changed from dd-validation-flaky.
        var marker = Path.Combine(Path.GetTempPath(), "dd-validation-flaky-mitigation");
        if (!File.Exists(marker))
        {
            File.WriteAllText(marker, "1");
            throw new System.Exception("first attempt fails so Datadog can retry it");
        }
    }
}

package validation

import (
    "errors"
    "os"
    "path/filepath"
    "testing"
)

func TestFlakyValidationTest(t *testing.T) {
    // Changed from dd-validation-flaky.
    marker := filepath.Join(os.TempDir(), "dd-validation-flaky-mitigation")
    if _, err := os.Stat(marker); errors.Is(err, os.ErrNotExist) {
        if writeErr := os.WriteFile(marker, []byte("1"), 0600); writeErr != nil {
            t.Fatal(writeErr)
        }
        t.Fatal("first attempt fails so Datadog can retry it")
    }
}

import XCTest

final class ValidationFlakyTests: XCTestCase {
    func testFlakyValidationTest() throws {
        // Changed from dd-validation-flaky.
        let marker = FileManager.default.temporaryDirectory
            .appendingPathComponent("dd-validation-flaky-mitigation")
        if !FileManager.default.fileExists(atPath: marker.path) {
            try "1".write(to: marker, atomically: true, encoding: .utf8)
            XCTFail("first attempt fails so Datadog can retry it")
        }
    }
}

Commit and push the change on the same branch:

git add -A
git commit -m "Validate Test Optimization mitigation"
git push origin validate-test-optimization

Wait for CI to run, then confirm the following results:
- In Test Runs, Auto Test Retries reruns the test after its first failed attempt, and the test passes on retry. Use this query, with the following filters:
  - @test.name:*flaky*validation*
  - @git.branch:validate-test-optimization
  - @test.retry_reason:auto_test_retry
- In Flaky Test Management, the test appears as QUARANTINED. Its failures no longer block the test job. Use this query, with the following filters:
  - @test.name:*flaky*validation*
  - first_flaked_branch:validate-test-optimization
  - flaky_test_state:quarantined

Step 4: Remediation

Test Optimization helps remediate flaky tests through Attempt to Fix and Bits AI-powered flaky test fixes. This section validates the Attempt to Fix workflow by fixing the same test used for Prevention and Mitigation.

In Flaky Test Management, open the quarantined validation test.
Click Actions, select Link commit to fix, and copy the generated key (it starts with DD_).

Replace the flaky test with the passing version for your language:

test('flaky validation test', () => {
    expect(true).toBe(true);
});

def test_flaky_validation_test():
    assert True

@Test
void flakyValidationTest() {
    // intentionally empty - the test passes
}

it 'flaky validation test' do
  expect(true).to be(true)
end

[Fact]
public void FlakyValidationTest()
{
    Assert.True(true);
}

func TestFlakyValidationTest(t *testing.T) {
}

func testFlakyValidationTest() {
    XCTAssertTrue(true)
}

Commit the fix with the generated key in the commit body. Replace <YOUR_DD_KEY> with the key you copied:

git add -A
git commit -m "Fix flaky validation test" -m "<YOUR_DD_KEY>"
git push origin validate-test-optimization

Wait for CI to finish, then confirm the following results:
- In Test Runs, Attempt to Fix retried the fix candidate, and every attempt passed. Use this query, which has the following filters:
  - @test.name:*flaky*validation*
  - @git.branch:validate-test-optimization
  - @test.test_management.is_attempt_to_fix:true
- In Flaky Test Management, the test is marked Fix in progress. Use this query, which has the following filters:
  - @test.name:*flaky*validation*
  - first_flaked_branch:validate-test-optimization
  - fix_in_progress:true

Step 5: Post-validation cleanup

Close the pull request without merging.
Delete the validate-test-optimization branch. The branch-specific Quarantine auto-rule no longer applies after the branch is deleted, and the dedicated validation service no longer receives test executions.
Notify the team that owns the repository that the New Flaky Test PR Gate remains active for the entire repository. The gate is non-blocking by default.
Optionally, enable the Test Optimization features configured for the validate-test-optimization service at the repository level.

Validate Optimizations

Prerequisites

Option 1: Validate locally with a coding agent

Join the Preview!

Option 2: Validate the full workflow

Step 1: Set up validation

Step 2: Prevention

Step 3: Mitigation

Step 4: Remediation

Step 5: Post-validation cleanup

Further reading

How can I help you today?