Flaky Test Management

This product is not supported for your selected Datadog site. ().
이 페이지는 아직 영어로 제공되지 않습니다. 번역 작업 중입니다.
현재 번역 프로젝트에 대한 질문이나 피드백이 있으신 경우 언제든지 연락주시기 바랍니다.

Test Optimization is not available in the selected site () at this time.

Join the Preview!

Flaky Test Management is in Preview. Complete the form to request access.

Request Access

Overview

The Flaky Test Management page provides a centralized view to track, triage, and remediate flaky tests across your organization. You can view every test’s status along with key impact metrics like number of pipeline failures, CI time wasted, and failure rate.

From this UI, you can act on flaky tests to mitigate their impact. Quarantine or disable problematic tests to keep known flakes from breaking builds, and create cases and Jira issues to track work toward fixes.

Overview of the Flaky Tests Management UI

Change a flaky test’s status

Use the status drop-down to change how a flaky test is handled in your CI pipeline. This can help reduce CI noise while retaining traceability and control. Available statuses are:

StatusDescription
ActiveThe test is known to be flaky and is running in CI.
QuarantinedKeep the test running in the background, but failures don’t affect CI status or break pipelines. This is useful for isolating flaky tests without blocking merges.
DisabledSkip the test entirely in CI. Use this when a test is no longer relevant or needs to be temporarily removed from the pipeline.
FixedThe test has passed consistently and is no longer flaky. If supported, use the remediation flow to confirm a fix and automatically apply this status, instead of manually changing it.
Note: Status actions have minimum version requirements for each programming language. See Compatibility for details.

Investigate a flaky test

For more information about a specific flaky test, use these options in the actions menu at the end of each row:

  • View Last Failed Test Run: Open the side panel with the details of the test’s most recent failed run.
  • View related test executions: Open the Test Optimization Explorer populated with all of the test’s recent runs.

Create cases for flaky tests

For any flaky test, you can create a case and use Case Management to track any work toward remediation. Click the Create Case button or use the actions menu at the end of the row.

Confirm fixes for flaky tests

When you fix a flaky test, Test Optimization’s remediation flow can confirm the fix by retrying the test multiple times. If successful, the test’s status is automatically updated to Fixed. To enable the remediation flow:

  1. For the test you are fixing, click Fix this test in the Flaky Test Management UI.
  2. Copy the unique flaky test key that is displayed (for example, DD_ABC123).
  3. Include the test key in your Git commit title or message for the fix (for example, git commit -m "DD_ABC123").
  4. When Datadog detects the test key in your commit, it automatically triggers the remediation flow for that test:
    • Retries any tests you’re attempting to fix 20 times.
    • Runs tests even if they are marked as Disabled.
    • If all retries pass, updates the test’s status to Fixed.
    • If any retry fails, keeps the test’s current status (Active, Quarantined, or Disabled).

Compatibility

Flaky Test Management features have minimum version requirements for each programming language. The table below outlines the minimum versions needed to quarantine, disable, and attempt to fix flaky tests:

LanguageQuarantine & DisableAttempt to fix
.NET3.13.0+3.17.0+
Go1.73.0+Not available
Java1.48.0+1.50.0+
JavaScript5.44.0+5.52.0+
Python3.3.0+3.8.0+
Ruby1.13.0+1.17.0+

Further reading

추가 유용한 문서, 링크 및 기사:

Learn about Test OptimizationDOCUMENTATION more more Learn about working with flaky testsDOCUMENTATION more more Flaky Tests OverviewKNOWLEDGE CENTER more more