--- title: Set up a New Flaky Test PR Gate description: Learn how to set up a New Flaky Test PR Gate. breadcrumbs: >- Docs > Test Optimization in Datadog > Test Optimization Guides > Set up a New Flaky Test PR Gate --- # Set up a New Flaky Test PR Gate {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). (). {% /alert %} {% /callout %} ## Overview{% #overview %} A [New Flaky Test PR Gate](https://app.datadoghq.com/ci/pr-gates/rule/create?dataSource=test_optimization) blocks pull requests that introduce at least one new flaky test. This helps keep flaky tests out of your default branch. A new flaky test is a test that: - was added in the pull request branch and has not run in any other branch. - has exhibited [flaky behavior](https://docs.datadoghq.com/tests/flaky_tests) for the first time. {% alert level="info" %} [Early Flake Detection](https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection) and automatic fix verification with [Attempt To Fix](https://docs.datadoghq.com/tests/flaky_management/#confirm-fixes-for-flaky-tests) are supported only with native libraries, not [JUnit XML uploads](https://docs.datadoghq.com/tests/setup/junit_xml/). If you use JUnit XML uploads, verify fixes manually and mark the test as **Fixed** in [Flaky Tests Management](https://docs.datadoghq.com/tests/flaky_management). {% /alert %} Before you begin: - Set up [Test Optimization](https://docs.datadoghq.com/tests/) for your language. - To improve detection of new flaky tests, enable [Early Flake Detection](https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection). See [supported features](https://docs.datadoghq.com/tests/#supported-features). ## How it works{% #how-it-works %} If Datadog detects at least one new flaky test in the feature branch, the PR gate fails. ## Set up the PR gate{% #set-up-the-pr-gate %} 1. Go to [Create Rule](https://app.datadoghq.com/ci/pr-gates/rule/create?dataSource=test_optimization) and select **New Flaky Tests**. {% image source="https://docs.dd-static.net/images/pr_gates/setup/flaky_tests.b1de875eaf501c25fcf4c81cb4633e4a.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/flaky_tests.b1de875eaf501c25fcf4c81cb4633e4a.png?auto=format&fit=max&w=850&dpr=2 2x" alt="A PR Gate rule that fails when a pull request introduces at least one new flaky test" /%} Select the repositories where the rule should be evaluated. By default, it applies to all repositories in your organization. **Optional**: enable [Early Flake Detection](https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection) to improve detection of new flaky tests. After you create the PR gate, a new status check appears in pull requests from the selected repositories. {% image source="https://docs.dd-static.net/images/pr_gates/setup/pr_gate_preview.1c27d4b61c6403f6e8be83c7c42e0f3a.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/pr_gate_preview.1c27d4b61c6403f6e8be83c7c42e0f3a.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Preview from new flaky test PR gate" /%} ## In a pull request{% #in-a-pull-request %} The following screenshots use GitHub as an example. The status check and optional PR comment appear in the pull request. If Datadog detects a new flaky test, the status check fails: {% image source="https://docs.dd-static.net/images/pr_gates/setup/failed_pr_gate.340f689e06e353559d009c926477ca24.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/failed_pr_gate.340f689e06e353559d009c926477ca24.png?auto=format&fit=max&w=850&dpr=2 2x" alt="GitHub pull request check failing because a new flaky test is detected" /%} If [PR comments](https://docs.datadoghq.com/source_code/features/#pr-comments) are enabled, Datadog adds a comment listing the new flaky tests and their error messages: {% image source="https://docs.dd-static.net/images/pr_gates/setup/new_flaky_test_pr_comment.054b84d318e66487fd2e396e98719730.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/new_flaky_test_pr_comment.054b84d318e66487fd2e396e98719730.png?auto=format&fit=max&w=850&dpr=2 2x" alt="GitHub pull request comment showing the new flaky test" /%} When you open the failed check, Datadog shows the PR gate details in the Datadog UI: {% image source="https://docs.dd-static.net/images/pr_gates/setup/pr_gate_detail.370daebed189f6dbd1e2634aaf31fd09.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/pr_gate_detail.370daebed189f6dbd1e2634aaf31fd09.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Datadog PR gate detail view" /%} ## Make the PR gate pass{% #make-the-pr-gate-pass %} The PR gate stays red until the flaky test is marked as **Fixed** in [Flaky Tests Management](https://docs.datadoghq.com/tests/flaky_management). There are two ways to mark the new flaky test as Fixed: 1. **(Recommended)** Automatically triggered [Attempt To Fix](https://docs.datadoghq.com/tests/flaky_management/#confirm-fixes-for-flaky-tests) 1. Manually mark the test as [Fixed](https://docs.datadoghq.com/tests/flaky_management/#change-a-flaky-tests-state) ### Automatically triggered Attempt To Fix{% #automatically-triggered-attempt-to-fix %} If the new flaky test is reported by one of the [supported native libraries](https://docs.datadoghq.com/tests/#setup), the next test execution after detection automatically triggers [Attempt To Fix](https://docs.datadoghq.com/tests/flaky_management/#confirm-fixes-for-flaky-tests). The native library retries the test to verify whether the flakiness has been resolved. The expected flow is: 1. A developer pushes a new test. 1. CI runs and the test is detected as newly flaky. 1. The New Flaky PR gate fails. 1. A developer pushes a fix. 1. The native library retries the newly flaky test to verify the fix. - If the fix worked, the PR gate passes. - If the fix did not work, the PR gate fails. ### Manually mark the test as Fixed{% #manually-mark-the-test-as-fixed %} **Not**: This is not necessary if the test is reported with a [supported native library](https://docs.datadoghq.com/tests/#setup). If the new flaky test is reported with [JUnit XML upload](https://docs.datadoghq.com/tests/setup/junit_xml/), you must verify the fix yourself and then mark the test as fixed manually. The expected flow is: 1. A developer pushes a new test. 1. CI runs and the test is detected as newly flaky. 1. The New Flaky PR gate fails. 1. A developer pushes a fix. 1. Verify the fix manually, for example by retrying CI or using programmatic retries. - If the fix worked, mark the test as Fixed manually. The PR gate passes. - If the fix did not work, keep iterating. The PR gate fails. #### How to manually mark the new flaky test as Fixed{% #how-to-manually-mark-the-new-flaky-test-as-fixed %} In the Datadog UI, follow the link from the PR gate details: {% image source="https://docs.dd-static.net/images/pr_gates/setup/pr_gate_detail_test.3d22a1b36666bab6db527dd75a38def7.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/pr_gate_detail_test.3d22a1b36666bab6db527dd75a38def7.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Datadog PR gate detail view with a link to Flaky Tests Management" /%} This link opens [Flaky Tests Management](https://docs.datadoghq.com/tests/flaky_management) in the Datadog UI with the test selected: {% image source="https://docs.dd-static.net/images/pr_gates/setup/flaky_test_detail.0abd5bc60dd3715cd111f213a9d3a0ea.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/flaky_test_detail.0abd5bc60dd3715cd111f213a9d3a0ea.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Datadog Flaky Tests Management view with the selected test" /%} Change the status from **Active** to **Fixed**: {% image source="https://docs.dd-static.net/images/pr_gates/setup/fix_flaky_test.5edff86f189504ac13299fa21fe9fe2c.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/fix_flaky_test.5edff86f189504ac13299fa21fe9fe2c.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Datadog Flaky Tests Management status menu showing Fixed" /%} ## Require the PR gate in GitHub{% #require-the-pr-gate-in-github %} If you use GitHub, you can mark the PR gate as a required status check in branch protection. By default, the PR gate is optional, so pull requests can still be merged when a new flaky test is detected: {% image source="https://docs.dd-static.net/images/pr_gates/setup/github_required_check.110eb6cff31e7c135dd6eb59155b0333.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/pr_gates/setup/github_required_check.110eb6cff31e7c135dd6eb59155b0333.png?auto=format&fit=max&w=850&dpr=2 2x" alt="GitHub branch protection rule requiring the PR gate status check" /%} For more information, see the GitHub documentation for [status checks](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks). ## Further Reading{% #further-reading %} - [Learn about Test Optimization](https://docs.datadoghq.com/tests) - [Learn about Flaky tests](https://docs.datadoghq.com/tests/flaky_tests) - [Learn about Flaky Test Management](https://docs.datadoghq.com/tests/flaky_management)