- 필수 기능
- 시작하기
- Glossary
- 표준 속성
- Guides
- Agent
- 통합
- 개방형텔레메트리
- 개발자
- API
- Datadog Mobile App
- CoScreen
- Cloudcraft
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- APM
- Continuous Profiler
- 스팬 시각화
- 데이터 스트림 모니터링
- 데이터 작업 모니터링
- 디지털 경험
- 소프트웨어 제공
- 보안
- AI Observability
- 로그 관리
- 관리
A flaky test is a test that exhibit both a passing and failing status across multiple test runs for the same commit. If you commit some code and run it through CI, and a test fails, and you run it through CI again and the test passes, that test is unreliable as proof of quality code.
Flaky tests introduce risk and unpredictability into your CI system and end product. When people have to remember which tests are flaky, they lose trust in their test results, and a tremendous amount of time and resources are wasted on pipeline retries.
For each branch, the list shows the number of new flaky tests introduced by the commit, the number of flaky commits, total test time, and the branch’s latest commit details.
Use the following information to help prioritize flaky tests:
Once you identify a flaky test you want to fix, click on the test to see links to view the most recent failed test run or the first flaky test run.
If a flaky test has not failed in the past 30 days, it is automatically removed from the table. You can also manually remove a flaky test by clicking on the trash icon that appears when you hover over the test row. It is added again if it re-exhibits flaky behavior.
The flaky test table for the default branch includes tests that have flaked in the default branch as well as any tests that have exhibited flakiness in a feature branch that was merged into the default branch.
Flaky tests from merged feature branches are found by checking which tests have exhibited flakiness in the most recent 5,000 commits using the Git commit history. The Git commit history is collected by the Test Optimization libraries and uploaded along with the test results every time the testing phase of a particular commit is executed in your CI build.
Limitations:
For the default branches, a metric tracks the flaky tests over time. This metric is generated every 30 minutes and counts all the flaky tests in the default branch at that time.
Find this metric under the graph Total Flaky Tests in the default branch view:
The metric is also available in the CI Visibility - Tests dashboard.
New flaky tests are tests that exhibit flaky behavior and didn’t previously exist in the Flaky Tests table for the current branch or default branch of the repository.
true
.
All test runs that exhibited flaky behavior for the first time as per the definition above are displayed.You can ignore new flaky tests for a particular commit if you determine that those flaky tests were detected by mistake. The tests reappear if the commit exhibits flakiness again.
Click on the New Flaky number and then click Ignore flaky tests.
Known flaky failed tests are tests that have flaky behavior on the current or default branch of the repository.
true
.
Failed test runs that were known to be flaky as per the definition above are displayed.추가 유용한 문서, 링크 및 기사: