Intelligent Test Runner allows you to streamline your organization’s test impact analysis by intelligently excluding irrelevant tests that are not affected by recent code changes.
With Test Visibility, development teams can configure Intelligent Test Runner for their test services, set branches to exclude (such as the default branch), and define files to be tracked (which triggers full runs of all tests when any tracked file changes).
A Venn diagram displaying how Intelligent Test Runner defines an excluded test by using tracked files, excluded branches, and passed tests.
Configure and enable Intelligent Test Runner for your test services to reduce unnecessary testing time, enhance CI test efficiency, and reduce costs, while maintaining the reliability and performance across your CI environments.
Disabling the Intelligent Test Runner on critical branches (such as your default branch) ensures comprehensive test coverage, whereas enabling it to run on feature or development branches helps maximize testing efficiency.
Configure the Intelligent Test Runner
You can configure the Intelligent Test Runner to prevent specific tests from being skipped. These tests are known as unskippable tests, and are run regardless of code coverage data.
To configure the Intelligent Test Runner:
Click the Status toggle to enable Intelligent Test Runner.
Specify any branches to exclude (typically the default branch of a repository). Intelligent Test Runner does not skip tests for these branches.
Specify file directories and files to track (for example, documentation/content/** or domains/shopist/apps/api/BUILD.bazel). Intelligent Test Runner runs all CI tests when any of these tracked files change.
Click Save Settings.
Once you’ve configured Intelligent Test Runner on a test service, execute a test suite run on your default branch. This establishes a baseline for the Intelligent Test Runner to accurately skip irrelevant tests in future commits.
Use Intelligent Test Runner data
Explore the data collected by enabling Intelligent Test Runner, such as the time savings achieved by skipping tests, as well as your organization’s usage of Intelligent Test Runner, to improve your CI efficiency.
You can create dashboards to visualize your testing metrics, or use an out-of-the-box dashboard containing widgets populated with data collected by the Intelligent Test Runner to help you identify areas of improvement with usage patterns and trends.
Examine results in the Test Visibility Explorer
The Test Visibility Explorer allows you to create visualizations and filter test spans using the data collected from Test Visibility and the Intelligent Test Runner. When the Intelligent Test Runner is active, it displays the amount of time saved for each test session or commit. The duration bars turn purple to indicate active test skipping.
Use the following out-of-the-box Intelligent Test Runner facets to customize the search query:
Code Coverage Enabled
Indicates whether code coverage tracking was active during the test session.
Skipped by ITR
Number of tests that were skipped during the session by the Intelligent Test Runner.
Test Skipping Enabled
Indicates if Intelligent Test Runner was enabled for the test session.
Test Skipping Type
The method or criteria used by the Intelligent Test Runner to determine which tests to skip.
Tests Skipped
The total count of tests that were not executed during the test session, which may include tests that were configured to skip, or were set as manual exclusions.
Time Saved
The length of time saved for the session by Intelligent Test Runner usage.
For example, to filter test session runs that have Test Skipping Enabled, you can use @test.itr.tests_skipping.enabled:true in the search query.
Then, click on a test session run and see the amount of time saved by the Intelligent Test Runner in the Test Session Details section on the test session side panel.
Further Reading
Additional helpful documentation, links, and articles:
