Overview
Intelligent Test Runner is Datadog’s test impact analysis solution. It automatically selects and runs only the relevant tests for a given commit based on the code being changed. Significantly reduce time spent testing and overall CI costs, while maintaining test coverage.
Intelligent Test Runner works by analyzing your test suite to determine the code each test covers, and then cross-referencing that coverage with the files impacted by a new code change. Datadog uses this information to run a selection of relevant, impacted tests, omitting the ones unaffected by the code change and reducing the overall testing duration.
By minimizing the number of tests run per commit, Intelligent Test Runner reduces the frequency of flaky tests disrupting your pipelines. This can be particularly frustrating when the test flaking is unrelated to the code change being tested. After enabling Intelligent Test Runner for your test services, you can limit each commit to its relevant tests to ensure that flaky tests unrelated to your code change don’t end up arbitrarily breaking your build.
Limitations
There are known limitations in the current implementation of Intelligent Test Runner that can cause it to skip tests that should be run under certain conditions. Specifically, Intelligent Test Runner is not able to detect changes in library dependencies, compiler options, external services or changes to data files in data-driven tests.
To override Intelligent Test Runner and run all tests, add ITR:NoSkip (case insensitive) anywhere in your Git commit message. You can also add a list of excluded branches and tracked files.
Set up a Datadog library
Before setting up Intelligent Test Runner, you must configure Test Visibility for your particular language. If you are reporting data through the Agent, use v6.40 or 7.40 and later.
Choose a language to set up Intelligent Test Runner in Datadog:
Configuration
Once you have set up your Datadog library for Intelligent Test Runner, configure it from the Test Service Settings page. Enabling Intelligent Test Runner requires the Intelligent Test Runner Activation Write permission.
Excluded branches
Due to the limitations described above, the default branch of your repository is automatically excluded from having Intelligent Test Runner enabled. Datadog recommends this configuration to ensure that all of your tests run prior to reaching production.
If there are other branches you want to exclude, add them on the Test Service Settings page. The query bar supports using the wildcard character * to exclude any branches that match, such as release_*.
Tracked files
Additionally, you may specify a set of tracked files. Intelligent Test Runner runs all tests if any of these files change.
You may also use the * and ** wildcard characters to match multiple files or directories. For instance, **/*.mdx matches any .mdx file in the repository.
We recommend that you use tracked files for your dependency files (for example, package.json, requirements.txt) and for data files from data-driven tests, such as tests/data/**.
Explore test sessions
You can explore the time savings you get from Intelligent Test Runner by looking at the test commit page and test sessions panel.
When Intelligent Test Runner is active and skipping tests, purple text displays the amount of time saved on each test session or on each commit. The duration bar also changes color to purple so you can quickly identify which test sessions are using Intelligent Test Runner on the Test Runs page.
Explore adoption and global savings
Track your organization’s savings and adoption of Intelligent Test Runner through the out-of-the-box Intelligent Test Runner dashboard. The dashboard includes widgets to track your overall savings as well as a per-repository, per-committer, and per-service view of the data. View the dashboard to understand which parts of your organization are using and getting the most out of Intelligent Test Runner.
The dashboard also tracks adoption of Intelligent Test Runner throughout your organization.
Further Reading
Additional helpful documentation, links, and articles:
