Intelligent Test Runner is Datadog’s test impact analysis solution. It allows you to only run the impacted tests for a given commit and skip any that are irrelevant.
By only running tests on relevant code, when tests fail, it is likely a legitimate failure that pertains to the modified code.
Limitations during beta
During the beta of Intelligent Test Runner there are certain limitations:
Some of the environment variables required in the following sections are only required during the beta.
Intelligent Test Runner only works without the Datadog Agent. The configuration steps in this page cause the Datadog Library to send data directly to the backend without going through the Agent. If you are currently using the Agent, you will lose correlation with host metrics.
There are known limitations in the current implementation of Intelligent Test Runner that can cause it to skip tests that should be run. Intelligent Test Runner is not able to detect:
Changes in library dependencies.
Changes in compiler options.
Changes in external services.
Changes to data files in data-driven tests.
To override Intelligent Test Runner and run all tests, add ITR:NoSkip (case insensitive) anywhere in your Git commit message.
Setup Datadog Library
Prior to setting up Intelligent Test Runner, you must have finished setting up Test Visibility for your particular language.
JavaScript
To enable Intelligent Test Runner, the following environment variables need to be set:
DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)
Enables or disables Agentless mode. Default: false Note: Required only during Beta
DD_API_KEY (Required)
The Datadog API key used to upload the test results. Default: (empty)
Flag to enable git metadata upload. Default: false Note: Required only during Beta
DD_CIVISIBILITY_ITR_ENABLED=true (Required)
Flag to enable test skipping. Default: false Note: Required only during Beta
After setting these environment variables, run your tests as you normally do:
NODE_OPTIONS="-r dd-trace/ci/init"DD_ENV=ci DD_SERVICE=my-javascript-app DD_CIVISIBILITY_AGENTLESS_ENABLED=trueDD_API_KEY=$API_KEYDD_CIVISIBILITY_GIT_UPLOAD_ENABLED=trueDD_CIVISIBILITY_ITR_ENABLED=true yarn test
UI activation
In addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.
Compatibility
Intelligent Test Runner is only supported in the following versions and testing frameworks:
dd-trace>=3.4.0
jest>=24.8.0
Only jest-circus/runner is supported as testRunner.
Only jsdom and node are supported as test environments.
.NET
To enable Intelligent Test Runner, the version of the dd-trace tool must be >= 2.16.0 (execute dd-trace --version to get the version of the tool) and the following environment variables must be set:
DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)
Enables or disables Agentless mode. Default: false Note: Required only during Beta
DD_API_KEY (Required)
The Datadog API key used to upload the test results. Default: (empty)
The Datadog site to upload results to. Default: datadoghq.com Selected site:
DD_CIVISIBILITY_ITR_ENABLED=true (Required)
Flag to enable Intelligent Test Runner. Default: false Note: Required only during Beta
After setting these environment variables, run your tests as you normally do by using dotnet test or VSTest.Console.exe:
dd-trace ci run --dd-service=my-dotnet-app --dd-env=ci -- dotnet test
dd-trace ci run --dd-service=my-dotnet-app --dd-env=ci -- VSTest.Console.exe {test_assembly}.dll
UI activation
In addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.
Swift
To enable Intelligent Test Runner, the version of the dd-sdk-swift framework must be >= 2.2.0. The Code Coverage option must also be enabled in the test settings of your scheme or test plan, or --enable-code-coverage must be added to your swift test command (if using a SPM target).
The following environment variables must also be set:
DD_TEST_RUNNER
Enables or disables the instrumentation of tests. Set this value to $(DD_TEST_RUNNER) so you can enable and disable test instrumentation with a environment variable defined outside of the test process (for example, in the CI build). Default: false Recommended: $(DD_TEST_RUNNER)
DD_API_KEY (Required)
The Datadog API key used to upload the test results. Default: (empty)
The Datadog site to upload results to. Default: datadoghq.com Selected site:
UI activation
In addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.
Setup CI Job
Intelligent Test Runner uses git metadata information (commit history) to work. However, some CI providers use a git shallow clone (git clone --depth=0) which only downloads the target commit without downloading any historical commit information. This setup does not contain enough information for Intelligent Test Runner to work. If your CI is using shallow clones, it must be changed.
An efficient alternative to shallow clones are partial clones (supported in Git v2.27+), which will clone the current commit plus the necessary git metadata without retrieving all past versions of all files: git clone --filter=blob:none.
Configuration
The default branch of your repository is automatically excluded from having Intelligent Test Runner enabled. Due to the limitations described above, the Intelligent Test Runner might skip some of the tests that should be run, therefore Datadog recommends to continue running all tests in your default branch (or the branch you release from).
If there are other branches you want to exclude, you can add them from the Intelligent Test Runner settings page. The query bar supports the wildcard character * to exclude any branches that match.
Further Reading
Additional helpful documentation, links, and articles: