Intelligent Test Runner for CI Visibility is in private beta. You can request access by completing the form on the CI Visibility Getting Started page.
Request AccessIntelligent Test Runner is Datadog’s test impact analysis solution. It allows you to only run the impacted tests for a given commit and skip any that are irrelevant.
By only running tests on relevant code, when tests fail, it is likely a legitimate failure that pertains to the modified code.
During the beta of Intelligent Test Runner there are certain limitations:
To override Intelligent Test Runner and run all tests, add ITR:NoSkip (case insensitive) anywhere in your Git commit message.
Prior to setting up Intelligent Test Runner, you must have finished setting up Test Visibility for your particular language.
To enable Intelligent Test Runner, the following environment variables need to be set:
DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)falseDD_API_KEY (Required)(empty)DD_APPLICATION_KEY (Required)(empty)DD_SITE (Required)datadoghq.comDD_CIVISIBILITY_GIT_UPLOAD_ENABLED=true (Required)falseDD_CIVISIBILITY_ITR_ENABLED=true (Required)falseAfter setting these environment variables, run your tests as you normally do:
NODE_OPTIONS="-r dd-trace/ci/init" DD_ENV=ci DD_SERVICE=my-javascript-app DD_CIVISIBILITY_AGENTLESS_ENABLED=true DD_API_KEY=$API_KEY DD_CIVISIBILITY_GIT_UPLOAD_ENABLED=true DD_CIVISIBILITY_ITR_ENABLED=true yarn testIn addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.
Intelligent Test Runner is only supported in the following versions and testing frameworks:
dd-trace>=3.4.0jest>=24.8.0jest-circus/runner is supported as testRunner.jsdom and node are supported as test environments.To enable Intelligent Test Runner, the version of the dd-trace tool must be >= 2.16.0 (execute dd-trace --version to get the version of the tool) and the following environment variables must be set:
DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)falseDD_API_KEY (Required)(empty)DD_APPLICATION_KEY (Required)(empty)DD_SITE (Required)datadoghq.comDD_CIVISIBILITY_ITR_ENABLED=true (Required)falseAfter setting these environment variables, run your tests as you normally do by using dotnet test or VSTest.Console.exe:
dd-trace ci run --dd-service=my-dotnet-app --dd-env=ci -- dotnet testdd-trace ci run --dd-service=my-dotnet-app --dd-env=ci -- VSTest.Console.exe {test_assembly}.dllIn addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.
To enable Intelligent Test Runner, the version of the dd-sdk-swift framework must be >= 2.2.0. The Code Coverage option must also be enabled in the test settings of your scheme or test plan, or --enable-code-coverage must be added to your swift test command (if using a SPM target).
The following environment variables must also be set:
DD_TEST_RUNNER$(DD_TEST_RUNNER) so you can enable and disable test instrumentation with a environment variable defined outside of the test process (for example, in the CI build).false$(DD_TEST_RUNNER)DD_API_KEY (Required)(empty)DD_APPLICATION_KEY (Required)(empty)DD_SITE (Required)datadoghq.comIn addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.
Intelligent Test Runner uses git metadata information (commit history) to work. However, some CI providers use a git shallow clone (git clone --depth=0) which only downloads the target commit without downloading any historical commit information. This setup does not contain enough information for Intelligent Test Runner to work. If your CI is using shallow clones, it must be changed.
An efficient alternative to shallow clones are partial clones (supported in Git v2.27+), which will clone the current commit plus the necessary git metadata without retrieving all past versions of all files: git clone --filter=blob:none.
The default branch of your repository is automatically excluded from having Intelligent Test Runner enabled. Due to the limitations described above, the Intelligent Test Runner might skip some of the tests that should be run, therefore Datadog recommends to continue running all tests in your default branch (or the branch you release from).
If there are other branches you want to exclude, you can add them from the Intelligent Test Runner settings page. The query bar supports the wildcard character * to exclude any branches that match.
Additional helpful documentation, links, and articles:
