Intelligent Test Runner

Join the Beta!

Intelligent Test Runner for CI Visibility is in private beta. You can request access by completing the form on the CI Visibility Getting Started page.

Request Access

Overview

Intelligent Test Runner is Datadog’s test impact analysis solution. It allows you to only run the impacted tests for a given commit and skip any that are irrelevant.

Intelligent test runner enabled in test service settings in the CI section of Datadog.

By only running tests on relevant code, when tests fail, it is likely a legitimate failure that pertains to the modified code.

Limitations during beta

During the beta of Intelligent Test Runner there are certain limitations:

  • Some of the environment variables required in the following sections are only required during the beta.
  • Intelligent Test Runner only works without the Datadog Agent. The configuration steps in this page cause the Datadog Library to send data directly to the backend without going through the Agent. If you are currently using the Agent, you will lose correlation with host metrics.
  • There are known limitations in the current implementation of Intelligent Test Runner that can cause it to skip tests that should be run. Intelligent Test Runner is not able to detect:
    • Changes in library dependencies.
    • Changes in compiler options.
    • Changes in external services.
    • Changes to data files in data-driven tests.

To override Intelligent Test Runner and run all tests, add ITR:NoSkip (case insensitive) anywhere in your Git commit message.

Setup Datadog Library

Prior to setting up Intelligent Test Runner, you must have finished setting up Test Visibility for your particular language.

JavaScript

To enable Intelligent Test Runner, the following environment variables need to be set:

DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)
Enables or disables Agentless mode.
Default: false
Note: Required only during Beta
DD_API_KEY (Required)
The Datadog API key used to upload the test results.
Default: (empty)
DD_APPLICATION_KEY (Required)
The Datadog Application key used to query the tests to be skipped.
Default: (empty)
DD_SITE (Required)
The Datadog site to upload results to.
Default: datadoghq.com
Selected site:
DD_CIVISIBILITY_GIT_UPLOAD_ENABLED=true (Required)
Flag to enable git metadata upload.
Default: false
Note: Required only during Beta
DD_CIVISIBILITY_ITR_ENABLED=true (Required)
Flag to enable test skipping.
Default: false
Note: Required only during Beta

After setting these environment variables, run your tests as you normally do:

NODE_OPTIONS="-r dd-trace/ci/init" DD_ENV=ci DD_SERVICE=my-javascript-app DD_CIVISIBILITY_AGENTLESS_ENABLED=true DD_API_KEY=$API_KEY DD_CIVISIBILITY_GIT_UPLOAD_ENABLED=true DD_CIVISIBILITY_ITR_ENABLED=true yarn test

UI activation

In addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.

Compatibility

Intelligent Test Runner is only supported in the following versions and testing frameworks:

  • dd-trace>=3.4.0
  • jest>=24.8.0
    • Only jest-circus/runner is supported as testRunner.
    • Only jsdom and node are supported as test environments.

.NET

To enable Intelligent Test Runner, the version of the dd-trace tool must be >= 2.16.0 (execute dd-trace --version to get the version of the tool) and the following environment variables must be set:

DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)
Enables or disables Agentless mode.
Default: false
Note: Required only during Beta
DD_API_KEY (Required)
The Datadog API key used to upload the test results.
Default: (empty)
DD_APPLICATION_KEY (Required)
The Datadog Application key used to query the tests to be skipped.
Default: (empty)
DD_SITE (Required)
The Datadog site to upload results to.
Default: datadoghq.com
Selected site:
DD_CIVISIBILITY_ITR_ENABLED=true (Required)
Flag to enable Intelligent Test Runner.
Default: false
Note: Required only during Beta

After setting these environment variables, run your tests as you normally do by using dotnet test or VSTest.Console.exe:

dd-trace ci run --dd-service=my-dotnet-app --dd-env=ci -- dotnet test
dd-trace ci run --dd-service=my-dotnet-app --dd-env=ci -- VSTest.Console.exe {test_assembly}.dll

UI activation

In addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.

Swift

To enable Intelligent Test Runner, the version of the dd-sdk-swift framework must be >= 2.2.0. The Code Coverage option must also be enabled in the test settings of your scheme or test plan, or --enable-code-coverage must be added to your swift test command (if using a SPM target).

The following environment variables must also be set:

DD_TEST_RUNNER
Enables or disables the instrumentation of tests. Set this value to $(DD_TEST_RUNNER) so you can enable and disable test instrumentation with a environment variable defined outside of the test process (for example, in the CI build).
Default: false
Recommended: $(DD_TEST_RUNNER)
DD_API_KEY (Required)
The Datadog API key used to upload the test results.
Default: (empty)
DD_APPLICATION_KEY (Required)
The Datadog Application key used to query the tests to be skipped.
Default: (empty)
DD_SITE (Required)
The Datadog site to upload results to.
Default: datadoghq.com
Selected site:

UI activation

In addition to setting the environment variables, you or a user in your organization with admin permissions must activate the Intelligent Test Runner on the Test Service Settings page.

Setup CI Job

Intelligent Test Runner uses git metadata information (commit history) to work. However, some CI providers use a git shallow clone (git clone --depth=0) which only downloads the target commit without downloading any historical commit information. This setup does not contain enough information for Intelligent Test Runner to work. If your CI is using shallow clones, it must be changed.

An efficient alternative to shallow clones are partial clones (supported in Git v2.27+), which will clone the current commit plus the necessary git metadata without retrieving all past versions of all files: git clone --filter=blob:none.

Configuration

The default branch of your repository is automatically excluded from having Intelligent Test Runner enabled. Due to the limitations described above, the Intelligent Test Runner might skip some of the tests that should be run, therefore Datadog recommends to continue running all tests in your default branch (or the branch you release from).

If there are other branches you want to exclude, you can add them from the Intelligent Test Runner settings page. The query bar supports the wildcard character * to exclude any branches that match.

Select branches to exclude from intelligent test runner

Further Reading