Compatibility

Intelligent Test Runner is only supported in the following versions and testing frameworks:

  • jest>=24.8.0
    • From dd-trace>=3.16.0 or dd-trace>=2.29.0.
    • Only jest-circus/runner is supported as testRunner.
    • Only jsdom and node are supported as test environments.
  • mocha>=5.2.0
    • From dd-trace>=3.16.0 or dd-trace>=2.29.0.
    • Run mocha with nyc to enable code coverage.
  • cucumber-js>=7.0.0
    • From dd-trace>=3.16.0 or dd-trace>=2.29.0.
    • Run cucumber-js with nyc to enable code coverage.
  • cypress>=6.7.0
    • From dd-trace>=4.2.0, dd-trace>=3.23.0 or dd-trace>=2.36.0.
    • Instrument your web application with code coverage: see more in Cypress setup
  • If you’re reporting data through the Datadog Agent, use v6.40+/v7.40+.

Setup

Test visibility setup

Prior to setting up Intelligent Test Runner, set up Test Visibility for Javascript and Typescript. If you are reporting data through the Agent, use v6.40+/v7.40+.

UI activation

You, or a user in your organization with “Intelligent Test Runner Activation” permissions, must activate the Intelligent Test Runner on the Test Service Settings page.

Intelligent test runner enabled in test service settings in the CI section of Datadog.

Application key and permissions

The Intelligent Test Runner requires an application key mapped to an owner role with the CI Visibility Read (ci_visibility_read) permission.

For more information on roles and permissions, see the CI Visibility section of the Datadog Role Permissions documentation.

Configuring the test runner environment

Additional Datadog Agent configuration

In addition to the Datadog API Key, a Datadog application key must be specified in the Agent Configuration File.

app_key (Required)
The Datadog application key used to query the tests to be skipped.
Default: (empty)

After configuring the Datadog Agent, run your tests as you normally do:

NODE_OPTIONS="-r dd-trace/ci/init" DD_ENV=ci DD_SERVICE=my-javascript-app yarn test

Environment variables

In addition to the Datadog API Key, a Datadog application key must be provided.

DD_APP_KEY (Required)
The Datadog application key used to query the tests to be skipped.
Default: (empty)

After setting these environment variables, run your tests as you normally do:

NODE_OPTIONS="-r dd-trace/ci/init" DD_ENV=ci DD_SERVICE=my-javascript-app DD_CIVISIBILITY_AGENTLESS_ENABLED=true DD_API_KEY=$DD_API_KEY DD_APPLICATION_KEY=$DD_APPLICATION_KEY yarn test

Cypress setup

For Intelligent Test Runner for Cypress to work, you must instrument your web application with code coverage. You can read more about enabling code coverage in the Cypress documentation. To check that you’ve successfully enabled code coverage, navigate to your web app with Cypress and check the global variable window.__coverage__. This is what dd-trace uses to collect code coverage for Intelligent Test Runner.

Suite skipping

Intelligent Test Runner for Javascript skips entire test suites (test files) rather than individual tests.

Inconsistent test durations

In some frameworks, such as jest, there are cache mechanisms that make tests faster after other tests have run (see jest cache docs). If Intelligent Test Runner is skipping all but a few test files, these suites might run slower than they usually do. This is because they run with a colder cache. Regardless of this, total execution time for your test command should still be reduced.

Further Reading