Getting Started with Intelligent Test Runner

Cette page n'est pas encore disponible en français, sa traduction est en cours.
Si vous avez des questions ou des retours sur notre projet de traduction actuel, n'hésitez pas à nous contacter.

Overview

Intelligent Test Runner allows you to streamline your organization’s test impact analysis by intelligently excluding irrelevant tests that are not affected by recent code changes.

With Test Visibility, development teams can configure Intelligent Test Runner for their test services, set branches to exclude (such as the default branch), and define files to be tracked (which triggers full runs of all tests when any tracked file changes).

A Venn diagram of the components for the Intelligent Test Runner: tracked files, excluded branches, and skipped tests
A Venn diagram displaying how Intelligent Test Runner defines an excluded test by using tracked files, excluded branches, and passed tests.

Configure and enable Intelligent Test Runner for your test services to reduce unnecessary testing time, enhance CI test efficiency, and reduce costs, while maintaining the reliability and performance across your CI environments.

Intelligent Test Runner uses code coverage data to determine whether or not tests should be skipped. For more information, see How Intelligent Test Runner Works in Datadog.

Set up the Intelligent Test Runner

To set up the Intelligent Test Runner, see the following documentation for your programming language:

.net
java
javascript
python
swift
ruby

Enable the Intelligent Test Runner

To enable the Intelligent Test Runner:

  1. Navigate to Software Delivery > Test Visibility > Settings.
  2. On the Test Services tab, click Configure in the Intelligent Test Runner column for a service.
Enable Intelligent Test Runner for a test service on the Test Service Settings page

You must have the Intelligent Test Runner Activation Write permission. For more information, see the Datadog Role Permissions documentation.

Disabling the Intelligent Test Runner on critical branches (such as your default branch) ensures comprehensive test coverage, whereas enabling it to run on feature or development branches helps maximize testing efficiency.

Configure the Intelligent Test Runner

You can configure the Intelligent Test Runner to prevent specific tests from being skipped. These tests are known as unskippable tests, and are run regardless of code coverage data.

To configure the Intelligent Test Runner:

  1. Click the Status toggle to enable Intelligent Test Runner.
  2. Specify any branches to exclude (typically the default branch of a repository). Intelligent Test Runner does not skip tests for these branches.
  3. Specify file directories and files to track (for example, documentation/content/** or domains/shopist/apps/api/BUILD.bazel). Intelligent Test Runner runs all CI tests when any of these tracked files change.
  4. Click Save Settings.
Enable the Intelligent Test Runner, provide branches for the Intelligent Test Runner to exclude, and add files for the Intelligent Test Runner to track and run tests when any changes happen

Once you’ve configured Intelligent Test Runner on a test service, execute a test suite run on your default branch. This establishes a baseline for the Intelligent Test Runner to accurately skip irrelevant tests in future commits.

Use Intelligent Test Runner data

Explore the data collected by enabling Intelligent Test Runner, such as the time savings achieved by skipping tests, as well as your organization’s usage of Intelligent Test Runner, to improve your CI efficiency.

The out-of-the-box dashboard displaying information about the time saved by tests skipped by Intelligent Test Runner, and your organization's usage of Intelligent Test Runner

You can create dashboards to visualize your testing metrics, or use an out-of-the-box dashboard containing widgets populated with data collected by the Intelligent Test Runner to help you identify areas of improvement with usage patterns and trends.

Examine results in the Test Visibility Explorer

The Test Visibility Explorer allows you to create visualizations and filter test spans using the data collected from Test Visibility and the Intelligent Test Runner. When the Intelligent Test Runner is active, it displays the amount of time saved for each test session or commit. The duration bars turn purple to indicate active test skipping.

Navigate to Software Delivery > Test Visibility > Test Runs and select Session to start filtering your test session span results.

Test session results in the Test Visibility Explorer filtered on tests skipped by the Intelligent Test Runner

Navigate to Software Delivery > Test Visibility > Test Runs and select Module to start filtering your test module span results.

Test module results in the Test Visibility Explorer filtered on tests skipped by the Intelligent Test Runner

Navigate to Software Delivery > Test Visibility > Test Runs and select Suite to start filtering your test suite span results.

Test suite results in the Test Visibility Explorer filtered on tests skipped by the Intelligent Test Runner

Navigate to Software Delivery > Test Visibility > Test Runs and select Test to start filtering your test span results.

Test results in the Test Visibility Explorer filtered on tests skipped by the Intelligent Test Runner

Use the following out-of-the-box Intelligent Test Runner facets to customize the search query:

Code Coverage Enabled
Indicates whether code coverage tracking was active during the test session.
Skipped by ITR
Number of tests that were skipped during the session by the Intelligent Test Runner.
Test Skipping Enabled
Indicates if Intelligent Test Runner was enabled for the test session.
Test Skipping Type
The method or criteria used by the Intelligent Test Runner to determine which tests to skip.
Tests Skipped
The total count of tests that were not executed during the test session, which may include tests that were configured to skip, or were set as manual exclusions.
Time Saved
The length of time saved for the session by Intelligent Test Runner usage.

For example, to filter test session runs that have Test Skipping Enabled, you can use @test.itr.tests_skipping.enabled:true in the search query.

A side panel displaying the first test session run where the Test Skipping feature is enabled for the Intelligent Test Runner

Then, click on a test session run and see the amount of time saved by the Intelligent Test Runner in the Test Session Details section on the test session side panel.

Further Reading