Compatibility

Test Impact Analysis is only supported in the following versions and testing frameworks:

• datadog-ci >= 1.0
• Ruby >= 2.7
  • JRuby is not supported.
• rspec >= 3.0.0
• minitest >= 5.0.0
• cucumber >= 3.0.0

Setup

Test Optimization

Prior to setting up Test Impact Analysis, set up Test Optimization for Ruby. If you are reporting data through the Agent, use v6.40 and later or v7.40 and later.

Activate Test Impact Analysis for the test service

You, or a user in your organization with the Intelligent Test Runner Activation (intelligent_test_runner_activation_write) permission, must activate Test Impact Analysis on the Test Service Settings page.

Run tests with Test Impact Analysis enabled

After completing setup, run your tests as you normally do.

Known limitations

Test Impact Analysis uses code coverage data to determine whether or not tests should be skipped. In certain situations, code coverage data alone is not enough to determine whether to skip a test.

Coverage limitations

The following limitations apply to how code coverage is collected:

• Non-Ruby files are not tracked by default: Changes to non-Ruby files such as fixtures, YAML configuration, i18n translation files, or other data files are not detected by code coverage. Tests that read data from these files may be incorrectly skipped when these files change.
• Suite-level hooks: Code coverage for suite-level hooks (for example, before(:all) or before(:context) in RSpec) is attributed to the entire test suite rather than individual tests. This may affect skip decisions for tests that depend on setup performed in these hooks.
• Forked processes: Per-test code coverage only collects coverage for the main process. Tests that spawn child processes or use forked execution do not have coverage collected for code running in those processes.
• Constant references across files: Accessing a constant defined in another file does not count as covered code. If a test’s behavior depends on constants defined elsewhere, changes to those constants may not trigger the test to run. This is supported as an experimental feature. See Static dependencies analysis for more information.

External dependencies

Tests that interact with external systems may be incorrectly skipped:

• Tests that make calls to external APIs or services (such as remote REST APIs)
• Tests that run external processes or shell commands
• Tests that depend on global shared state (for example, caches, databases, or files created by a different test or process)

Recommendations

When you encounter these limitations, consider the following approaches:

• Mark tests as unskippable: For tests that make external calls, fork processes, or depend on global shared state, mark them as unskippable to ensure they always run.
• Configure tracked files: If your tests depend on non-Ruby files like fixtures, i18n files, or configuration files, add these files to your tracked files configuration. This causes all tests to run when these files change.

Rails system tests

Test Impact Analysis supports Rails system tests that use ActionDispatch::SystemTestCase or ApplicationSystemTestCase, as long as the server runs in the same process as the test code. This is the default behavior in Rails system tests.

Static dependencies analysis (experimental)

By default, Ruby’s code coverage does not track constant references across files. When a test accesses a constant defined in another file, changes to that constant’s file may not trigger the test to run.

Static dependencies analysis addresses this limitation by analyzing Ruby’s compiled bytecode to find constant references and resolve them to source files.

Compatibility

Static dependencies analysis requires:

• datadog-ci >= 1.26.0
• Ruby >= 3.2

Enabling static dependencies analysis

To enable this experimental feature, set the following environment variable in your CI configuration:

DD_TEST_OPTIMIZATION_TIA_STATIC_DEPS_COVERAGE_ENABLED (Optional)

Enable static dependencies analysis to track constant references across files.
Default: false

Limitations

The following limitations apply to static dependencies analysis:

• Requires eager loading: Only works when your code is eager loaded in tests. If constants are loaded dynamically, their source locations cannot be resolved correctly.
• Dynamic lookups not supported: Constants accessed through metaprogramming (such as const_get or constantize) are not detected.
• Unqualified constant names: Constants accessed without their full namespace path may not be resolved correctly (for example MyConst instead of MyModule::MyConst)

Unskippable tests

You can override the Test Impact Analysis’s behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.

Marking tests as unskippable

# mark the whole file as unskippable
RSpec.describe MyClass, datadog_itr_unskippable: true do
  describe "#my_method" do
    context "when called without arguments" do
      it "works" do
      end
    end
  end
end

# mark one test as unskippable
RSpec.describe MyClass do
  describe "#my_method" do
    context "when called without arguments" do
      it "works", datadog_itr_unskippable: true do
      end
    end
  end
end

# mark specific block as unskippable
RSpec.describe MyClass do
  describe "#my_method", datadog_itr_unskippable: true do
    context "when called without arguments" do
      it "works" do
      end
    end
  end
end

@datadog_itr_unskippable
Feature: Unskippable feature
  Scenario: Say greetings
    When the greeter says greetings
    Then I should have heard "greetings"

Feature: An unskippable scenario

  @datadog_itr_unskippable
  Scenario: Unskippable scenario
    When the ITR wants to skip this scenario
    Then it will never be skipped

  Scenario: Skippable scenario
    When the ITR wants to skip this scenario
    Then it will be skipped

# mark the whole class unskippable
class MyTest < Minitest::Test
  datadog_itr_unskippable

  def test_my_method
  end
end

# here only test1 and test2 are unskippable
class MyTest < Minitest::Test
  datadog_itr_unskippable "test1", "test2"

  def test1
  end

  def test2
  end

  def test3
  end
end

Temporarily disabling Test Impact Analysis

Test Impact Analysis can be disabled locally by setting the DD_CIVISIBILITY_ITR_ENABLED environment variable to false or 0.

DD_CIVISIBILITY_ITR_ENABLED (Optional)

Enable the Test Impact Analysis coverage and test skipping features
Default: (true)

Further Reading

Additional helpful documentation, links, and articles:

Explore Test Results and PerformanceDOCUMENTATION Troubleshooting Test OptimizationDOCUMENTATION