Compatibility

Intelligent Test Runner is only supported in the following versions and testing frameworks:

• datadog-ci >= 1.0.0.beta3
• Ruby >= 2.7
  • JRuby is not supported.
• rspec >= 3.0.0
• minitest >= 5.0.0
  • Rails parallel testing is not supported as of library version 1.0.0.beta3.
• cucumber >= 3.0.0

Setup

Test Visibility

Prior to setting up Intelligent Test Runner, set up Test Visibility for Ruby. If you are reporting data through the Agent, use v6.40 and later or v7.40 and later.

Activate Intelligent Test Runner for the test service

You, or a user in your organization with the Intelligent Test Runner Activation (intelligent_test_runner_activation_write) permission, must activate the Intelligent Test Runner on the Test Service Settings page.

Use the latest version of a Test Visibility library

Intelligent Test Runner for Ruby is available in datadog-ci gem version 1.0.0.beta3 and later.

Add to your Gemfile:

group :test do
  gem 'datadog-ci', '~> 1.0.0.beta3'
end

If you use other datadog products, upgrade to 2.0.0.beta2 version of gem datadog:

gem 'datadog', '~> 2.0.0.beta2'

group :test do
  gem 'datadog-ci', '~> 1.0.0.beta3'
end

Run tests with the Intelligent Test Runner enabled

DD_CIVISIBILITY_ITR_ENABLED="true" DD_ENV=ci DD_SERVICE=my-app bundle exec rake test

DD_CIVISIBILITY_ITR_ENABLED="true" DD_ENV=ci DD_SERVICE=my-app DD_CIVISIBILITY_AGENTLESS_ENABLED=true DD_API_KEY=$DD_API_KEY bundle exec rake test

Disabling skipping for specific tests

You can override the Intelligent Test Runner’s behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.

Why make tests unskippable?

The Intelligent Test Runner uses code coverage data to determine whether or not tests should be skipped. In some cases, this data may not be sufficient to make this determination.

Examples include:

• Tests that read data from text files
• Tests that interact with APIs outside of the code being tested (such as remote REST APIs)
• Tests that run external processes
• Tests that use threads or fork process (code coverage tracks only code executed in main thread)

Designating tests as unskippable ensures that the Intelligent Test Runner runs them regardless of coverage data.

Marking tests as unskippable

# mark the whole file as unskippable
RSpec.describe MyClass, datadog_itr_unskippable: true do
  describe "#my_method" do
    context "when called without arguments" do
      it "works" do
      end
    end
  end
end

# mark one test as unskippable
RSpec.describe MyClass do
  describe "#my_method" do
    context "when called without arguments" do
      it "works", datadog_itr_unskippable: true do
      end
    end
  end
end

# mark specific block as unskippable
RSpec.describe MyClass do
  describe "#my_method", datadog_itr_unskippable: true do
    context "when called without arguments" do
      it "works" do
      end
    end
  end
end

@datadog_itr_unskippable
Feature: Unskippable feature
  Scenario: Say greetings
    When the greeter says greetings
    Then I should have heard "greetings"

Feature: An unskippable scenario

  @datadog_itr_unskippable
  Scenario: Unskippable scenario
    When the ITR wants to skip this scenario
    Then it will never be skipped

  Scenario: Skippable scenario
    When the ITR wants to skip this scenario
    Then it will be skipped

# mark the whole class unskippable
class MyTest < Minitest::Test
  datadog_itr_unskippable

  def test_my_method
  end
end

# here only test1 and test2 are unskippab;e
class MyTest < Minitest::Test
  datadog_itr_unskippable "test1", "test2"

  def test1
  end

  def test2
  end

  def test3
  end
end

Further Reading

Additional helpful documentation, links, and articles:

Explore Test Results and PerformanceDOCUMENTATION Troubleshooting CI VisibilityDOCUMENTATION