Compatibility

Intelligent Test Runner is only supported in the following versions and testing frameworks:

• pytest>=6.8.0 up to <=7.0.0
  • From ddtrace>=2.1.0.
  • From Python>=3.7.
  • Requires coverage>=5.5.
  • Incompatible with pytest-cov (see known limitations)
• unittest
  • From ddtrace>=2.2.0.
  • From Python>=3.7.
• coverage
  • Incompatible for coverage collection (see known limitations)

Setup

Test Visibility

Prior to setting up Intelligent Test Runner, set up Test Visibility for Python. If you are reporting data through the Agent, use v6.40 and later or v7.40 and later.

Activate Intelligent Test Runner for the test service

You, or a user in your organization with the Intelligent Test Runner Activation (intelligent_test_runner_activation_write) permission, must activate the Intelligent Test Runner on the Test Service Settings page.

Required dependencies

The Intelligent Test Runner requires the coverage package.

Install the package in your CI test environment by specifying it in the relevant requirements file, for example, or using pip:

pip install coverage

See known limitations if you are already using the coverage package or a plugin like pytest-cov.

Running tests with the Intelligent Test Runner enabled

The Intelligent Test Runner is enabled when you run tests with the Datadog integration active. Run your tests with the following command:

DD_ENV=ci DD_SERVICE=my-python-app pytest --ddtrace

DD_ENV=ci DD_SERVICE=my-python-app ddtrace-run python -m unittest

Temporarily disabling the Intelligent Test Runner

The Intelligent Test Runner can be disabled locally by setting the DD_CIVISIBILITY_ITR_ENABLED environment variable to false or 0.

DD_CIVISIBILITY_ITR_ENABLED (Optional)

Enable the Intelligent Test Runner coverage and test skipping features
Default: (true)

Run the following command to disable the Intelligent Test Runner:

DD_ENV=ci DD_SERVICE=my-python-app DD_CIVISIBILITY_ITR_ENABLED=false pytest --ddtrace

DD_ENV=ci DD_SERVICE=my-python-app DD_CIVISIBILITY_ITR_ENABLED=false ddtrace-run python -m unittest

Disabling skipping for specific tests

You can override the Intelligent Test Runner’s behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.

Why make tests unskippable?

The Intelligent Test Runner uses code coverage data to determine whether or not tests should be skipped. In some cases, this data may not be sufficient to make this determination.

Examples include:

• Tests that read data from text files
• Tests that interact with APIs outside of the code being tested (such as remote REST APIs)

Designating tests as unskippable ensures that the Intelligent Test Runner runs them regardless of coverage data.

import pytest

@pytest.mark.skipif(False, reason="datadog_itr_unskippable")
def test_function():
    assert True

import pytest

pytestmark = pytest.mark.skipif(False, reason="datadog_itr_unskippable")

def test_function():
    assert True

import unittest

class MyTestCase(unittest.TestCase):
  @unittest.skipIf(False, reason="datadog_itr_unskippable")
  def test_function(self):
      assert True

Known limitations

Code coverage collection

Interaction with coverage tools

Coverage data may appear incomplete when the Intelligent Test Runner is enabled. Lines of code that would normally be covered by tests are not be covered when these tests are skipped.

Interaction with the coverage package

The Intelligent Test Runner uses the coverage package’s API to collect code coverage. Data from coverage run or plugins like pytest-cov is incomplete as a result of ddtrace’s use of the Coverage class.

Some race conditions may cause exceptions when using pytest plugins such as pytest-xdist that change test execution order or introduce parallelization.

Further reading

Additional helpful documentation, links, and articles:

Explore Test Results and PerformanceDOCUMENTATION Troubleshooting CI VisibilityDOCUMENTATION