Intelligent Test Runner for Python

Compatibility

Intelligent Test Runner is only supported in the following versions and testing frameworks:

• pytest>=6.8.0
  • From ddtrace>=1.18.0.
  • From Python>=3.7.
  • Requires coverage>=5.5.

Setup

Test Visibility setup

Prior to setting up Intelligent Test Runner, set up Test Visibility for Python. If you are reporting data through the Agent, use v6.40+/v7.40+.

UI activation

You, or a user in your organization with “Intelligent Test Runner Activation” permissions, must activate the Intelligent Test Runner on the Test Service Settings page.

Application key and permissions

The Intelligent Test Runner requires an application key mapped to an owner role with the CI Visibility Read (ci_visibility_read) permission.

For more information on roles and permissions, see the CI Visibility section of the Datadog Role Permissions documentation.

Configuring the test runner environment

Running tests with the Intelligent Test Runner enabled

Environment variable

DD_CIVISIBILITY_ITR_ENABLED (Optional)

Enable the Intelligent Test Runner coverage and test skipping features
Default: (false)

After completing setup, run your tests as you normally do:

DD_ENV=ci DD_SERVICE=my-python-app DD_CIVISIBILITY_ITR_ENABLED=true pytest --ddtrace

Disabling skipping for specific tests

You can override the Intelligent Test Runner’s behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.

Why make tests unskippable?

The Intelligent Test Runner uses code coverage data to determine whether or not tests should be skipped. In some cases, this data may not be sufficient to make this determination.

Examples include:

• Tests that read data from text files
• Tests that interact with APIs outside of the code being tested (such as remote REST APIs)

Designating tests as unskippable ensures that the Intelligent Test Runner runs them regardless of coverage data.

Compatibility

Unskippable tests are supported in the following versions and testing frameworks:

• pytest
  • From ddtrace>=1.19.0

Marking tests as unskippable in pytest

You can use pytest’s skipif mark to prevent the Intelligent Test Runner from skipping individual tests or modules. Specify the condition as False, and the reason as "datadog_itr_unskippable".

Individual tests

Individual tests can be marked as unskippable using the @pytest.mark.skipif decorator as follows:

import pytest

@pytest.mark.skipif(False, reason="datadog_itr_unskippable")
def test_function():
    assert True

Modules

Modules can be skipped using the pytestmark global as follows:

import pytest

pytestmark = pytest.mark.skipif(False, reason="datadog_itr_unskippable")

def test_function():
    assert True

Note: This does not override any other skip marks, or skipif marks that have a condition evaluating to True.

Further reading

Additional helpful documentation, links, and articles:

Explore Test Results and PerformanceDOCUMENTATION Troubleshooting CIDOCUMENTATION