Prior to setting up Test Impact Analysis, set up Test Optimization for Python. If you are reporting data through the Agent, use v6.40 and later or v7.40 and later.
Activate Intelligent Test Runner for the test service
You, or a user in your organization with the Intelligent Test Runner Activation (intelligent_test_runner_activation_write) permission, must activate the Intelligent Test Runner on the Test Service Settings page.
You can override Test Impact Analysis’s behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.
Why make tests unskippable?
Test Impact Analysis uses code coverage data to determine whether or not tests should be skipped. In some cases, this data may not be sufficient to make this determination.
Examples include:
Tests that read data from text files
Tests that interact with APIs outside of the code being tested (such as remote REST APIs)
Designating tests as unskippable ensures that Test Impact Analysis runs them regardless of coverage data.
Compatibility
Unskippable tests are supported in the following versions:
pytest
From ddtrace>=1.19.0.
Marking tests as unskippable
You can use pytest’s skipif mark to prevent Test Impact Analysis from skipping individual tests or modules. Specify the condition as False, and the reason as "datadog_itr_unskippable".
Individual tests
Individual tests can be marked as unskippable using the @pytest.mark.skipif decorator as follows:
Note: This does not override any other skip marks, or skipif marks that have a condition evaluating to True.
Compatibility
Unskippable tests are supported in the following versions:
unittest
From ddtrace>=2.2.0.
Marking tests as unskippable in unittest
You can use unittest’s skipif mark to prevent Test Impact Analysis from skipping individual tests. Specify the condition as False, and the reason as "datadog_itr_unskippable".
Individual tests
Individual tests can be marked as unskippable using the @unittest.skipif decorator as follows:
Using @unittest.skipif does not override any other skip marks, or skipIf marks that have a condition evaluating to True.
Known limitations
Code coverage collection
Interaction with coverage tools
Coverage data may appear incomplete when Test Impact Analysis is enabled. Lines of code that would normally be covered by tests are not be covered when these tests are skipped.
Interaction with the coverage package
Test Impact Analysis uses the coverage package’s API to collect code coverage. Data from coverage run or plugins like pytest-cov is incomplete as a result of ddtrace’s use of the Coverage class.
Some race conditions may cause exceptions when using pytest plugins such as pytest-xdist that change test execution order or introduce parallelization.
Further reading
Additional helpful documentation, links, and articles: