このページは日本語には対応しておりません。随時翻訳に取り組んでいます。翻訳に関してご質問やご意見ございましたら、お気軽にご連絡ください。

Compatibility

Test Impact Analysis is only supported in the following versions and testing frameworks:

  • pytest>=7.2.0
    • From ddtrace>=2.1.0.
    • From Python>=3.7.
    • Requires coverage>=5.5.
    • Incompatible with pytest-cov (see known limitations)
  • unittest
    • From ddtrace>=2.2.0.
    • From Python>=3.7.
  • coverage

Setup

Test Optimization

Prior to setting up Test Impact Analysis, set up Test Optimization for Python. If you are reporting data through the Agent, use v6.40 and later or v7.40 and later.

テストサービスの Intelligent Test Runner を有効にする

あなた、またはあなたの組織で Intelligent Test Runner Activation (intelligent_test_runner_activation_write) 権限を持つユーザーが、テストサービス設定ページで Intelligent Test Runner を有効にする必要があります。

Intelligent test runner enabled in test service settings in the CI section of Datadog.

Required dependencies

Test Impact Analysis requires the coverage package.

Install the package in your CI test environment by specifying it in the relevant requirements file, for example, or using pip:

pip install coverage

See known limitations if you are already using the coverage package or a plugin like pytest-cov.

Running tests with Test Impact Analysis enabled

Test Impact Analysis is enabled when you run tests with the Datadog integration active. Run your tests with the following command:

DD_ENV=ci DD_SERVICE=my-python-app pytest --ddtrace
DD_ENV=ci DD_SERVICE=my-python-app ddtrace-run python -m unittest

Temporarily disabling Test Impact Analysis

Test Impact Analysis can be disabled locally by setting the DD_CIVISIBILITY_ITR_ENABLED environment variable to false or 0.

DD_CIVISIBILITY_ITR_ENABLED (Optional)
Enable Test Impact Analysis coverage and test skipping features
Default: (true)

Run the following command to disable Test Impact Analysis:

DD_ENV=ci DD_SERVICE=my-python-app DD_CIVISIBILITY_ITR_ENABLED=false pytest --ddtrace
DD_ENV=ci DD_SERVICE=my-python-app DD_CIVISIBILITY_ITR_ENABLED=false ddtrace-run python -m unittest

Disabling skipping for specific tests

You can override Test Impact Analysis’s behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.

Why make tests unskippable?

Test Impact Analysis uses code coverage data to determine whether or not tests should be skipped. In some cases, this data may not be sufficient to make this determination.

Examples include:

  • Tests that read data from text files
  • Tests that interact with APIs outside of the code being tested (such as remote REST APIs)

Designating tests as unskippable ensures that Test Impact Analysis runs them regardless of coverage data.

Compatibility

Unskippable tests are supported in the following versions:

  • pytest
    • From ddtrace>=1.19.0.

Marking tests as unskippable

You can use pytest’s skipif mark to prevent Test Impact Analysis from skipping individual tests or modules. Specify the condition as False, and the reason as "datadog_itr_unskippable".

Individual tests

Individual tests can be marked as unskippable using the @pytest.mark.skipif decorator as follows:

import pytest

@pytest.mark.skipif(False, reason="datadog_itr_unskippable")
def test_function():
    assert True

Modules

Modules can be skipped using the pytestmark global variable as follows:

import pytest

pytestmark = pytest.mark.skipif(False, reason="datadog_itr_unskippable")

def test_function():
    assert True

Note: This does not override any other skip marks, or skipif marks that have a condition evaluating to True.

Compatibility

Unskippable tests are supported in the following versions:

  • unittest
    • From ddtrace>=2.2.0.

Marking tests as unskippable in unittest

You can use unittest’s skipif mark to prevent Test Impact Analysis from skipping individual tests. Specify the condition as False, and the reason as "datadog_itr_unskippable".

Individual tests

Individual tests can be marked as unskippable using the @unittest.skipif decorator as follows:

import unittest

class MyTestCase(unittest.TestCase):
  @unittest.skipIf(False, reason="datadog_itr_unskippable")
  def test_function(self):
      assert True

Using @unittest.skipif does not override any other skip marks, or skipIf marks that have a condition evaluating to True.

Known limitations

Code coverage collection

Interaction with coverage tools

Coverage data may appear incomplete when Test Impact Analysis is enabled. Lines of code that would normally be covered by tests are not be covered when these tests are skipped.

Interaction with the coverage package

Test Impact Analysis uses the coverage package’s API to collect code coverage. Data from coverage run or plugins like pytest-cov is incomplete as a result of ddtrace’s use of the Coverage class.

Some race conditions may cause exceptions when using pytest plugins such as pytest-xdist that change test execution order or introduce parallelization.

Further reading