Test Impact Analysis Troubleshooting

Docs > Test Optimization in Datadog > Test Impact Analysis > Test Impact Analysis Troubleshooting

Overview

This page provides information to help you troubleshot issues with Test Impact Analysis. If you need additional help, contact Datadog Support.

Test Impact Analysis is not working

Test Impact Analysis works by analyzing your commit history along with code coverage information about past test runs to determine which tests need to be run and which ones can be safely skipped. A minimum amount of information needs to exist in order for Test Impact Analysis to work correctly:

git executable needs to be available in the host running tests.
Your repository needs to have a commit history of at least two commits in the past month.
You need to have collected test code coverage in past commits, which happens on test runs where Test Impact Analysis was enabled.
Your git clone must contain commit and tree history. Test Impact Analysis tries to unshallow git clones that do not contain history (git clone --depth=1), but that might not work on older versions of git. Automatic unshallowing might require additional set up in some CI providers (Harness CI, for example, requires extra configuration to make sure your pipeline can execute git commands). If your CI job is using shallow git clones, you can change it to use partial git clones by using the following command: git clone --filter=blob:none.

Due to these restrictions, the first time you enable Test Impact Analysis, you cannot see any tests skipped and the test execution time may be slower than usual because the code coverage is collected automatically.

Test Impact Analysis only takes into account the commit history and test code coverage information for the past month. Additionally, it does not take into account code coverage information that is generated more than one week after a commit was made.

Synchronizing a fork through GitHub’s UI

Synchronizing a fork through GitHub’s UI causes all tests to be run for the generated synchronization commit.

Collecting coverages on GitHub Actions CI triggered by pull request events

Tests run in GitHub Actions CI using the pull_request trigger may not be skipped on subsequent commits within the pull request’s branch. The trigger introduces changes using a new merge commit that is not considered by Test Impact Analysis.

Squash and merge your commits

Using squash and merge to integrate commits into the base branch causes the git history of the feature branch to be lost. As a result, Test Impact Analysis may not skip tests in the base branch that were recently skipped in the feature branch.

Test Impact Analysis incorrectly skipped a test

Test Impact Analysis performs test impact analysis based on code coverage to determine which tests are impacted by a given commit or set of commits. While this strategy works for the majority of tests, there are known scenarios where Test Impact Analysis could skip a test that should have been run:

Changes in library dependencies.
Changes in compiler options.
Changes in external services.
Changes to data files in data-driven tests.

If you are authoring a commit that includes any of those cases, you can force-disable test skipping in Test Impact Analysis by adding ITR:NoSkip (case insensitive) anywhere in your Git commit message.