Continuous Testing and CI/CD Troubleshooting


This page provides information to help you troubleshoot issues with Continuous Testing and CI/CD. If you need additional help, contact Datadog support.


CI batch
The group of Continuous Testing tests triggered through a continuous integration or continuous delivery (CI/CD) pipeline or the Datadog Synthetic Monitoring API.
Test run
A single run of a Continuous Testing test, which can be an API or browser test. If you have configured retries, they count as individual test runs. For example, a test with two retries can have up to three associated test runs.
Parallel test
A Continuous Testing test that is run at the same time as another Continuous Testing test in your CI/CD pipeline. To set how many tests you would like to run in parallel, configure parallelization on the Continuous Testing Settings page.
Batch timeout
A batch timeout occurs when your batch does not complete within a reasonable time based on the polling timeout set in your configuration file.
Execution rule
An execution rule defines the impact of a test failure on a CI/CD pipeline from most to least impactful: skipped, non_blocking, and blocking. These options are weighted and default to the most impactful. If your test is configured as skipped in the UI and blocking in the configuration file, it is skipped during the test run.

You can set the execution rule in your tests’ properties, global configuration file, or an individual test’s override file.


CI metadata does not appear

Check whether you are using API endpoints to trigger your CI/CD test runs. In order to populate the CI Results Explorer with CI metadata, you must use one of Datadog’s native integrations, or the NPM package.

Within your CI/CD pipeline

My tests are timing out in my CI pipeline

The first thing to check is which failure mode flags you are passing in your global configuration file. For CI runs that contain multiple tests, some tests are queued based on the parallelization setting defined on the Continuous Testing Settings page. You may need to adapt both your configuration and parallelization based on your organizational needs.

Further reading