Continuous Testing and CI/CD Troubleshooting
Overview
This page provides information to help you troubleshoot issues with Continuous Testing and CI/CD. If you need additional help, contact Datadog support.
Terminology
- CI batch
- The group of Continuous Testing tests triggered through a continuous integration or continuous delivery (CI/CD) pipeline or the Datadog Synthetic Monitoring API.
- Test run
- A single run of a Continuous Testing test, which can be an API or browser test. If you have configured retries, they count as individual test runs. For example, a test with two retries can have up to three associated test runs.
- Parallel test
- A Continuous Testing test that is run at the same time as another Continuous Testing test in your CI/CD pipeline. To set how many tests you would like to run in parallel, configure parallelization on the Continuous Testing Settings page.
- Batch timeout
- A batch timeout occurs when your batch does not complete within a reasonable time based on the polling timeout set in your configuration file.
- Execution rule
- An execution rule defines the impact of a test failure on a CI/CD pipeline from most to least impactful:
skipped, non_blocking, and blocking. These options are weighted and default to the most impactful. If your test is configured as skipped in the UI and blocking in the configuration file, it is skipped during the test run.
You can set the execution rule in your tests’ properties, global configuration file, or an individual test’s override file.
Results Explorer
Check whether you are using API endpoints to trigger your CI/CD test runs. In order to populate the Synthetic Monitoring & Continuous Testing Results Explorer with CI metadata, you must use one of Datadog’s native integrations, or the NPM package.
Within your CI/CD pipeline
My tests are timing out in my CI pipeline
The first thing to check is which failure mode flags you are passing in your global configuration file. For CI runs that contain multiple tests, some tests are queued based on the parallelization setting defined on the Continuous Testing Settings page. You may need to adapt both your configuration and parallelization based on your organizational needs.
Synthetics monitors
The CI does not trigger Synthetics monitors or incorporate them into monitor evaluations; however, failing runs will result in the CI showing a red status.
Further reading
Additional helpful documentation, links, and articles:
