Getting Started with Continuous Testing

Overview

Continuous Testing lets you automatically run and monitor the same Synthetic tests you’ve configured in your staging, QA, and pre-production environments to proactively alert your team and block your pipeline deployments should code changes cause testing failures.

Your codeless tests can:

Once your tests run, examine test results and CI batches in the Synthetic Monitoring & Continuous Testing Explorer.

Improve your developer workflow with Continuous Testing:

Continuous Testing also offers parallelization, which allows you to execute multiple tests in your CI/CD pipelines simultaneously rather than sequentially to help speed up your building, testing, and deployment processes.

Prerequisites

If you haven’t already, create a Datadog account.

Create a Continuous Testing test

To set up a Continuous Testing test, first create a Synthetic test in Datadog. In this example, create a browser test on the site https://www.shopist.io, a test e-commerce web application.

Browser tests simulate a user’s journey through your web application beginning at your Starting URL. Ensuring your Starting URL is a resource in your staging environment makes it easier to test changes before moving them into production.

Configure your test details

  1. On the Datadog site, hover over UX Monitoring and click Continuous Testing.

  2. In the top right corner, click New Test > Browser Test.

  3. Define your browser test:

    • Add the URL of the website you want to monitor into the Starting URL field. For this example, enter https://www.shopist.io.
    • Select Advanced Options to set custom request options, certificates, authentication credentials, and more. In this example, no specific advanced option is needed.
    • Name your test and set a Team Tag such as team-checkout. Tags allow you to keep your test suite organized and find tests you’re interested in through the Synthetic Monitoring & Continuous Testing Explorer.
    • Choose the browsers and devices to test.
  4. Continue filling out your test details and your recording like you normally would.

new_browser_test

Run your Continuous Testing tests

To improve your development workflow, you can use datadog-ci in your CLI as a CI environment to configure your test. Then run your test directly in your IDE as a developer environment.

Running tests in the CLI

Extend your use of Continuous Testing by using the datadog-ci NPM package. datadog-ci lets you execute commands from within your CI/CD scripts to test your application before deployment. You can automate blocking and rolling back changes when tests fail. Read the datadog-ci Configuration page for installation and setup instructions.

You can use datadog-ci to execute only those tests tagged with specific Team tags. For example, to run all tests tagged as team-checkout:

  1. Navigate to your command line.
  2. Run the following:
    yarn datadog-ci synthetics run-tests -search 'tag:team-checkout' --config global.config.json
    

For more information about running the Synthetics command and using reporters, see Configuration Documentation

Running tests in your IDE

Separately, you can use the Datadog Synthetics VS Code Integration to help you:

  • Use a Private Location or Tunnel to accelerate development locally.
  • Run HTTP API tests and browser tests and see their results within VS Code.
  • Test only what matters by executing relevant tests at the same time.
vscode-extension-demo

Running tests in VS Code

  1. Open VS Code and install the Datadog Extension from the VS Code Extensions view.
  2. Open the Datadog Synthetics extension and sign in when prompted.
  3. Select a Synthetic test to run.
  4. Set a Start URL.
  5. Run the test.

Examine results in the Synthetic Monitoring and Continuous Testing Explorer

The Synthetic Monitoring and Continuous Testing Explorer allows you to create visualizations and filter CI batches and test runs for your Continuous Testing tests. Navigate to UX Monitoring > Continuous Testing.

Select CI Batches or Test Runs to see results from your CI batches or test runs in the Explorer. Select a CI batch or test from the list to get a more detailed view of the result.

ci_explorer_test_results

Create a search query

Click one of the following out-of-the-box search queries to filter your CI batches or test runs:

example-search-queries

Optionally, you can create a query to search your test runs. Using the browser test you created above, locate the Test ID and create a search query using the common test run facets. To find the ID of your browser test:

  1. Navigate to the Synthetic Tests page.
  2. Select a test.
  3. Look for the Test ID in the Properties section.
example_test_id

To export your view of the Synthetic Monitoring & Continuous Testing Explorer, click >Views and click Save. For more information, see Saved Views.

For more information about using facets in your search query, see Search Test Runs.

Set parallelization preferences

By default, Synthetic tests are not parallelized. Parallelization allows you to run multiple tests in your CI/CD pipelines simultaneously. If you want to parallelize your tests, you can use the Estimate Parallelization calculator to determine your needs.

Navigate to UX Monitoring > Settings and click Parallelization Settings to locate the calculator.

parallelization_estimate

For instance, if you have 24 tests per CI batch, each taking 2 minutes to complete, and your target is for all tests to be completed within 4 minutes, you need to run 12 tests in parallel.

$$\text"estimated parallelization" = {\text"24 tests per CI batch"* \text"2 minute duration"} / \text"4 minute expected duration in your CI pipeline"$$

Once you’re done estimating your parallelization, input the number of test runs you want to execute at the same time in the Parallelization modal. Then, click Save Selection.

See the Parallelization documentation for more details.

Further Reading