Continuous Testing and CircleCI Orb
Overview
Run Datadog Synthetic tests in your CircleCI pipelines using the Datadog CircleCI orb.
For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.
Setup
To get started:
- Add your Datadog API and application keys as environment variables to your CircleCI project.
- Ensure the image running the orb is a Linux-x64-based image with
curl installed. - Customize your CircleCI workflow by adding a
synthetics-ci/run-tests step and specifying inputs as listed below.
Your workflow can be simple or complex.
Simple usage
Example orb usage using public IDs
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@5.1.0
jobs:
e2e-tests:
docker:
- image: cimg/base:stable
steps:
- synthetics-ci/run-tests:
public_ids: |
abc-d3f-ghi
jkl-mn0-pqr
workflows:
run-tests:
jobs:
- e2e-tests
Example orb usage using a global configuration override
This orb overrides the path to the pattern for test files.
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@5.1.0
jobs:
e2e-tests:
docker:
- image: cimg/base:stable
steps:
- synthetics-ci/run-tests:
files: e2e-tests/*.synthetics.json
workflows:
run-tests:
jobs:
- e2e-tests
For another example pipeline that triggers Synthetic tests, see the simple-example.yml file.
Complex usage
Example orb usage using the test_search_query
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@5.1.0
jobs:
e2e-tests:
docker:
- image: cimg/base:stable
steps:
- synthetics-ci/run-tests:
test_search_query: 'tag:e2e-tests'
workflows:
run-tests:
jobs:
- e2e-tests
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@5.1.0
jobs:
e2e-tests:
docker:
- image: your-image
steps:
- checkout
- run:
name: Running server in background
command: npm start
background: true
- synthetics-ci/run-tests:
config_path: tests/tunnel-config.json
files: tests/*.synthetics.json
test_search_query: 'tag:e2e-tests'
tunnel: true
workflows:
test-server:
jobs:
- build-image
- integration-tests:
requires:
- build-image
For additional options such as customizing the batchTimeout for your CircleCI pipelines, see CI/CD Integrations Configuration. For another example pipeline that starts a local server and triggers Synthetic tests using the Continuous Testing tunnel, see the advanced-example.yml file.
For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.
| Name | Description |
|---|
api_key | Name of the environment variable containing your Datadog API key. This key is created in your Datadog organization and should be stored as a secret.
Default: DATADOG_API_KEY |
app_key | Name of the environment variable containing your Datadog application key. This key is created in your Datadog organization and should be stored as a secret.
Default: DATADOG_APP_KEY |
background | Whether or not this step should run in the background. See official CircleCI documentation.
Default: false |
batch_timeout | Specifies the timeout duration in milliseconds for the CI batch. When a batch times out, the CI job fails and no new test runs are triggered, but ongoing test runs complete normally.
Default: 1800000 (30 minutes) |
config_path | The path to the global configuration file that configures datadog-ci.
Default: datadog-ci.json |
datadog_site | Your Datadog site. The possible values are listed in this table.
Default: datadoghq.com
Set it to |
fail_on_critical_errors | Fail the CI job if a critical error that is typically transient occurs, such as rate limits, authentication failures, or Datadog infrastructure issues.
Default: false |
fail_on_missing_tests | Fail the CI job if the list of tests to run is empty or if some explicitly listed tests are missing.
Default: false |
fail_on_timeout | Fail the CI job if the CI batch fails as timed out.
Default: true |
files | Glob patterns to detect Synthetic test configuration files, separated by new lines.
Default: {,!(node_modules)/**/}*.synthetics.json |
junit_report | The filename for a JUnit report if you want to generate one.
Default: none |
locations | Override the list of locations to run the test from, separated by new lines or commas. The possible values are listed in this API response.
Default: none |
no_output_timeout | Elapsed time the command can run without output. The string is a decimal with unit suffix, such as 20m, 1.25h, 5s. See official CircleCI documentation.
Default: 35m |
public_ids | Public IDs of Synthetic tests to run, separated by new lines or commas. If no value is provided, tests are discovered in Synthetic test configuration files.
Default: none |
selective_rerun | Whether to only rerun failed tests. If a test has already passed for a given commit, it is not rerun in subsequent CI batches. By default, your organization’s default setting is used. Set it to false to force full runs when your configuration enables it by default.
Default: none |
subdomain | The custom subdomain to access your Datadog organization. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.
Default: app |
test_search_query | Use a search query to select which Synthetic tests to run. Use the Synthetic Tests list page’s search bar to craft your query, then copy and paste it.
Default: none |
tunnel | Use the Continuous Testing tunnel to launch tests against internal environments.
Default: false |
variables | Override existing or inject new local and global variables in Synthetic tests as key-value pairs, separated by new lines or commas. For example: START_URL=https://example.org,MY_VARIABLE=My title.
Default: none |
Further reading
Additional helpful documentation, links, and articles:
