Continuous Testing and Datadog CI Azure DevOps Extension

Visual Studio Marketplace Version Build Status License

Overview

With the SyntheticsRunTests task, you can run Synthetic tests within your Azure Pipeline configuration and ensure all your teams using Azure DevOps can benefit from Synthetic tests at every stage of the software lifecycle.

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

Authentication

Service Connection

To connect to your Datadog site, Datadog recommends setting up a custom service connection when configuring the SyntheticsRunTests task.

You need to provide the following inputs:

API and Application keys

Setup

To connect to your Datadog account, create a Datadog CI service connection in your Azure pipelines project. Once created, all you need is the name of the service connection in the tasks.

  1. Install the Datadog Continuous Testing extension from the Visual Studio Marketplace in your Azure Organization.
  2. Add your Datadog API and application keys in the Datadog CI service connection, or as secrets to your Azure Pipelines project.
  3. In your Azure DevOps pipeline, use the SyntheticsRunTests task.

Your task can be simple or complex.

Simple usage

Example task using public IDs

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    publicIds: |
      abc-d3f-ghi
      jkl-mn0-pqr

Example task using existing synthetics.json files

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    files: 'e2e-tests/*.synthetics.json'

For an example test file, see this test.synthetics.json file.

Example task using pipeline secrets for authentication

- task: SyntheticsRunTests@1
  inputs:
    authenticationType: 'apiAppKeys'
    apiKey: '$(DatadogApiKey)'
    appKey: '$(DatadogAppKey)'
    datadogSite: '$(DatadogSite)'
    subdomain: 'myorg'

Complex usage

Example task using the testSearchQuery

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'

Example task using the testSearchQuery and variable overrides

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'
    variables: |
      START_URL=https://staging.website.com
      PASSWORD=$(StagingPassword)

Example task using a global configuration file with configPath

By default, the path to the global configuration file is datadog-ci.json. You can override this path with the config_path input.

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    configPath: './global.config.json'
    connectedService: 'my-datadog-ci-connected-service'

Inputs

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

NameDescription
apiKeyYour Datadog API key. This key is created in your Datadog organization and should be stored as a secret.
Required when authenticationType == apiAppKeys
appKeyYour Datadog application key. This key is created in your Datadog organization and should be stored as a secret.
Required when authenticationType == apiAppKeys
authenticationType(Required) How to store and retrieve credentials.
Must be either apiAppKeys or connectedService
batchTimeoutSpecifies the timeout duration in milliseconds for the CI batch. When a batch times out, the CI job fails and no new test runs are triggered, but ongoing test runs complete normally.
Default: 1800000 (30 minutes)
connectedServiceThe name of the Datadog CI service connection.
Required when authenticationType == connectedService
configPathThe path to the global configuration file that configures datadog-ci.
Default: datadog-ci.json
datadogSiteYour Datadog site. The possible values are listed in this table.
Default: datadoghq.com

Set it to (ensure the correct SITE is selected on the right).
failOnCriticalErrorsFail the CI job if a critical error that is typically transient occurs, such as rate limits, authentication failures, or Datadog infrastructure issues.
Default: false
failOnMissingTestsFail the CI job if the list of tests to run is empty or if some explicitly listed tests are missing.
Default: false
failOnTimeoutFail the CI job if the CI batch fails as timed out.
Default: true
filesGlob patterns to detect Synthetic test configuration files, separated by new lines.
Default: {,!(node_modules)/**/}*.synthetics.json
jUnitReportThe filename for a JUnit report if you want to generate one.
Default: none
locationsOverride the list of locations to run the test from, separated by new lines or commas. The possible values are listed in this API response.
Default: none
publicIdsPublic IDs of Synthetic tests to run, separated by new lines or commas. If no value is provided, tests are discovered in Synthetic test configuration files.
Default: none
selectiveRerunWhether to only rerun failed tests. If a test has already passed for a given commit, it is not rerun in subsequent CI batches. By default, your organization’s default setting is used. Set it to false to force full runs when your configuration enables it by default.
Default: none
subdomainThe custom subdomain to access your Datadog organization when authenticationType == apiAppKeys. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.
Default: app
testSearchQueryUse a search query to select which Synthetic tests to run. Use the Synthetic Tests list page’s search bar to craft your query, then copy and paste it.
Default: none
variablesOverride existing or inject new local and global variables in Synthetic tests as key-value pairs, separated by new lines or commas. For example: START_URL=https://example.org,MY_VARIABLE=My title.
Default: none

Outputs

NameDescription
batchUrlThe URL of the batch.
criticalErrorsCountThe number of critical errors.
failedCountThe number of failed results.
failedNonBlockingCountThe number of failed non-blocking results.
passedCountThe number of passed results.
previouslyPassedCountThe number of previously passed results.
testsNotFoundCountThe number of not found tests.
testsSkippedCountThe number of skipped tests.
timedOutCountThe number of timed out results.
rawResultsThe list of results, as a raw JSON string.

Further reading

Additional helpful documentation, links, and articles: