GitHub Release Build Status License

Overview

With the synthetics-test-automation-bitrise-step-run-tests step, you can run Synthetic tests during your Bitrise CI, ensuring that all your teams using Bitrise can benefit from Synthetic tests at every stage of the software lifecycle. This step uses the Datadog CI Synthetics command.

Setup

This step is not available on the official Bitrise Step Library. To get started:

  1. Add the following git URL to your workflow. See the official Bitrise documentation on how to do that though the Bitrise app. You can also configure it locally by referencing the git URL in your bitrise.yml file.
- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@1.0.1:
  1. Add your API and application keys to your secrets in Bitrise.
  2. Configure your step inputs. You can also configure them in your bitrise.yml file. The only required inputs are the two secrets you configured earlier. For a comprehensive list of inputs, see the Inputs section.

How to use this step locally

You can run this step directly using the Bitrise CLI.

To run this step locally:

  1. Open your terminal or command line.
  2. git clone the Bitrise repository.
  3. cd into the directory of the step (the one you just git cloned).
  4. Create a .bitrise.secrets.yml file in the same directory of bitrise.yml. The .bitrise.secrets.yml file is a Git-ignored file, so you can store your secrets in it.
  5. Check the bitrise.yml file for any secret you should set in .bitrise.secrets.yml.
  6. Once you have the required secret parameters in your .bitrise.secrets.yml file, run this step with the Bitrise CLI: bitrise run test.

An example .bitrise.secrets.yml file:

envs:
- A_SECRET_PARAM_ONE: the value for secret one
- A_SECRET_PARAM_TWO: the value for secret two

Simple usage

Example using public IDs

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@1.0.1:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - public_ids: 'abc-d3f-ghi, jkl-mn0-pqr'

Example task using existing synthetics.json files

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@1.0.1:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - files: 'e2e-tests/*.synthetics.json'

For an example test file, see this test.synthetics.json file.

Complex usage

Example task using the testSearchQuery

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@1.0.1:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - test_search_query: 'tag:e2e-tests'

Example task using the testSearchQuery and variable overrides

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@1.0.1:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - test_search_query: 'tag:e2e-tests'
   - variables: |
      START_URL=https://staging.website.com
      PASSWORD=$STAGING_PASSWORD      

Example task using a global configuration override with configPath

This task overrides the path to the global datadog-ci.config.json file.

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@1.0.1:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - config_path: './synthetics-config.json'

For an example configuration file, see the global.config.json file.

Example including all possible configurations

For reference, this is an example of a complete configuration:

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@1.0.1:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - config_path: './global.config.json'
   - device_ids: 'apple iphone se (2022),15.4.1, apple iphone 14 pro,16.1'
   - fail_on_critical_errors: true
   - fail_on_missing_tests: true
   - fail_on_timeout: true
   - files: 'e2e-tests/*.synthetics.json'
   - junit_report: 'e2e-test-junit'
   - locations: 'aws:us-west-1'
   - mobile_application_version: '01234567-8888-9999-abcd-efffffffffff'
   - mobile_application_version_file_path: 'path/to/application.apk'
   - polling_timeout: 4200000
   - public_ids: 'abc-d3f-ghi, jkl-mn0-pqr'
   - site: 'datadoghq.com'
   - subdomain: 'myorg'
   - test_search_query: 'tag:e2e-tests'
   - tunnel: true
   - variables: |
      START_URL=https://staging.website.com
      PASSWORD=$STAGING_PASSWORD      

Inputs

NameRequirementDescription
apiKeyrequiredYour Datadog API key. This key is created by your Datadog organization and will be accessed as an environment variable.
appKeyrequiredYour Datadog application key. This key is created by your Datadog organization and will be accessed as an environment variable.
configPathoptionalThe global JSON configuration is used when launching tests. See the example configuration for more details.
deviceIdsoptionalOverride the mobile device(s) to run your mobile test.
locationsoptionalString of locations separated by semicolons to override the locations where your tests run.
failOnCriticalErrorsoptionalA boolean flag that fails the CI job if no tests were triggered, or results could not be fetched from Datadog. The default is set to false.
failOnMissingTestsoptionalFail the CI job if at least one specified test with a public ID (using publicIds or listed in a test file) is missing in a run (for example, if it has been deleted programmatically or on the Datadog site).
failOnTimeoutoptionalA boolean flag that fails the CI job if at least one test exceeds the default test timeout. The default is set to true.
filesoptionalGlob patterns to detect Synthetic test configuration files.
jUnitReportoptionalThe filename for a JUnit report if you want to generate one.
mobileApplicationVersionoptionalOverride the default mobile application version for a Synthetic mobile application test. The version must be uploaded and available within Datadog. This version is also outputted by the datadog-mobile-app-upload step.
mobileApplicationVersionFilePathoptionalOverride the application version for Synthetic mobile application tests.
pollingTimeoutoptionalThe duration (in milliseconds) after which the batch is deemed as failed because of a timeout. The default is 30 minutes.
publicIdsoptionalString of public IDs separated by commas for Synthetic tests you want to trigger.
siteoptionalThe Datadog site to send data to. If the DD_SITE environment variable is set, it takes precedence. Your Datadog site is . .
subdomainoptionalThe name of the custom subdomain set to access your Datadog application. If the URL used to access Datadog is myorg.datadoghq.com, the subdomain value needs to be set to myorg.
testSearchQueryoptionalTrigger tests corresponding to a search query. This can be useful if you are tagging your test configurations. See best practices for more information on tagging.
tunneloptionalEnable Local and Staging Environments to interact with the Datadog API.
variablesoptionalKey-value pairs for injecting variables into tests. Must be formatted using KEY=VALUE.

Further reading

Additional helpful documentation, links, and articles: