Continuous Testing and Bitrise

Docs > Continuous Testing > Continuous Testing and CI/CD > Continuous Testing and Bitrise

GitHub Release

Overview

With the synthetics-test-automation-bitrise-step-run-tests step, you can run Synthetic tests during your Bitrise CI, ensuring that all your teams using Bitrise can benefit from Synthetic tests at every stage of the software lifecycle. This step uses the Datadog CI Synthetics command.

Setup

This step is not available on the official Bitrise Step Library. To get started:

Add the following git URL to your workflow. See the official Bitrise documentation on how to do that though the Bitrise app. You can also configure it locally by referencing the git URL in your bitrise.yml file.

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:

Add your API and application keys to your secrets in Bitrise.
Configure your step inputs. You can also configure them in your bitrise.yml file. The only required inputs are the two secrets you configured earlier. For a comprehensive list of inputs, see the Inputs section.

How to use this step locally

You can run this step directly using the Bitrise CLI.

To run this step locally:

Open your terminal or command line.
git clone the Bitrise repository.
cd into the directory of the step (the one you just git cloned).
Create a .bitrise.secrets.yml file in the same directory of bitrise.yml. The .bitrise.secrets.yml file is a Git-ignored file, so you can store your secrets in it.
Check the bitrise.yml file for any secret you should set in .bitrise.secrets.yml.
Once you have the required secret parameters in your .bitrise.secrets.yml file, run this step with the Bitrise CLI: bitrise run test.

An example .bitrise.secrets.yml file:

envs:
- A_SECRET_PARAM_ONE: the value for secret one
- A_SECRET_PARAM_TWO: the value for secret two

Simple usage

Example using public IDs

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - public_ids: |
      abc-d3f-ghi
      jkl-mn0-pqr

Example task using existing synthetics.json files

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - files: 'e2e-tests/*.synthetics.json'

For an example test file, see this test.synthetics.json file.

Complex usage

Example task using the testSearchQuery

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - test_search_query: 'tag:e2e-tests'

Example task using the testSearchQuery and variable overrides

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - test_search_query: 'tag:e2e-tests'
   - variables: |
      START_URL=https://staging.website.com
      PASSWORD=$STAGING_PASSWORD

Example task using a global configuration override with configPath

This task overrides the path to the global datadog-ci.config.json file.

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - config_path: './synthetics-config.json'

For an example configuration file, see the global.config.json file.

Example including all possible configurations

For reference, this is an example of a complete configuration:

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - batch_timeout: 4200000
   - config_path: './global.config.json'
   - device_ids: 'apple iphone se (2022),15.4.1, apple iphone 14 pro,16.1'
   - fail_on_critical_errors: true
   - fail_on_missing_tests: true
   - fail_on_timeout: true
   - files: 'e2e-tests/*.synthetics.json'
   - junit_report: 'e2e-test-junit'
   - locations: 'aws:us-west-1'
   - mobile_application_version: '01234567-8888-9999-abcd-efffffffffff'
   - mobile_application_version_file_path: 'path/to/application.apk'
   - public_ids: 'abc-d3f-ghi,jkl-mn0-pqr'
   - site: 'datadoghq.com'
   - subdomain: 'myorg'
   - test_search_query: 'tag:e2e-tests'
   - tunnel: true
   - variables: |
      START_URL=https://staging.website.com
      PASSWORD=$STAGING_PASSWORD

Inputs

Name	Requirement	Description
`api_key`	required	Your Datadog API key. This key is created by your Datadog organization and will be accessed as an environment variable.
`app_key`	required	Your Datadog application key. This key is created by your Datadog organization and will be accessed as an environment variable.
`batch_timeout`	optional	The duration (in milliseconds) after which the batch fails as timed out. The default is 30 minutes.
`config_path`	optional	The global JSON configuration is used when launching tests. See the example configuration for more details.
`device_ids`	optional	Override the mobile device(s) to run your mobile test.
`locations`	optional	String of locations separated by semicolons to override the locations where your tests run.
`fail_on_critical_errors`	optional	A boolean flag that fails the CI job if no tests were triggered, or results could not be fetched from Datadog. The default is set to `false`.
`fail_on_missing_tests`	optional	Fail the CI job if at least one specified test with a public ID (using `publicIds` or listed in a test file) is missing in a run (for example, if it has been deleted programmatically or on the Datadog site).
`fail_on_timeout`	optional	A boolean flag that fails the CI job if at least one test exceeds the default test timeout. The default is set to `true`.
`files`	optional	Glob patterns to detect Synthetic test configuration files.
`junit_report`	optional	The filename for a JUnit report if you want to generate one.
`mobile_application_version`	optional	Override the default mobile application version for a Synthetic mobile application test. The version must be uploaded and available within Datadog. This version is also outputted by the `datadog-mobile-app-upload` step.
`mobile_application_version_file_path`	optional	Override the application version for Synthetic mobile application tests.
`public_ids`	optional	Public IDs of Synthetic tests to run, separated by newlines or commas. If no value is provided, tests are discovered in `*.synthetics.json` files.
`site`	optional	The Datadog site to send data to. If the `DD_SITE` environment variable is set, it takes precedence. Your Datadog site is `datadoghq.com`. .
`subdomain`	optional	The name of the custom subdomain set to access your Datadog application. If the URL used to access Datadog is `myorg.datadoghq.com`, the `subdomain` value needs to be set to `myorg`.
`test_search_query`	optional	Trigger tests corresponding to a search query. This can be useful if you are tagging your test configurations. See best practices for more information on tagging.
`tunnel`	optional	Enable Local and Staging Environments to interact with the Datadog API.
`variables`	optional	Key-value pairs for injecting variables into tests, separated by newlines or commas. For example: `START_URL=https://example.org,MY_VARIABLE=My title`.