Overview

Test Visibility provides a test-first view into your CI health by displaying important metrics and results from your tests. It can help you investigate performance problems and test failures that concern you the most because you work on the related code, not because you maintain the pipelines they are run in.

Setup

In addition to tests, Test Visibility provides visibility over the whole testing phase of your project.

Supported features

* the feature is opt-in and needs to be enabled in the test service settings UI.

Default configurations

Tests evaluate the behavior of code for a set of given conditions. Some of those conditions are related to the environment where the tests are run, such as the operating system or the runtime used. The same code executed under different sets of conditions can behave differently, so developers usually configure their tests to run in different sets of conditions and validate that the behavior is the expected for all of them. This specific set of conditions is called a configuration.

In Test Visibility, a test with multiple configurations is treated as multiple tests with a separate test for each configuration. In the case where one of the configurations fails but the others pass, only that specific test and configuration combination is marked as failed.

For example, suppose you’re testing a single commit and you have a Python test that runs against three different Python versions. If the test fails for one of those versions, that specific test is marked as failed, while the other versions are marked as passed. If you retry the tests against the same commit and now the test for all three Python versions pass, the test with the version that previously failed is now marked as both passed and flaky, while the other two versions remain passed, with no flakiness detected.

Test configuration attributes

When you run your tests with Test Visibility, the library detects and reports information about the environment where tests are run as test tags. For example, the operating system name, such as Windows or Linux, and the architecture of the platform, such as arm64 or x86_64, are added as tags on each test. These values are shown in the commit and on branch overview pages when a test fails or is flaky for a specific configuration but not others.

The following tags are automatically collected to identify test configurations, and some may only apply to specific platforms:

Parameterized test configurations

When you run parameterized tests, the library detects and reports information about the parameters used. Parameters are a part of test configuration, so the same test case executed with different parameters will be considered as two different tests in Test Visibility.

If a test parameter is non-deterministic and has a different value every time a test is run, each test execution will be considered a new test in Test Visibility. As a consequence, some product features may not work correctly for such tests: history of executions, flakiness detection, Intelligent Test Runner, and others.

Some examples of non-deterministic test parameters are:

• current date
• a random value
• a value that depends on the test execution environment (such as an absolute file path or the current username)
• a value that has no deterministic string representation (for example an instance of a Java class whose toString() method is not overridden)

Avoid using non-deterministic test parameters. In case this is not possible, some testing frameworks provide a way to specify a deterministic string representation for a non-deterministic parameter (such as overriding parameter display name).

Custom configurations

There are some configurations that cannot be directly identified and reported automatically because they can depend on environment variables, test run arguments, or other approaches that developers use. For those cases, you must provide the configuration details to the library so Test Visibility can properly identify them.

Define these tags as part of the DD_TAGS environment variable using the test.configuration prefix.

For example, the following test configuration tags identify a test configuration where disk response time is slow and available memory is low:

DD_TAGS=test.configuration.disk:slow,test.configuration.memory:low

All tags with the test.configuration prefix are used as configuration tags, in addition to the automatically collected ones.

Note: Nested test.configuration tags, such as test.configuration.cpu.memory, are not supported.

In order to filter using these configurations tags, you must create facets for these tags.

Use CI tests data

When CI Visibility is enabled, the following data is collected from your project:

• Test names and durations.
• Predefined environment variables set by CI providers.
• Git commit history including the hash, message, author information, and files changed (without file contents).
• Information from the CODEOWNERS file.

Integrations

Alert on test data

When you evaluate failed or flaky tests, or the performance of a CI test on the Test Runs page, click Create Monitor to create a CI Test monitor.

Further reading

Additional helpful documentation, links, and articles:

Check out the latest Software Delivery releases! (App login required)RELEASE NOTES Creating CI Test MonitorsDOCUMENTATION Finding Flaky TestsDOCUMENTATION Linking CI Visibility and RUMDOCUMENTATION Troubleshooting CI VisibilityDOCUMENTATION Troubleshoot end-to-end tests with CI Visibility and RUMBLOG