If you are using auto-instrumentation for one of these providers, you can skip the rest of the setup steps below.
If you are using a cloud CI provider without access to the underlying worker nodes, such as GitHub Actions or CircleCI, configure the library to use the Agentless mode. For this, set the following environment variables:
DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)
Enables or disables Agentless mode. Default: false
DD_API_KEY (Required)
The Datadog API key used to upload the test results. Default: (empty)
Additionally, configure the Datadog site to which you want to send data.
DD_SITE (Required)
The Datadog site to upload results to. Default: datadoghq.com
If you are running tests on an on-premises CI provider, such as Jenkins or self-managed GitLab CI, install the Datadog Agent on each worker node by following the Agent installation instructions.
This is the recommended option as it allows you to automatically link test results to logs and underlying host metrics.
If you are using a Kubernetes executor, Datadog recommends using the Datadog Operator.
The operator includes Datadog Admission Controller which can automatically inject the tracer library into the build pods.
Note: If you use the Datadog Operator, there is no need to download and inject the tracer library since the Admission Controller can do this for you, so you can skip the corresponding step below.
However, you still need to make sure that your pods set the environment variables or command-line parameters necessary to enable Test Visibility.
If you are not using Kubernetes or can’t use the Datadog Admission Controller and the CI provider is using a container-based executor, set the DD_TRACE_AGENT_URL environment variable (which defaults to http://localhost:8126) in the build container running the tracer to an endpoint that is accessible from within that container. Note: Using localhost inside the build references the container itself and not the underlying worker node or any container where the Agent might be running in.
DD_TRACE_AGENT_URL includes the protocol and port (for example, http://localhost:8126) and takes precedence over DD_AGENT_HOST and DD_TRACE_AGENT_PORT, and is the recommended configuration parameter to configure the Datadog Agent’s URL for CI Visibility.
If you still have issues connecting to the Datadog Agent, use the Agentless Mode.
Note: When using this method, tests are not correlated with logs and infrastructure metrics.
To enable instrumentation of pytest tests, add the --ddtrace option when running pytest, specifying the name of the service or library under test in the DD_SERVICE environment variable, and the environment where tests are being run (for example, local when running tests on a developer workstation, or ci when running them on a CI provider) in the DD_ENV environment variable:
To add custom tags to your tests, declare ddspan as an argument in your test:
fromddtraceimporttracer# Declare `ddspan` as argument to your testdeftest_simple_case(ddspan):# Set your tagsddspan.set_tag("test_owner","my_team")# test continues normally# ...
To create filters or group by fields for these tags, you must first create facets. For more information about adding tags, see the Adding Tags section of the Python custom instrumentation documentation.
Adding custom measures to tests
Just like tags, to add custom measures to your tests, use the current active span:
fromddtraceimporttracer# Declare `ddspan` as an argument to your testdeftest_simple_case(ddspan):# Set your tagsddspan.set_tag("memory_allocations",16)# test continues normally# ...
To instrument your benchmark tests with pytest-benchmark, run your benchmark tests with the --ddtrace option when running pytest, and Datadog detects metrics from pytest-benchmark automatically:
To enable instrumentation of unittest tests, run your tests by appending ddtrace-run to the beginning of your unittest command.
Make sure to specify the name of the service or library under test in the DD_SERVICE environment variable.
Additionally, you may declare the environment where tests are being run in the DD_ENV environment variable:
Note: The Test Optimization manual testing API is in beta and subject to change.
As of version 2.13.0, the Datadog Python tracer provides the Test Optimization API (ddtrace.ext.test_visibility) to submit test optimization results as needed.
API execution
The API uses classes to provide namespaced methods to submit test optimization events.
Test execution has two phases:
Discovery: inform the API what items to expect
Execution: submit results (using start and finish calls)
The distinct discovery and execution phases allow for a gap between the test runner process collecting the tests and the tests starting.
API users must provide consistent identifiers (described below) that are used as references for Test Optimization items within the API’s state storage.
Enable test_visibility
You must call the ddtrace.ext.test_visibility.api.enable_test_visibility() function before using the Test Optimization API.
Call the ddtrace.ext.test_visibility.api.disable_test_visibility() function before process shutdown to ensure proper flushing of data.
Domain model
The API is based around four concepts: test session, test module, test suite, and test.
Modules, suites, and tests form a hierarchy in the Python Test Optimization API, represented by the item identifier’s parent relationship.
Test session
A test session represents a project’s test execution, typically corresponding to the execution of a test command. Only one session can be discovered, started, and finished in the execution of Test Optimization program.
Call ddtrace.ext.test_visibility.api.TestSession.discover() to discover the session, passing the test command, a given framework name, and version.
Call ddtrace.ext.test_visibility.api.TestSession.start() to start the session.
When tests have completed, call ddtrace.ext.test_visibility.api.TestSession.finish() .
Test module
A test module represents a smaller unit of work within a project’s tests run (a directory, for example).
Call ddtrace.ext.test_visibility.api.TestModuleId(), providing the module name as a parameter, to create a TestModuleId.
Call ddtrace.ext.test_visibility.api.TestModule.discover(), passing the TestModuleId object as an argument, to discover the module.
Call ddtrace.ext.test_visibility.api.TestModule.start(), passing the TestModuleId object as an argument, to start the module.
After all the children items within the module have completed, call ddtrace.ext.test_visibility.api.TestModule.finish(), passing the TestModuleId object as an argument.
Test suite
A test suite represents a subset of tests within a project’s modules (.py file, for example).
Call ddtrace.ext.test_visibility.api.TestSuiteId(), providing the parent module’s TestModuleId and the suite’s name as arguments, to create a TestSuiteId.
Call ddtrace.ext.test_visibility.api.TestSuite.discover(), passing the TestSuiteId object as an argument, to discover the suite.
Call ddtrace.ext.test_visibility.api.TestSuite.start(), passing the TestSuiteId object as an argument, to start the suite.
After all the child items within the suite have completed, call ddtrace.ext.test_visibility.api.TestSuite.finish(), passing the TestSuiteId object as an argument.
Test
A test represents a single test case that is executed as part of a test suite.
Call ddtrace.ext.test_visibility.api.TestId(), providing the parent suite’s TestSuiteId and the test’s name as arguments, to create a TestId. The TestId() method accepts a JSON-parseable string as the optional parameters argument. The parameters argument can be used to distinguish parametrized tests that have the same name, but different parameter values.
Call ddtrace.ext.test_visibility.api.Test.discover(), passing the TestId object as an argument, to discover the test. The Test.discover() classmethod accepts a string as the optional resource parameter, which defaults to the TestId’s name.
Call ddtrace.ext.test_visibility.api.Test.start(), passing the TestId object as an argument, to start the test.
Call ddtrace.ext.test_visibility.api.Test.mark_pass(), passing the TestId object as an argument, to mark that the test has passed successfully.
Call ddtrace.ext.test_visibility.api.Test.mark_fail(), passing the TestId object as an argument, to mark that the test has failed. mark_fail() accepts an optional TestExcInfo object as the exc_info parameter.
Call ddtrace.ext.test_visibility.api.Test.mark_skip(), passing the TestId object as an argument, to mark that the test was skipped. mark_skip() accepts an optional string as the skip_reason parameter.
Exception information
The ddtrace.ext.test_visibility.api.Test.mark_fail() classmethod holds information about exceptions encountered during a test’s failure.
The ddtrace.ext.test_visibility.api.TestExcInfo() method takes three positional parameters:
exc_type: the type of the exception encountered
exc_value: the BaseException object for the exception
exc_traceback: the Traceback object for the exception
Codeowner information
The ddtrace.ext.test_visibility.api.Test.discover() classmethod accepts an optional list of strings as the codeowners parameter.
Test source file information
The ddtrace.ext.test_visibility.api.Test.discover() classmethod accepts an optional TestSourceFileInfo object as the source_file_info parameter. A TestSourceFileInfo object represents the path and optionally, the start and end lines for a given test.
The ddtrace.ext.test_visibility.api.TestSourceFileInfo() method accepts three positional parameters:
path: a pathlib.Path object (made relative to the repo root by the Test Optimization API)
start_line: an optional integer representing the start line of the test in the file
end_line: an optional integer representing the end line of the test in the file
Setting parameters after test discovery
The ddtrace.ext.test_visibility.api.Test.set_parameters() classmethod accepts a TestId object as an argument, and a JSON-parseable string, to set the parameters for the test.
Note: this overwrites the parameters associated with the test, but does not modify the TestId object’s parameters field.
Setting parameters after a test has been discovered requires that the TestId object be unique even without the parameters field being set.
Code example
fromddtrace.ext.test_visibilityimportapiimportpathlibimportsysif__name__=="__main__":# Enable the Test Optimization serviceapi.enable_test_visibility()# Discover itemsapi.TestSession.discover("manual_test_api_example","my_manual_framework","1.0.0")test_module_1_id=api.TestModuleId("module_1")api.TestModule.discover(test_module_1_id)test_suite_1_id=api.TestSuiteId(test_module_1_id,"suite_1")api.TestSuite.discover(test_suite_1_id)test_1_id=api.TestId(test_suite_1_id,"test_1")api.Test.discover(test_1_id)# A parameterized test with codeowners and a source filetest_2_codeowners=["team_1","team_2"]test_2_source_info=api.TestSourceFileInfo(pathlib.Path("/path/to_my/tests.py"),16,35)parametrized_test_2_a_id=api.TestId(test_suite_1_id,"test_2",parameters='{"parameter_1": "value_is_a"}')api.Test.discover(parametrized_test_2_a_id,codeowners=test_2_codeowners,source_file_info=test_2_source_info,resource="overriden resource name A",)parametrized_test_2_b_id=api.TestId(test_suite_1_id,"test_2",parameters='{"parameter_1": "value_is_b"}')api.Test.discover(parametrized_test_2_b_id,codeowners=test_2_codeowners,source_file_info=test_2_source_info,resource="overriden resource name B")test_3_id=api.TestId(test_suite_1_id,"test_3")api.Test.discover(test_3_id)test_4_id=api.TestId(test_suite_1_id,"test_4")api.Test.discover(test_4_id)# Start and execute itemsapi.TestSession.start()api.TestModule.start(test_module_1_id)api.TestSuite.start(test_suite_1_id)# test_1 passes successfullyapi.Test.start(test_1_id)api.Test.mark_pass(test_1_id)# test_2's first parametrized test succeeds, but the second fails without attaching exception infoapi.Test.start(parametrized_test_2_a_id)api.Test.mark_pass(parametrized_test_2_a_id)api.Test.start(parametrized_test_2_b_id)api.Test.mark_fail(parametrized_test_2_b_id)# test_3 is skippedapi.Test.start(test_3_id)api.Test.mark_skip(test_3_id,skip_reason="example skipped test")# test_4 fails, and attaches exception infoapi.Test.start(test_4_id)try:raise(ValueError("this test failed"))except:api.Test.mark_fail(test_4_id,exc_info=api.TestExcInfo(*sys.exc_info()))# Finish suites and modulesapi.TestSuite.finish(test_suite_1_id)api.TestModule.finish(test_module_1_id)api.TestSession.finish()
Configuration settings
The following is a list of the most important configuration settings that can be used with the tracer, either in code or using environment variables:
DD_SERVICE
Name of the service or library under test. Environment variable: DD_SERVICE Default: pytest Example: my-python-app
DD_ENV
Name of the environment where tests are being run. Environment variable: DD_ENV Default: none Examples: local, ci
Datadog uses Git information for visualizing your test results and grouping them by repository, branch, and commit. Git metadata is automatically collected by the test instrumentation from CI provider environment variables and the local .git folder in the project path, if available.
If you are running tests in non-supported CI providers or with no .git folder, you can set the Git information manually using environment variables. These environment variables take precedence over any auto-detected information. Set the following environment variables to provide Git information:
DD_GIT_REPOSITORY_URL
URL of the repository where the code is stored. Both HTTP and SSH URLs are supported. Example: git@github.com:MyCompany/MyApp.git, https://github.com/MyCompany/MyApp.git
DD_GIT_BRANCH
Git branch being tested. Leave empty if providing tag information instead. Example: develop
DD_GIT_TAG
Git tag being tested (if applicable). Leave empty if providing branch information instead. Example: 1.0.1
DD_GIT_COMMIT_SHA
Full commit hash. Example: a18ebf361cc831f5535e58ec4fae04ffd98d8152
DD_GIT_COMMIT_MESSAGE
Commit message. Example: Set release number
DD_GIT_COMMIT_AUTHOR_NAME
Commit author name. Example: John Smith
DD_GIT_COMMIT_AUTHOR_EMAIL
Commit author email. Example: john@example.com
DD_GIT_COMMIT_AUTHOR_DATE
Commit author date in ISO 8601 format. Example: 2021-03-12T16:00:28Z
DD_GIT_COMMIT_COMMITTER_NAME
Commit committer name. Example: Jane Smith
DD_GIT_COMMIT_COMMITTER_EMAIL
Commit committer email. Example: jane@example.com
DD_GIT_COMMIT_COMMITTER_DATE
Commit committer date in ISO 8601 format. Example: 2021-03-12T16:00:28Z
Known limitations
Plugins for pytest that alter test execution may cause unexpected behavior.
Parallelization
Plugins that introduce parallelization to pytest (such as pytest-xdist or pytest-forked) create one session event for each parallelized instance. Multiple module or suite events may be created if tests from the same package or module execute in different processes.
The overall count of test events (and their correctness) remain unaffected. Individual session, module, or suite events may have inconsistent results with other events in the same pytest run.
Test ordering
Plugins that change the ordering of test execution (such as pytest-randomly) can create multiple module or suite events. The duration and results of module or suite events may also be inconsistent with the results reported by pytest.
The overall count of test events (and their correctness) remain unaffected.
In some cases, if your unittest test execution is run in a parallel manner, this may break the instrumentation and affect test optimization.
Datadog recommends you use up to one process at a time to prevent affecting test optimization.
Further reading
Additional helpful documentation, links, and articles: