CI Visibility is not available in the selected site () at this time.

If your CI provider is Jenkins, you can use UI-based configuration to enable Test Visibility for your jobs and pipelines.


Supported languages:

Python 2>= 2.7
Python 3>= 3.6

Supported test frameworks:

Test FrameworkVersion
pytest>= 3.0.0 <= 7.0.0
pytest-benchmark>= 3.1.0
unittest>= 3.7

Configuring reporting method

To report test results to Datadog, you need to configure the Datadog Python library:

If you are using a cloud CI provider without access to the underlying worker nodes, such as GitHub Actions or CircleCI, configure the library to use the Agentless mode. For this, set the following environment variables:

Enables or disables Agentless mode.
Default: false
DD_API_KEY (Required)
The Datadog API key used to upload the test results.
Default: (empty)

Additionally, configure the Datadog site to which you want to send data.

DD_SITE (Required)
The Datadog site to upload results to.

If you are running tests on an on-premises CI provider, such as Jenkins or self-managed GitLab CI, install the Datadog Agent on each worker node by following the Agent installation instructions. This is the recommended option as it allows you to automatically link test results to logs and underlying host metrics.

If you are using a Kubernetes executor, Datadog recommends using the Datadog Operator. The operator includes Datadog Admission Controller which can automatically inject the tracer library into the build pods. Note: If you use the Datadog Operator, there is no need to download and inject the tracer library since the Admission Controller can do this for you, so you can skip the corresponding step below. However, you still need to make sure that your pods set the environment variables or command-line parameters necessary to enable Test Visibility.

If you are not using Kubernetes or can’t use the Datadog Admission Controller and the CI provider is using a container-based executor, set the DD_TRACE_AGENT_URL environment variable (which defaults to http://localhost:8126) in the build container running the tracer to an endpoint that is accessible from within that container. Note: Using localhost inside the build references the container itself and not the underlying worker node or any container where the Agent might be running in.

DD_TRACE_AGENT_URL includes the protocol and port (for example, http://localhost:8126) and takes precedence over DD_AGENT_HOST and DD_TRACE_AGENT_PORT, and is the recommended configuration parameter to configure the Datadog Agent’s URL for CI Visibility.

If you still have issues connecting to the Datadog Agent, use the Agentless Mode. Note: When using this method, tests are not correlated with logs and infrastructure metrics.

Installing the Python tracer

Install the Python tracer by running:

pip install -U ddtrace

For more information, see the Python tracer installation documentation.

Instrumenting your tests

To enable instrumentation of pytest tests, add the --ddtrace option when running pytest, specifying the name of the service or library under test in the DD_SERVICE environment variable, and the environment where tests are being run (for example, local when running tests on a developer workstation, or ci when running them on a CI provider) in the DD_ENV environment variable:

DD_SERVICE=my-python-app DD_ENV=ci pytest --ddtrace

If you also want to enable the rest of the APM integrations to get more information in your flamegraph, add the --ddtrace-patch-all option:

DD_SERVICE=my-python-app DD_ENV=ci pytest --ddtrace --ddtrace-patch-all

Adding custom tags to tests

To add custom tags to your tests, declare ddspan as an argument in your test:

from ddtrace import tracer

# Declare `ddspan` as argument to your test
def test_simple_case(ddspan):
    # Set your tags
    ddspan.set_tag("test_owner", "my_team")
    # test continues normally
    # ...

To create filters or group by fields for these tags, you must first create facets. For more information about adding tags, see the Adding Tags section of the Python custom instrumentation documentation.

Adding custom metrics to tests

Just like tags, to add custom metrics to your tests, use the current active span:

from ddtrace import tracer

# Declare `ddspan` as an argument to your test
def test_simple_case(ddspan):
    # Set your tags
    ddspan.set_tag("memory_allocations", 16)
    # test continues normally
    # ...

Read more about custom metrics in the Add Custom Metrics Guide.

Known limitations

Plugins for pytest that alter test execution may cause unexpected behavior.


Plugins that introduce parallelization to pytest (such as pytest-xdist or pytest-forked) create one session event for each parallelized instance. Multiple module or suite events may be created if tests from the same package or module execute in different processes.

The overall count of test events (and their correctness) remain unaffected. Individual session, module, or suite events may have inconsistent results with other events in the same pytest run.

Test ordering

Plugins that chage the ordering of test execution (such as pytest-randomly) can create multiple module or suite events. The duration and results of module or suite events may also be inconsistent with the results reported by pytest.

The overall count of test events (and their correctness) remain unaffected.

To instrument your benchmark tests with pytest-benchmark, run your benchmark tests with the --ddtrace option when running pytest, and Datadog detects metrics from pytest-benchmark automatically:

def square_value(value):
    return value * value

def test_square_value(benchmark):
    result = benchmark(square_value, 5)
    assert result == 25

To enable instrumentation of unittest tests, run your tests by appending ddtrace-run to the beginning of your unittest command.

Make sure to specify the name of the service or library under test in the DD_SERVICE environment variable. Additionally, you may declare the environment where tests are being run in the DD_ENV environment variable:

DD_SERVICE=my-python-app DD_ENV=ci ddtrace-run python -m unittest

Alternatively, if you wish to enable unittest instrumentation manually, use patch() to enable the integration:

from ddtrace import patch
import unittest

class MyTest(unittest.TestCase):
def test_will_pass(self):
assert True

Known limitations

In some cases, if your unittest test execution is run in a parallel manner, this may break the instrumentation and affect test visibility.

Datadog recommends you use up to one process at a time to prevent affecting test visibility.

Configuration settings

The following is a list of the most important configuration settings that can be used with the tracer, either in code or using environment variables:

Name of the service or library under test.
Environment variable: DD_SERVICE
Default: pytest
Example: my-python-app
Name of the environment where tests are being run.
Environment variable: DD_ENV
Default: none
Examples: local, ci

For more information about service and env reserved tags, see Unified Service Tagging.

The following environment variable can be used to configure the location of the Datadog Agent:

Datadog Agent URL for trace collection in the form http://hostname:port.
Default: http://localhost:8126

All other Datadog Tracer configuration options can also be used.

Collecting Git metadata

Datadog uses Git information for visualizing your test results and grouping them by repository, branch, and commit. Git metadata is automatically collected by the test instrumentation from CI provider environment variables and the local .git folder in the project path, if available.

If you are running tests in non-supported CI providers or with no .git folder, you can set the Git information manually using environment variables. These environment variables take precedence over any auto-detected information. Set the following environment variables to provide Git information:

URL of the repository where the code is stored. Both HTTP and SSH URLs are supported.
Git branch being tested. Leave empty if providing tag information instead.
Example: develop
Git tag being tested (if applicable). Leave empty if providing branch information instead.
Example: 1.0.1
Full commit hash.
Example: a18ebf361cc831f5535e58ec4fae04ffd98d8152
Commit message.
Example: Set release number
Commit author name.
Example: John Smith
Commit author email.
Commit author date in ISO 8601 format.
Example: 2021-03-12T16:00:28Z
Commit committer name.
Example: Jane Smith
Commit committer email.
Commit committer date in ISO 8601 format.
Example: 2021-03-12T16:00:28Z

Further reading