Java Tests

Compatibility

Supported test frameworks:

If your test framework is not supported, you can try instrumenting your tests using Manual Testing API.

Supported build systems:

If your build system is not supported, you can try configuring tracer injection manually.

Setup

To set up Test Visibility for Java, you need to perform the following steps:

1. Configure tracer reporting method.
2. Download tracer library to the hosts where your tests are executed.
3. Run your tests with the tracer attached.

Configuring reporting method

This step involves configuring how Datadog Java Tracer injected into your tests reports data to Datadog. There are two main options:

• Reporting the data to Datadog Agent, which will forward it to Datadog.
• Reporting the data directly to Datadog.

Downloading tracer library

You only need to download the tracer library once for each server.

If the tracer library is already available locally on the server, you can proceed directly to running the tests.

Declare DD_TRACER_FOLDER variable with the path to the folder where you want to store the downloaded tracer JAR:

export DD_TRACER_FOLDER=... // e.g. ~/.datadog

Run the command below to download the tracer JAR to the specified folder:

wget -O $DD_TRACER_FOLDER/dd-java-agent.jar https://dtdg.co/latest-java-tracer

You can run the java -jar $DD_TRACER_FOLDER/dd-java-agent.jar command to check the version of the tracer library.

Running your tests

MAVEN_OPTS=-javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=\
dd.civisibility.enabled=true,\
dd.env=ci,\
dd.service=my-java-app \
mvn clean verify

./gradlew cleanTest test -Dorg.gradle.jvmargs=\
-javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=\
dd.civisibility.enabled=true,\
dd.env=ci,\
dd.service=my-java-app

<project name="my-project" default="test" xmlns:if="ant:if">
  <!-- ... -->
  <property environment="env"/>

  <target name="test" depends="compile-tests">
    <junit printsummary="on" fork="true">
      <jvmarg if:set="ci-visibility" value="-javaagent:${env.DD_TRACER_FOLDER}/dd-java-agent.jar=dd.civisibility.enabled=true,dd.env=ci,dd.service=my-java-app"/>
      <!-- ... -->
    </junit>
  </target>
</project>

ant test -Dci-visibility=true

Configuration

Default configuration values work well in most cases.

However, if there is a need to fine-tune the tracer’s behavior, Datadog Tracer configuration options can be used.

Collecting Git metadata

Datadog uses Git information for visualizing your test results and grouping them by repository, branch, and commit. Git metadata is automatically collected by the test instrumentation from CI provider environment variables and the local .git folder in the project path, if available.

If you are running tests in non-supported CI providers or with no .git folder, you can set the Git information manually using environment variables. These environment variables take precedence over any auto-detected information. Set the following environment variables to provide Git information:

DD_GIT_REPOSITORY_URL

URL of the repository where the code is stored. Both HTTP and SSH URLs are supported.
Example: git@github.com:MyCompany/MyApp.git, https://github.com/MyCompany/MyApp.git

DD_GIT_BRANCH

Git branch being tested. Leave empty if providing tag information instead.
Example: develop

DD_GIT_TAG

Git tag being tested (if applicable). Leave empty if providing branch information instead.
Example: 1.0.1

DD_GIT_COMMIT_SHA

Full commit hash.
Example: a18ebf361cc831f5535e58ec4fae04ffd98d8152

DD_GIT_COMMIT_MESSAGE

Commit message.
Example: Set release number

DD_GIT_COMMIT_AUTHOR_NAME

Commit author name.
Example: John Smith

DD_GIT_COMMIT_AUTHOR_EMAIL

Commit author email.
Example: john@example.com

DD_GIT_COMMIT_AUTHOR_DATE

Commit author date in ISO 8601 format.
Example: 2021-03-12T16:00:28Z

DD_GIT_COMMIT_COMMITTER_NAME

Commit committer name.
Example: Jane Smith

DD_GIT_COMMIT_COMMITTER_EMAIL

Commit committer email.
Example: jane@example.com

DD_GIT_COMMIT_COMMITTER_DATE

Commit committer date in ISO 8601 format.
Example: 2021-03-12T16:00:28Z

Extensions

The dd-trace-api library exposes a set of APIs that can be used to extend the tracer’s functionality programmatically. To use the extensions described below, add this library as a compile-time dependency to your project.

Adding custom tags to tests

You can add custom tags to your tests by using the current active span:

// inside your test
final Span span = GlobalTracer.get().activeSpan();
if (span != null) {
  span.setTag("test_owner", "my_team");
}
// test continues normally
// ...

To create filters or group by fields for these tags, you must first create facets.

For more information about adding tags, see the Adding Tags section of the Java custom instrumentation documentation.

Adding custom metrics to tests

Just like tags, you can add custom metrics to your tests by using the current active span:

// inside your test
final Span span = GlobalTracer.get().activeSpan();
if (span != null) {
  span.setTag("test.memory.usage", 1e8);
}
// test continues normally
// ...

For more information about custom metrics, see the Add Custom Metrics guide.

Using manual testing API

If you use one of the supported testing frameworks, the Java Tracer automatically instruments your tests and sends the results to the Datadog backend.

If you are using a framework that is not supported, or an ad-hoc testing solution, you can harness the manual testing API, which also reports test results to the backend.

Domain model

The API is based around four concepts: test session, test module, test suite, and test.

Test session

A test session represents a project build, which typically corresponds to execution of a test command issued by a user or by a CI script.

To start a test session, call datadog.trace.api.civisibility.CIVisibility#startSession and pass the name of the project and the name of the testing framework you used.

When all your tests have finished, call datadog.trace.api.civisibility.DDTestSession#end, which forces the library to send all remaining test results to the backend.

Test module

A test module represents a smaller unit of work within a project build, typically corresponding to a project module. For example, a Maven submodule or Gradle subproject.

To start a test mode, call datadog.trace.api.civisibility.DDTestSession#testModuleStart and pass the name of the module.

When the module has finished building and testing, call datadog.trace.api.civisibility.DDTestModule#end.

Test Suite

A test suite comprises a set of tests that share common functionality. They can share a common initialization and teardown, and can also share some variables. A single suite usually corresponds to a Java class that contains test cases.

Create test suites in a test module by calling datadog.trace.api.civisibility.DDTestModule#testSuiteStart and passing the name of the test suite.

Call datadog.trace.api.civisibility.DDTestSuite#end when all the related tests in the suite have finished their execution.

Test

A test represents a single test case that is executed as part of a test suite. Usually it corresponds to a method that contains testing logic.

Create tests in a suite by calling datadog.trace.api.civisibility.DDTestSuite#testStart and passing the name of the test.

Call datadog.trace.api.civisibility.DDTest#end when a test has finished execution.

Code Example

The following code represents a simple usage of the API:

package com.datadog.civisibility.example;

import datadog.trace.api.civisibility.CIVisibility;
import datadog.trace.api.civisibility.DDTest;
import datadog.trace.api.civisibility.DDTestModule;
import datadog.trace.api.civisibility.DDTestSession;
import datadog.trace.api.civisibility.DDTestSuite;
import java.lang.reflect.Method;

// the null arguments in the calls below are optional startTime/endTime values:
// when they are not specified, current time is used
public class ManualTest {
    public static void main(String[] args) throws Exception {
        DDTestSession testSession = CIVisibility.startSession("my-project-name", "my-test-framework", null);
        testSession.setTag("my-tag", "additional-session-metadata");
        try {
            runTestModule(testSession);
        } finally {
            testSession.end(null);
        }
    }

    private static void runTestModule(DDTestSession testSession) throws Exception {
        DDTestModule testModule = testSession.testModuleStart("my-module", null);
        testModule.setTag("my-module-tag", "additional-module-metadata");
        try {
            runFirstTestSuite(testModule);
            runSecondTestSuite(testModule);
        } finally {
            testModule.end(null);
        }
    }

    private static void runFirstTestSuite(DDTestModule testModule) throws Exception {
        DDTestSuite testSuite = testModule.testSuiteStart("my-suite", ManualTest.class, null);
        testSuite.setTag("my-suite-tag", "additional-suite-metadata");
        try {
            runTestCase(testSuite);
        } finally {
            testSuite.end(null);
        }
    }

    private static void runTestCase(DDTestSuite testSuite) throws Exception {
        Method myTestCaseMethod = ManualTest.class.getDeclaredMethod("myTestCase");
        DDTest ddTest = testSuite.testStart("myTestCase", myTestCaseMethod, null);
        ddTest.setTag("my-test-case-tag", "additional-test-case-metadata");
        ddTest.setTag("my-test-case-tag", "more-test-case-metadata");
        try {
            myTestCase();
        } catch (Exception e) {
            ddTest.setErrorInfo(e); // pass error info to mark test case as failed
        } finally {
            ddTest.end(null);
        }
    }

    private static void myTestCase() throws Exception {
        // run some test logic
    }

    private static void runSecondTestSuite(DDTestModule testModule) {
        DDTestSuite secondTestSuite = testModule.testSuiteStart("my-second-suite", ManualTest.class, null);
        secondTestSuite.setSkipReason("this test suite is skipped"); // pass skip reason to mark test suite as skipped
        secondTestSuite.end(null);
    }
}

Always call datadog.trace.api.civisibility.DDTestSession#end at the end so that all the test info is flushed to Datadog.

Troubleshooting

The tests are not appearing in Datadog after enabling CI Visibility in the tracer

Ensure that you are using the latest version of the tracer.

Verify that your build system and testing framework are supported by CI Visibility. See the list of supported build systems and test frameworks.

Ensure that the dd.civisibility.enabled property is set to true in the tracer arguments.

Check the build output for any errors that indicate tracer misconfiguration, such as an unset DD_API_KEY environment variable.

Tests or source code compilation fails when building a project with the tracer attached

By default, CI Visibility runs Java code compilation with a compiler plugin attached.

The plugin is optional, as it only serves to reduce the performance overhead.

Depending on the build configuration, adding the plugin can sometimes disrupt the compilation process.

If the plugin interferes with the build, disable it by adding dd.civisibility.compiler.plugin.auto.configuration.enabled=false to the list of -javaagent arguments.

Tests fail when building a project with the tracer attached

In some cases attaching the tracer can break tests, especially if they run asserts on the internal state of the JVM or instances of third-party libraries’ classes.

While the best approach is such cases is to update the tests, there is also a quicker option of disabling the tracer’s third-party library integrations.

The integrations provide additional insights into what happens in the tested code and are especially useful in integration tests, to monitor things like HTTP requests or database calls. They are enabled by default.

To disable a specific integration, refer to the Datadog Tracer Compatibility table for the relevant configuration property names. For example, to disable OkHttp3 client request integration, add dd.integration.okhttp-3.enabled=false to the list of -javaagent arguments.

To disable all integrations, augment the list of -javaagent arguments with dd.trace.enabled=false.

Further reading

Additional helpful documentation, links, and articles:

Forwarding Environment Variables for Tests in ContainersDOCUMENTATION Explore Test Results and PerformanceDOCUMENTATION Troubleshooting CI VisibilityDOCUMENTATION