Test Impact Analysis for Java

Docs > Test Optimization in Datadog > Test Impact Analysis > Configure Test Impact Analysis > Test Impact Analysis for Java

Compatibility

Test Impact Analysis is supported in dd-java-agent >= 1.27.0.

The following test frameworks are supported:

JUnit >= 4.10 and >= 5.3
TestNG >= 6.4
Spock >= 2.0
Cucumber >= 5.4.0
Karate >= 1.0.0
Scalatest >= 3.0.8

Setup

Test Optimization

Prior to setting up Test Impact Analysis, set up Test Optimization for Java. If you are reporting data through the Agent, use v6.40 and later or v7.40 and later.

Activate Test Impact Analysis for the test service

You, or a user in your organization with the Intelligent Test Runner Activation (intelligent_test_runner_activation_write) permission, must activate Test Impact Analysis on the Test Service Settings page.

Test Impact Analysis enabled in test service settings in the CI section of Datadog.

Run tests with Test Impact Analysis enabled

After completing setup, run your tests as you normally do:

./gradlew cleanTest test -Dorg.gradle.jvmargs=\
-javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=\
dd.civisibility.enabled=true,\
dd.env=ci,\
dd.service=my-java-app

MAVEN_OPTS=-javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=\
dd.civisibility.enabled=true,\
dd.env=ci,\
dd.service=my-java-app \
mvn clean verify

Disabling skipping for specific tests

You can override the Test Impact Analysis behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.

Why make tests unskippable?

Test Impact Analysis uses code coverage data to determine whether or not tests should be skipped. In some cases, this data may not be sufficient to make this determination.

Examples include:

Tests that read data from text files
Tests that interact with APIs outside of the code being tested (such as remote REST APIs)
Designating tests as unskippable ensures that Test Impact Analysis runs them regardless of coverage data.

Compatibility

Unskippable tests are supported in the following versions and testing frameworks:

JUnit >= 4.10 and >= 5.3
TestNG >= 6.4
Spock >= 2.2
Cucumber >= 5.4.0
Scalatest >= 3.0.8

Marking tests as unskippable

Individual test case

Add a JUnit Tag with the value datadog_itr_unskippable to your test case to mark it as unskippable.

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Tags;
import org.junit.jupiter.api.Test;

public class MyTestSuite {

  @Test
  @Tags({@Tag("datadog_itr_unskippable")})
  public void myTest() {
    // ...
  }
}

Test suite

Add a JUnit Tag with the value datadog_itr_unskippable to your test suite to mark it as unskippable.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by Test Impact Analysis.

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Tags;
import org.junit.jupiter.api.Test;

@Tags({@Tag("datadog_itr_unskippable")})
public class MyTestSuite {

  @Test
  public void myTest() {
    // ...
  }
}

Individual test case

Add a JUnit Category with the value datadog_itr_unskippable to your test case to mark it as unskippable. You do not have to create the datadog_itr_unskippable for every test case or test suite, one category is enough for the entire project.

import org.junit.Test;
import org.junit.experimental.categories.Category;

public class MyTestSuite {

  @Category(datadog_itr_unskippable.class)
  @Test
  public void myTest() {
    // ...
  }

  public interface datadog_itr_unskippable {}
}

Test suite

Add a JUnit Tag with the value datadog_itr_unskippable to your test suite to mark it as unskippable. You do not have to create the datadog_itr_unskippable for every test case or test suite, one category is enough for the entire project.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by Test Impact Analysis.

import org.junit.Test;
import org.junit.experimental.categories.Category;

@Category(MyTestSuite.datadog_itr_unskippable.class)
public class MyTestSuite {

  @Test
  public void myTest() {
    // ...
  }

  public interface datadog_itr_unskippable {}
}

Individual test case

Add a group with the value datadog_itr_unskippable to your test case to mark it as unskippable.

import org.testng.annotations.Test;

public class MyTestSuite {

  @Test(groups = "datadog_itr_unskippable")
  public void myTest() {
    // ...
  }
}

Test suite

Add a group with the value datadog_itr_unskippable to your test suite to mark it as unskippable.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by Test Impact Analysis.

import org.testng.annotations.Test;

@Test(groups = "datadog_itr_unskippable")
public class MyTestSuite {

  @Test
  public void myTest() {
    // ...
  }
}

Individual test case

Add a spock.lang.Tag with the value datadog_itr_unskippable to your test case to mark it as unskippable.

import spock.lang.Specification
import spock.lang.Tag

class MyTestSuite extends Specification {

  @Tag("datadog_itr_unskippable")
  def myTest() {
    // ...
  }
}

Test suite

Add a spock.lang.Tag with the value datadog_itr_unskippable to your test suite to mark it as unskippable.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by Test Impact Analysis.

import spock.lang.Specification
import spock.lang.Tag

@Tag("datadog_itr_unskippable")
class MyTestSuite extends Specification {

  def myTest() {
    // ...
  }
}

Individual scenario

Add datadog_itr_unskippable tag to your gherkin scenario to mark it as unskippable.

Feature: My Feature

  @datadog_itr_unskippable
  Scenario: My Scenario
    # ...

Feature

Add datadog_itr_unskippable tag to your gherkin feature to mark it as unskippable.

If a feature is marked as unskippable, none of the scenarios from that feature can be skipped by Test Impact Analysis.

@datadog_itr_unskippable
Feature: My Feature

  Scenario: My Scenario
    # ...

Create a Tag with the value datadog_itr_unskippable and tag your test case with it:

import org.scalatest.Tag
import org.scalatest.flatspec.AnyFlatSpec

object ItrUnskippableTag extends Tag("datadog_itr_unskippable")

class MyTestSuite extends AnyFlatSpec {
  "myTest" should "assert something" taggedAs ItrUnskippableTag in {
    // ...
  }
}