Jenkins 또는 자체 관리형 GitLab CI와 같은 온프레미스 CI 공급자에서 테스트를 실행하는 경우, 에이전트 설치 지침에 따라 각 작업자 노드에 Datadog 에이전트를 설치합니다.
자동으로 테스트 결과를 로그와 기본 호트스 메트릭과 연결할 수 있기 때문에 이 옵션을 추천합니다.
쿠버네티스 실행기를 사용하는 경우 Datadog에서는 Datadog 연산자를 사용할 것을 권고합니다.
연산자에는 Datadog 허용 제어기가 포함되어 있어 빌드 파드에 자동으로 추적기 라이브러리를 삽입합니다.
참고: Datadog 연산자를 사용할 경우 허용 제어기가 작업을 해주기 때문에 추적기 라이브러리를 다운로드 받고 삽입할 필요가 없습니다. 따라서 아래 해당 단계를 건너뛰어도 됩니다.
그러나 테스트 가시화 기능을 사용할 때 필요한 파드의 환경 변수나 명령줄 파라미터는 설정해야 합니다.
쿠버네티스를 사용하지 않거나 Datadog 허용 제어기를 사용할 수 없고 CI 공급자가 컨테이너 기반 실행기를 사용하는 경우, 추적기를 실행하는 빌드 컨테이너에서 환경 변수 DD_TRACE_AGENT_URL(기본값 http://localhost:8126)를 해당 컨테이너 내에서 액세스할 수 있는 엔드포인트로 설정합니다. 참고: 빌드 내에서 localhost를 사용하면 기본 작업자 노드나 에이전트를 실행하는 컨테이너를 참조하지 않고 컨테이너 자체를 참조합니다.
DD_TRACE_AGENT_URL 은 프로토콜과 포트(예: http://localhost:8126)를 포함하고 DD_AGENT_HOST과 DD_TRACE_AGENT_PORT보다 우선하며, CI Visibility를 위해 Datadog 에이전트의 URL을 설정하는 데 권장되는 설정 파라미터입니다.
Datdog 에이전트에 연결하는 데 아직 문제가 있다면 에이전트리스 모드를 사용해 보세요.
참고: 이 방법을 사용할 경우 테스트가 로그 및 인프라스트럭처 메트릭과 상관 관계를 수립하지 않습니다.
Downloading tracer library
You only need to download the tracer library once for each server.
If the tracer library is already available locally on the server, you can proceed directly to running the tests.
Declare DD_TRACER_FOLDER variable with the path to the folder where you want to store the downloaded tracer JAR:
exportDD_TRACER_FOLDER=... // e.g. ~/.datadog
Run the command below to download the tracer JAR to the specified folder:
테스트 중인 Git 브랜치입니다. 대신 태그 정보를 제공하는 경우 비워 둡니다. 예시: develop
DD_GIT_TAG
테스트 중인 Git 태그입니다(해당되는 경우). 대신 브랜치 정보를 제공하는 경우 비워 둡니다. 예시: 1.0.1
DD_GIT_COMMIT_SHA
전체 커밋 해시입니다. 예시: a18ebf361cc831f5535e58ec4fae04ffd98d8152
DD_GIT_COMMIT_MESSAGE
커밋 메시지입니다. 예시: Set release number
DD_GIT_COMMIT_AUTHOR_NAME
커밋 작성자 이름입니다. 예시: John Smith
DD_GIT_COMMIT_AUTHOR_EMAIL
커밋 작성자 이메일입니다. 예시: john@example.com
DD_GIT_COMMIT_AUTHOR_DATE
ISO 8601 형식의 커밋 작성자 날짜입니다. 예시: 2021-03-12T16:00:28Z
DD_GIT_COMMIT_COMMITTER_NAME
커밋 커미터 이름입니다. 예시: Jane Smith
DD_GIT_COMMIT_COMMITTER_EMAIL
커밋 커미터 이메일입니다. 예시: jane@example.com
DD_GIT_COMMIT_COMMITTER_DATE
ISO 8601 형식의 커밋 커미터 날짜입니다. 예시: 2021-03-12T16:00:28Z
Extensions
The tracer exposes a set of APIs that can be used to extend its functionality programmatically.
Adding custom tags to tests
To add custom tags, include the opentelemetry-api library as a compile-time dependency and set dd.trace.otel.enabled (system property) or DD_TRACE_OTEL_ENABLED (environment variable) to true.
You can then add custom tags to your tests by using the active span:
importio.opentelemetry.api.trace.Span;// ...// inside your testSpanspan=Span.current();span.setAttribute("test_owner","my_team");// test continues normally// ...
For more information about adding tags, see the Adding Tags section of the Java custom instrumentation documentation.
To add custom tags, include the opentracing-util library as a compile-time dependency to your project.
You can then add custom tags to your tests by using the active span:
importio.opentracing.Span;importio.opentracing.util.GlobalTracer;// ...// inside your testfinalSpanspan=GlobalTracer.get().activeSpan();if(span!=null){span.setTag("test_owner","my_team");}// test continues normally// ...
To create filters or group by fields for these tags, you must first create facets.
For more information about adding tags, see the Adding Tags section of the Java custom instrumentation documentation.
Adding custom measures to tests
Just like tags, you can add custom measures to your tests by using the current active span:
importio.opentelemetry.api.trace.Span;// ...// inside your testSpanspan=Span.current();span.setAttribute("test.memory.usage",1e8);// test continues normally// ...
importio.opentracing.Span;importio.opentracing.util.GlobalTracer;// ...// inside your testfinalSpanspan=GlobalTracer.get().activeSpan();if(span!=null){span.setTag("test.memory.usage",1e8);}// test continues normally// ...
If you use one of the supported testing frameworks, the Java Tracer automatically instruments your tests and sends the results to the Datadog backend.
If you are using a framework that is not supported, or an ad-hoc testing solution, you can harness the manual testing API, which also reports test results to the backend.
To use the manual testing API, add the dd-trace-api library as a compile-time dependency to your project.
Domain model
The API is based around four concepts: test session, test module, test suite, and test.
Test session
A test session represents a project build, which typically corresponds to execution of a test command issued by a user or by a CI script.
To start a test session, call datadog.trace.api.civisibility.CIVisibility#startSession and pass the name of the project and the name of the testing framework you used.
When all your tests have finished, call datadog.trace.api.civisibility.DDTestSession#end, which forces the library to send all remaining test results to the backend.
Test module
A test module represents a smaller unit of work within a project build, typically corresponding to a project module. For example, a Maven submodule or Gradle subproject.
To start a test mode, call datadog.trace.api.civisibility.DDTestSession#testModuleStart and pass the name of the module.
When the module has finished building and testing, call datadog.trace.api.civisibility.DDTestModule#end.
Test Suite
A test suite comprises a set of tests that share common functionality.
They can share a common initialization and teardown, and can also share some variables.
A single suite usually corresponds to a Java class that contains test cases.
Create test suites in a test module by calling datadog.trace.api.civisibility.DDTestModule#testSuiteStart and passing the name of the test suite.
Call datadog.trace.api.civisibility.DDTestSuite#end when all the related tests in the suite have finished their execution.
Test
A test represents a single test case that is executed as part of a test suite.
Usually it corresponds to a method that contains testing logic.
Create tests in a suite by calling datadog.trace.api.civisibility.DDTestSuite#testStart and passing the name of the test.
Call datadog.trace.api.civisibility.DDTest#end when a test has finished execution.
Code Example
The following code represents a simple usage of the API:
packagecom.datadog.civisibility.example;importdatadog.trace.api.civisibility.CIVisibility;importdatadog.trace.api.civisibility.DDTest;importdatadog.trace.api.civisibility.DDTestModule;importdatadog.trace.api.civisibility.DDTestSession;importdatadog.trace.api.civisibility.DDTestSuite;importjava.lang.reflect.Method;// the null arguments in the calls below are optional startTime/endTime values:// when they are not specified, current time is usedpublicclassManualTest{publicstaticvoidmain(String[]args)throwsException{DDTestSessiontestSession=CIVisibility.startSession("my-project-name","my-test-framework",null);testSession.setTag("my-tag","additional-session-metadata");try{runTestModule(testSession);}finally{testSession.end(null);}}privatestaticvoidrunTestModule(DDTestSessiontestSession)throwsException{DDTestModuletestModule=testSession.testModuleStart("my-module",null);testModule.setTag("my-module-tag","additional-module-metadata");try{runFirstTestSuite(testModule);runSecondTestSuite(testModule);}finally{testModule.end(null);}}privatestaticvoidrunFirstTestSuite(DDTestModuletestModule)throwsException{DDTestSuitetestSuite=testModule.testSuiteStart("my-suite",ManualTest.class,null);testSuite.setTag("my-suite-tag","additional-suite-metadata");try{runTestCase(testSuite);}finally{testSuite.end(null);}}privatestaticvoidrunTestCase(DDTestSuitetestSuite)throwsException{MethodmyTestCaseMethod=ManualTest.class.getDeclaredMethod("myTestCase");DDTestddTest=testSuite.testStart("myTestCase",myTestCaseMethod,null);ddTest.setTag("my-test-case-tag","additional-test-case-metadata");ddTest.setTag("my-test-case-tag","more-test-case-metadata");try{myTestCase();}catch(Exceptione){ddTest.setErrorInfo(e);// pass error info to mark test case as failed}finally{ddTest.end(null);}}privatestaticvoidmyTestCase()throwsException{// run some test logic}privatestaticvoidrunSecondTestSuite(DDTestModuletestModule){DDTestSuitesecondTestSuite=testModule.testSuiteStart("my-second-suite",ManualTest.class,null);secondTestSuite.setSkipReason("this test suite is skipped");// pass skip reason to mark test suite as skippedsecondTestSuite.end(null);}}
Always call datadog.trace.api.civisibility.DDTestSession#end at the end so that all the test info is flushed to Datadog.
Best practices
Deterministic test parameters representation
Test Optimization works best when the test parameters are deterministic and stay the same between test runs.
If a test case has a parameter that varies between test executions (such as a current date, a random number, or an instance of a class whose toString() method is not overridden), some of the product features may not work as expected.
For example, the history of executions may not be available, or the test case may not be classified as flaky even if it exhibits flakiness.
The best way to fix this is to make sure that the test parameters are the same between test runs.
In JUnit 5, this can also be addressed by customizing the string representation of the test parameters without changing their values.
To do so, use org.junit.jupiter.api.Named interface or change the name parameter of the org.junit.jupiter.params.ParameterizedTest annotation:
@ParameterizedTest@MethodSource("namedArguments")voidparameterizedTest(Strings,Dated){// The second parameter in this test case is non-deterministic.// In the argument provider method it is wrapped with Named to ensure it has a deterministic name.}staticStream<Arguments>namedArguments(){returnStream.of(Arguments.of("a string",Named.of("current date",newDate())),Arguments.of("another string",Named.of("a date in the future",newDate(System.currentTimeMillis()+TimeUnit.DAYS.toMillis(1)))));}
@ParameterizedTest(name="[{index}] {0}, a random number from one to ten")@MethodSource("randomArguments")voidanotherParameterizedTest(Strings,inti){// The second parameter in this test case is non-deterministic.// The name of the parameterized test is customized to ensure it has a deterministic name.}staticStream<Arguments>randomArguments(){returnStream.of(Arguments.of("a string",ThreadLocalRandom.current().nextInt(10)+1),Arguments.of("another string",ThreadLocalRandom.current().nextInt(10)+1));}
Test session name DD_TEST_SESSION_NAME
Use DD_TEST_SESSION_NAME to define the name of the test session and the related group of tests. Examples of values for this tag would be:
unit-tests
integration-tests
smoke-tests
flaky-tests
ui-tests
backend-tests
If DD_TEST_SESSION_NAME is not specified, the default value used is a combination of the:
CI job name
Command used to run the tests (such as mvn test)
The test session name needs to be unique within a repository to help you distinguish different groups of tests.
When to use DD_TEST_SESSION_NAME
There’s a set of parameters that Datadog checks to establish correspondence between test sessions. The test command used to execute the tests is one of them. If the test command contains a string that changes for every execution, such as a temporary folder, Datadog considers the sessions to be unrelated to each other. For example:
mvn test --temp-dir=/var/folders/t1/rs2htfh55mz9px2j4prmpg_c0000gq/T
Datadog recommends using DD_TEST_SESSION_NAME if your test commands vary between executions.
Troubleshooting
The tests are not appearing in Datadog after enabling Test Optimization in the tracer
Verify that the tracer is injected into your build process by examining your build’s logs.
If the injection is successful, you can see a line containing DATADOG TRACER CONFIGURATION.
If the line is not there, make sure that the environment variables used to inject and configure the tracer are available to the build process.
A common mistake is to set the variables in a build step and run the tests in another build step. This approach may not work if the variables are not propagated between build steps.
Ensure that you are using the latest version of the tracer.
Ensure that the dd.civisibility.enabled property (or DD_CIVISIBILITY_ENABLED environment variable) is set to true in the tracer arguments.
Try running your build with tracer debug logging enabled by setting the DD_TRACE_DEBUG environment variable to true.
Check the build output for any errors that indicate tracer misconfiguration, such as an unset DD_API_KEY environment variable.
Tests or source code compilation fails when building a project with the tracer attached
By default, Test Optimization runs Java code compilation with a compiler plugin attached.
The plugin is optional, as it only serves to reduce the performance overhead.
Depending on the build configuration, adding the plugin can sometimes disrupt the compilation process.
If the plugin interferes with the build, disable it by adding dd.civisibility.compiler.plugin.auto.configuration.enabled=false to the list of -javaagent arguments
(or by setting DD_CIVISIBILITY_COMPILER_PLUGIN_AUTO_CONFIGURATION_ENABLED=false environment variable).
Builds fails because dd-javac-plugin-client artifact cannot be found
It is possible that the Java compiler plugin injected into the build is not available if the build uses a custom artifactory storage or if it is run in offline mode.
If this is the case, you can disable plugin injection by adding dd.civisibility.compiler.plugin.auto.configuration.enabled=false to the list of -javaagent arguments
(or by setting the DD_CIVISIBILITY_COMPILER_PLUGIN_AUTO_CONFIGURATION_ENABLED environment variable to false).
The plugin is optional, as it only serves to reduce the performance overhead.
Tests fail when building a project with the tracer attached
In some cases attaching the tracer can break tests, especially if they run asserts on the internal state of the JVM or instances of third-party libraries’ classes.
While the best approach is such cases is to update the tests, there is also a quicker option of disabling the tracer’s third-party library integrations.
The integrations provide additional insights into what happens in the tested code and are especially useful in integration tests, to monitor things like HTTP requests or database calls.
They are enabled by default.
To disable a specific integration, refer to the Datadog Tracer Compatibility table for the relevant configuration property names.
For example, to disable OkHttp3 client request integration, add dd.integration.okhttp-3.enabled=false to the list of -javaagent arguments.
To disable all integrations, augment the list of -javaagent arguments with dd.trace.enabled=false (or set DD_TRACE_ENABLED=false environment variable).