Compatibility

Supported languages:

Supported test frameworks:

Supported test runners:

Configuring reporting method

To report test results to Datadog, you need to configure the datadog-ci gem:

Installing the Ruby test optimization library

To install the Ruby test optimization gem run:

bundle add datadog-ci --group "test"

Alternatively, add it to your Gemfile manually:

1. Add the datadog-ci gem to your Gemfile:

gem "datadog-ci", "~> 1.0", group: :test

2. Install the gem by running bundle install

Instrumenting your tests

Follow these steps if your CI Provider is not supported for auto-instrumentation (see Configuring reporting method).

1. Set the following environment variables to configure the tracer:

DD_CIVISIBILITY_ENABLED=true (Required)

Enables the Test Optimization product.

DD_TEST_SESSION_NAME

Use this to identify a group of tests (for example: unit-tests or integration-tests).

DD_ENV (Required)

Environment where the tests are being run (for example: local when running tests on a developer workstation or ci when running them on a CI provider).

DD_SERVICE (Optional)

Name of the service or library being tested.

2. Prepend your test command with this datadog-ci CLI wrapper:

bundle exec ddcirb exec bundle exec rake test

Alternatively, set RUBYOPT environment variable to "-rbundler/setup -rdatadog/ci/auto_instrument" and don’t modify your test command.

Adding custom tags to tests

You can add custom tags to your tests by using the current active test:

require "datadog/ci"

# inside your test
Datadog::CI.active_test&.set_tag("test_owner", "my_team")
# test continues normally
# ...

To create filters or group by fields for these tags, you must first create facets. For more information about adding tags, see the Adding Tags section of the Ruby custom instrumentation documentation.

Adding custom measures to tests

Like tags, you can add custom measures to your tests by using the current active test:

require "datadog/ci"

# inside your test
Datadog::CI.active_test&.set_metric("memory_allocations", 16)
# test continues normally
# ...

For more information on custom measures, see the Add Custom Measures Guide.

Configuration settings

The following is a list of the most important configuration settings that can be used with the test optimization library, either in code by using a Datadog.configure block, or using environment variables:

service

Name of the service or library under test.
Environment variable: DD_SERVICE
Default: $PROGRAM_NAME
Example: my-ruby-app

env

Name of the environment where tests are being run.
Environment variable: DD_ENV
Default: none
Examples: local, ci

For more information about service and env reserved tags, see Unified Service Tagging.

The following environment variable can be used to configure the location of the Datadog Agent:

DD_TRACE_AGENT_URL

Datadog Agent URL for trace collection in the form http://hostname:port.
Default: http://localhost:8126

All other Datadog Tracer configuration options can also be used.

Using additional instrumentation

It can be useful to have rich tracing information about your tests that includes time spent performing database operations or other external calls, as seen in the following flame graph:

To achieve this, configure additional instrumentation in your configure block:

if ENV["DD_ENV"] == "ci"
  Datadog.configure do |c|
    #  ... ci configs and instrumentation here ...
    c.tracing.instrument :redis
    c.tracing.instrument :pg
    # ... any other instrumentations supported by datadog gem ...
  end
end

Alternatively, you can enable automatic APM instrumentation in test_helper/spec_helper:

require "datadog/auto_instrument" if ENV["DD_ENV"] == "ci"

Note: In CI mode, these traces are submitted to Test Optimization, and they do not show up in Datadog APM.

For the full list of available instrumentation methods, see the tracing documentation

Collecting Git metadata

Datadog은 Git 정보를 사용하여 테스트 결과를 시각화하고 리포지토리, 브랜치, 커밋별로 그룹화합니다. Git 메타데이터는 CI 공급자 환경 변수와 프로젝트 경로의 로컬 .git 폴더(사용 가능한 경우)에서 테스트 계측으로 자동 수집합니다.

지원되지 않는 CI 공급자이거나 .git 폴더가 없는 상태에서 테스트를 실행하는 경우, 환경 변수를 사용하여 Git 정보를 수동으로 설정할 수 있습니다. 해당 환경 변수는 자동 탐지된 정보보다 우선합니다. 다음 환경 변수를 설정하여 Git 정보를 제공합니다.

DD_GIT_REPOSITORY_URL

코드가 저장된 리포지토리 URL입니다. HTTP, SSH URL이 모두 지원됩니다.
예시: git@github.com:MyCompany/MyApp.git, https://github.com/MyCompany/MyApp.git

DD_GIT_BRANCH

테스트 중인 Git 브랜치입니다. 대신 태그 정보를 제공하는 경우 비워 둡니다.
예시: develop

DD_GIT_TAG

테스트 중인 Git 태그입니다(해당되는 경우). 대신 브랜치 정보를 제공하는 경우 비워 둡니다.
예시: 1.0.1

DD_GIT_COMMIT_SHA

전체 커밋 해시입니다.
예시: a18ebf361cc831f5535e58ec4fae04ffd98d8152

DD_GIT_COMMIT_MESSAGE

커밋 메시지입니다.
예시: Set release number

DD_GIT_COMMIT_AUTHOR_NAME

커밋 작성자 이름입니다.
예시: John Smith

DD_GIT_COMMIT_AUTHOR_EMAIL

커밋 작성자 이메일입니다.
예시: john@example.com

DD_GIT_COMMIT_AUTHOR_DATE

ISO 8601 형식의 커밋 작성자 날짜입니다.
예시: 2021-03-12T16:00:28Z

DD_GIT_COMMIT_COMMITTER_NAME

커밋 커미터 이름입니다.
예시: Jane Smith

DD_GIT_COMMIT_COMMITTER_EMAIL

커밋 커미터 이메일입니다.
예시: jane@example.com

DD_GIT_COMMIT_COMMITTER_DATE

ISO 8601 형식의 커밋 커미터 날짜입니다.
예시: 2021-03-12T16:00:28Z

Manually instrumenting your tests

Auto-instrumentation adds additional performance overhead at the code loading stage. It can be especially noticeable for large repositories with a lot of dependencies. If your project takes 20+ seconds to start, you are likely to benefit from manually instrumenting your tests.

require "rspec"
require "datadog/ci"

# Only activates test instrumentation on CI
if ENV["DD_ENV"] == "ci"
  Datadog.configure do |c|
    # enables test optimization
    c.ci.enabled = true

    # The name of the service or library under test
    c.service = "my-ruby-app"

    # Enables the RSpec instrumentation
    c.ci.instrument :rspec
  end
end

DD_ENV=ci bundle exec rake spec

require "minitest"
require "datadog/ci"

# Only activates test instrumentation on CI
if ENV["DD_ENV"] == "ci"
  Datadog.configure do |c|
    # enables test optimization
    c.ci.enabled = true

    # The name of the service or library under test
    c.service = "my-ruby-app"

    c.ci.instrument :minitest
  end
end

DD_ENV=ci bundle exec rake test

require "datadog/ci"
require "minitest/autorun"

if ENV["DD_ENV"] == "ci"
  Datadog.configure do |c|
    c.ci.enabled = true

    c.service = "my-ruby-app"

    c.ci.instrument :minitest
  end
end

require "cucumber"
require "datadog/ci"

# Only activates test instrumentation on CI
if ENV["DD_ENV"] == "ci"
  Datadog.configure do |c|
    # enables test optimization
    c.ci.enabled = true

    # The name of the service or library under test
    c.service = "my-ruby-app"

    # Enables the Cucumber instrumentation
    c.ci.instrument :cucumber
  end
end

DD_ENV=ci bundle exec rake cucumber

Using library’s public API for unsupported test frameworks

If you use RSpec, Minitest, or Cucumber, do not use the manual testing API, as Test Optimization automatically instruments them and sends the test results to Datadog. The manual testing API is incompatible with already supported testing frameworks.

Use the manual testing API only if you use an unsupported testing framework or have a different testing mechanism. Full public API documentation is available on YARD site.

Domain model

The API is based around four concepts: test session, test module, test suite, and test.

Test session

A test session represents a test command run.

To start a test session, call Datadog::CI.start_test_session and pass the Datadog service and tags (such as the test framework you are using).

When all your tests have finished, call Datadog::CI::TestSession#finish, which closes the session and sends the session trace to the backend.

Test module

A test module represents a smaller unit of work within a session. For supported test frameworks, test module is always same as test session. For your use case, this could be a package in your componentized application.

To start a test module, call Datadog::CI.start_test_module and pass the name of the module.

When the module run has finished, call Datadog::CI::TestModule#finish.

Test suite

A test suite comprises a set of tests that test similar functionality. A single suite usually corresponds to a single file where tests are defined.

Create test suites by calling Datadog::CI#start_test_suite and passing the name of the test suite.

Call Datadog::CI::TestSuite#finish when all the related tests in the suite have finished their execution.

Test

A test represents a single test case that is executed as part of a test suite. Usually it corresponds to a method that contains testing logic.

Create tests in a suite by calling Datadog::CI#start_test or Datadog::CI.trace_test and passing the name of the test and name of the test suite. Test suite name must be the same as name of the test suite started in previous step.

Call Datadog::CI::Test#finish when a test has finished execution.

Code example

The following code represents example usage of the API:

require "datadog/ci"

Datadog.configure do |c|
  c.service = "my-test-service"
  c.ci.enabled = true
end

def run_test_suite(tests, test_suite_name)
  test_suite = Datadog::CI.start_test_suite(test_suite_name)

  run_tests(tests, test_suite_name)

  test_suite.passed!
  test_suite.finish
end

def run_tests(tests, test_suite_name)
  tests.each do |test_name|
    Datadog::CI.trace_test(test_name, test_suite_name) do |test|
      test.passed!
    end
  end
end

Datadog::CI.start_test_session(
  tags: {
    Datadog::CI::Ext::Test::TAG_FRAMEWORK => "my-framework",
    Datadog::CI::Ext::Test::TAG_FRAMEWORK_VERSION => "0.0.1",
  }
)
Datadog::CI.start_test_module("my-test-module")

run_test_suite(["test1", "test2", "test3"], "test-suite-name")

Datadog::CI.active_test_module&.passed!
Datadog::CI.active_test_module&.finish

Datadog::CI.active_test_session&.passed!
Datadog::CI.active_test_session&.finish

Best practices

Test session name DD_TEST_SESSION_NAME

Use DD_TEST_SESSION_NAME to define the name of the test session and the related group of tests. Examples of values for this tag would be:

• unit-tests
• integration-tests
• smoke-tests
• flaky-tests
• ui-tests
• backend-tests

If DD_TEST_SESSION_NAME is not specified, the default value used is a combination of the:

• CI job name
• Command used to run the tests (such as yarn test)

The test session name needs to be unique within a repository to help you distinguish different groups of tests.

When to use DD_TEST_SESSION_NAME

There’s a set of parameters that Datadog checks to establish correspondence between test sessions. The test command used to execute the tests is one of them. If the test command contains a string that changes for every execution, such as a temporary folder, Datadog considers the sessions to be unrelated to each other. For example:

• yarn test --temp-dir=/var/folders/t1/rs2htfh55mz9px2j4prmpg_c0000gq/T
• pnpm vitest --temp-dir=/var/folders/t1/rs2htfh55mz9px2j4prmpg_c0000gq/T

Datadog recommends using DD_TEST_SESSION_NAME if your test commands vary between executions.

Further reading

추가 유용한 문서, 링크 및 기사:

Forwarding Environment Variables for Tests in ContainersDOCUMENTATION Explore Test Results and PerformanceDOCUMENTATION Troubleshooting Test OptimizationDOCUMENTATION