Instrumentation is the process of adding code to your application to capture and report observability data. Automatic instrumentation is a way to instrument applications and libraries without modifying their source code. Both OpenTelemetry and Datadog provide automatic instrumentation in their SDKs.

Datadog SDKs support adding OpenTelemetry instrumentation libraries to their existing automatic instrumentation. This provides observability for libraries not covered by Datadog SDKs without changing SDKs.

Prerequisites

  1. Enable OpenTelemetry support: Set the DD_TRACE_OTEL_ENABLED environment variable to true. This step isn’t required for the Datadog Go and Ruby APM SDKs.

  2. Run the Datadog Agent: Datadog SDKs provide an implementation of the OpenTelemetry API and submit spans to a Datadog Agent. Ensure the Datadog Agent is running to use OpenTelemetry instrumentation with Datadog SDKs.

  3. Disable duplicate instrumentation: When replacing a Datadog instrumentation with its OpenTelemetry equivalent, disable the Datadog instrumentation to prevent duplicate spans from appearing in the trace.

Configuration

Datadog SDKs support configuration through OpenTelemetry environment variables.

Language support

Datadog SDKs implement the OpenTelemetry API by overriding the default implementations in the OpenTelemetry SDK. However, note the following limitations:

  • Operations only supported by the OpenTelemetry SDK are not supported (for example, SpanProcessors or OTLP Trace Exporters).
  • Datadog SDKs do not support OpenTelemetry Metrics and Logs APIs. To use OpenTelemetry Logs and Metrics APIs, use OTLP Ingest.
LanguageMinimum version
Java1.35.0
Python2.10.0
Ruby2.1.0
Go1.67.0
Node.js4.3.0
PHP0.94.0
.NET2.53.0

Compatibility requirements

The Datadog Java SDK supports library instrumentations using OpenTelemetry’s instrumentation API and javaagent extension API.

Each instrumentation must be packaged as an OpenTelemetry extension in its own JAR.

OpenTelemetry provides an example extension project that registers a custom instrumentation for Servlet 3 classes.

The Datadog SDK for Java also accepts select individual instrumentation JARs produced by OpenTelemetry’s opentelemetry-java-instrumentation build, for example the CFX instrumentation JAR.

OpenTelemetry incubator APIs are not supported.

Setup

To use an OpenTelemetry instrumentation with the Datadog Java SDK:

  1. Set the dd.trace.otel.enabled system property or the DD_TRACE_OTEL_ENABLED environment variable to true.
  2. Copy the OpenTelemetry extension JAR containing the instrumentation to the same container as the application.
  3. Set the otel.javaagent.extensions system property or the OTEL_JAVAAGENT_EXTENSIONS environment variable to the extension JAR path.

Verified OpenTelemetry extensions

FrameworkVersionsOpenTelemetry ExtensionInstrumentation Names
Apache CXF (Jax-WS)3.0+opentelemetry-javaagent-jaxws-2.0-cxf-3.0cxf

Compatibility requirements

The Datadog Python SDK supports library instrumentations using the OpenTelemetry Python Trace API.

OpenTelemetry provides an example for instrumenting a sample application.

Setup

To use OpenTelemetry instrumentations with the Datadog Python SDK, perform the following steps:

  1. Follow the instructions in the OpenTelemetry API section in the Datadog Python library docs.
  2. Follow the steps for instrumenting your service with your chosen opentelemetry-python-contrib library.

Compatibility requirements

The Datadog Ruby SDK supports library instrumentation using the OpenTelemetry Ruby Trace API.

OpenTelemetry provides an example for instrumenting a sample application.

Setup

To use OpenTelemetry integrations with the Datadog Ruby SDK, perform the following steps:

  1. Follow the instructions in configuring OpenTelemetry in the Datadog Ruby SDK documentation.
  2. Follow the steps for instrumenting your service with your chosen opentelemetry-ruby-contrib library.

Compatibility requirements

The Datadog SDK for Go supports library instrumentations written using the Opentelemetry-Go Trace API, including the opentelemetry-go-contrib/instrumentation libraries.

Setup

To use OpenTelemetry integrations with the Datadog Go SDK, perform the following steps:

  1. Follow the instructions in the Imports and Setup sections of the Go Custom Instrumentation using OpenTelemetry API page.
  2. Follow the steps for instrumenting your service with your chosen opentelemetry-go-contrib library.

The following is an example instrumenting the net/http library with the Datadog Tracer and Opentelemetry’s net/http integration:

import (
	"fmt"
	"log"
	"net/http"

	ddotel "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/opentelemetry"
	ddtracer "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
)

func main() {
	// register tracer
	provider := ddotel.NewTracerProvider(ddtracer.WithDebugMode(true))
	defer provider.Shutdown()
	otel.SetTracerProvider(provider)

	// configure the server with otelhttp instrumentation as you normally would using opentelemetry: https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
	var mux http.ServeMux
	mux.Handle("/hello", http.HandlerFunc(hello))
	http.HandleFunc("/hello", hello)
	log.Fatal(http.ListenAndServe(":8080", otelhttp.NewHandler(&mux, "server")))
}

func hello(w http.ResponseWriter, req *http.Request) {
	fmt.Fprintf(w, "hello\n")
}
go-dd-otelhttp

Compatibility requirements

The Datadog Node.js SDK supports library instrumentations using the OpenTelemetry Node.js Trace API.

Setup

To use OpenTelemetry instrumentations with the Datadog Node.js SDK, perform the following steps:

  1. Follow the Setup instructions in Node.js Custom Instrumentation using OpenTelemetry API.
  2. Follow the steps for instrumenting your service with your chosen opentelemetry-js-contrib library.

The following example demonstrates how to instrument the http and express OpenTelemetry integrations with the Datadog Node.js SDK:

const tracer = require('dd-trace').init()
const { TracerProvider } = tracer
const provider = new TracerProvider()
provider.register()

const { registerInstrumentations } = require('@opentelemetry/instrumentation')
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http')
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express')

// Register the instrumentation with the Datadog trace provider
// and the OpenTelemetry instrumentation of your choice
registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation({
      ignoreIncomingRequestHook (req) {
        // Ignore spans created from requests to the agent
        return req.path === '/v0.4/traces' || req.path === '/v0.7/config' ||
        req.path === '/telemetry/proxy/api/v2/apmtelemetry'
      },
      ignoreOutgoingRequestHook (req) {
        // Ignore spans created from requests to the agent
        return req.path === '/v0.4/traces' || req.path === '/v0.7/config' ||
        req.path === '/telemetry/proxy/api/v2/apmtelemetry'
      }
    }),
    new ExpressInstrumentation()
  ],
  tracerProvider: provider
})

const express = require('express')
const http = require('http')

// app code below ....

Configuration

To avoid duplicate spans, disable the corresponding Datadog instrumentations.

Set the DD_TRACE_DISABLED_INSTRUMENTATIONS environment variable to a comma-separated list of integration names to disable. For example, to disable Datadog instrumentations for the libraries used in the Setup example, set the following:

DD_TRACE_DISABLED_INSTRUMENTATIONS=http,dns,express,net

Compatibility requirements

The Datadog .NET SDK supports library instrumentations that come with built-in OpenTelemetry support.

Setup

To use Opentelemetry instrumentation libraries with the Datadog .NET SDK:

  1. Set the DD_TRACE_OTEL_ENABLED environment variable to true.
  2. Follow the steps to configure each library, if any, to generate OpenTelemetry-compatible instrumentation via ActivitySource

Verified OpenTelemetry Instrumentation Libraries

LibraryVersionsNuGet packageIntegration NameSetup instructions
Azure Service Bus7.14.0+Azure.Messaging.ServiceBusAzureServiceBusSee Azure SDK section below

Azure SDK

The Azure SDK provides built-in OpenTelemetry support. Enable it by setting the AZURE_EXPERIMENTAL_ENABLE_ACTIVITY_SOURCE environment variable to true or by setting the Azure.Experimental.EnableActivitySource context switch to true in your application code. See Azure SDK documentation for more details.

Further reading