Getting Started with OpenTelemetry at Datadog

Overview

OpenTelemetry is an open source observability framework that provides IT teams with standardized protocols and tools for collecting and routing observability data from software applications. OpenTelemetry provides a consistent format for instrumenting, generating, gathering, and exporting application observability data—namely metrics, logs, and traces—to monitoring platforms for analysis and insight.

This guide demonstrates how to configure a sample OpenTelemetry application to send observability data to Datadog using the OpenTelemetry SDK, OpenTelemetry Collector, and Datadog Exporter. This guide also shows you how to explore this data in the Datadog UI.

Follow this guide to:

  1. Instrument the application with the OpenTelemetry API.
  2. Configure the application to send observability data to Datadog.
  3. Correlate observability data with unified service tagging.
  4. Run the application to generate observability data.
  5. Explore observability data in the Datadog UI.

Prerequisites

To complete this guide, you need the following:

  1. Create a Datadog account if you haven’t yet.
  2. Set up your Datadog API key:
    a. Find or create your Datadog API key.
    b. Export your Datadog API key to an environment variable:
    export DD_API_KEY=<Your API Key>
  3. Get the sample Calendar application.
    a. Clone the opentelemetry-examples repository to your device:
    git clone https://github.com/DataDog/opentelemetry-examples.git
    b. Navigate to the /calendar directory:
    cd opentelemetry-examples/apps/rest-services/java/calendar
  4. Install Docker Compose.
  5. (Optional) Use Linux to send infrastructure metrics.

The Calendar application uses OpenTelemetry tools to generate and collect metrics, logs, and traces. The following steps explain how to get this observability data into Datadog.

Instrumenting the application

The Calendar sample application is already partially instrumented:

  1. Go to the main CalendarController.java file located at: ./src/main/java/com/otel/controller/CalendarController.java.

  2. The following code instruments getDate() using the OpenTelemetry API:

    CalendarController.java

    private String getDate() {
  Span span = GlobalOpenTelemetry.getTracer("calendar").spanBuilder("getDate").startSpan();
  try (Scope scope = span.makeCurrent()) {
   ...
  } finally {
    span.end();
  }
}

When the Calendar application runs, the getDate() call generates traces and spans.

Configuring the application

OTLP Receiver

The Calendar application is already configured to send data from the OpenTelemetry SDK to the OpenTelemetry Protocol (OTLP) receiver in the OpenTelemetry Collector.

  1. Go to the Collector configuration file located at: ./src/main/resources/otelcol-config.yaml.

  2. The following lines configure the OTLP Receiver to receive metrics, traces, and logs:

    otelcol-config.yaml

    receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
...
service:
  pipelines:
    traces:
      receivers: [otlp]
    metrics:
      receivers: [otlp]
    logs:
      receivers: [otlp]

Datadog Exporter

The Datadog Exporter sends data collected by the OTLP Receiver to the Datadog backend.

  1. Go to the otelcol-config.yaml file.

  2. The following lines configure the Datadog Exporter to send observability data to Datadog:

    otelcol-config.yaml

    exporters:
  datadog:
    traces:
      span_name_as_resource_name: true
      trace_buffer: 500
    hostname: "otelcol-docker"
    api:
      key: ${DD_API_KEY}
      site: datadoghq.com
   
connectors:
    datadog/connector:

service:
  pipelines:
    metrics:
      receivers: [otlp, datadog/connector] # <- update this line
      exporters: [datadog]
    traces:
      exporters: [datadog, datadog/connector]
    logs:
      exporters: [datadog]

  3. Set exporters.datadog.api.site to your Datadog site. Otherwise, it defaults to US1.

This configuration allows the Datadog Exporter to send runtime metrics, traces, and logs to Datadog. However, sending infrastructure metrics requires additional configuration.

OpenTelemetry Collector

In this example, configure your OpenTelemetry Collector to send infrastructure metrics.

To send infrastructure metrics from the OpenTelemetry Collector to Datadog, you must use Linux. This is a limitation of the Docker Stats receiver.

To collect container metrics, configure the Docker stats receiver in your Datadog Exporter:

  1. Add a docker_stats block to the receivers section of otel-config.yaml:

    otelcol-config.yaml

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      # add the following block
      docker_stats: 
        endpoint: unix:///var/run/docker.sock # default; if this is not the Docker socket path, update to the correct path
        metrics:
          container.network.io.usage.rx_packets:
            enabled: true
          container.network.io.usage.tx_packets:
            enabled: true
          container.cpu.usage.system:
            enabled: true
          container.memory.rss:
            enabled: true
          container.blockio.io_serviced_recursive:
            enabled: true
          container.uptime:
            enabled: true
          container.memory.hierarchical_memory_limit:
            enabled: true

  2. Update service.pipelines.metrics.receivers to include docker_stats:

    otelcol-config.yaml

    service:
  pipelines:
    metrics:
      receivers: [otlp, datadog/connector, docker_stats] # <- update this line

This configuration allows the Calendar application to send container metrics to Datadog for you to explore in Datadog.

Sending observability data with OTLP

The Calendar application uses the OpenTelemetry logging exporter in its Logback configuration to send logs with OpenTelemetry Layer Processor (OTLP).

  1. Go to the Calendar application’s Logback XML configuration file at /src/main/resources/logback.xml.

  2. The following lines define the OpenTelemetry appender:

    logback.xml

    <appender name="OpenTelemetry" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
    <immediateFlush>true</immediateFlush>
    <captureExperimentalAttributes>true</captureExperimentalAttributes>
    <captureKeyValuePairAttributes>true</captureKeyValuePairAttributes>
  </appender>

  3. The <appender-ref ref="OpenTelemetry"/> line references the OpenTelemetry appender in the root level configuration:

    logback.xml

    <root level="INFO">
  <appender-ref ref="console"/>
  <appender-ref ref="OpenTelemetry"/>
</root>

Additionally, environment variables configure the OpenTelemetry environment to export logs, metrics, and traces:

  1. Go to the Calendar application’s Docker Compose file at ./deploys/docker/docker-compose-otel.yml.
  2. The OTEL_LOGS_EXPORTER=otlp configuration allows the logs to be sent with OTLP.
  3. The OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317 configuration allows the metrics and traces to be sent with OTLP.

Correlating observability data

Unified service tagging ties observability data together in Datadog so you can navigate across metrics, traces, and logs with consistent tags.

The Calendar application is already configured with unified service tagging:

  1. Go to the Calendar application’s Docker Compose file at ./deploys/docker/docker-compose-otel.yml.

  2. The following lines enable the correlation between application traces and other observability data:

    docker-compose-otel.yml

    environment:
  - OTEL_SERVICE_NAME=calendar-otel
  - OTEL_RESOURCE_ATTRIBUTES=deployment.environment=docker,host.name=otelcol-docker

Running the application

To start generating and forwarding observability data to Datadog, you need to run the Calendar application with the OpenTelemetry SDK:

  1. Run the application from the calendar/ folder:

    docker compose -f deploys/docker/docker-compose-otel.yml up
    This command creates a Docker container with the OpenTelemetry Collector and the Calendar service.

  2. To test that the Calendar application is running correctly, execute the following command from another terminal window:

    curl localhost:9090/calendar

  3. Verify that you receive a response like:

    {"date":"2022-12-30"}

  4. Run the curl command several times to ensure at least one trace exports to the Datadog backend.

    The Calendar application uses the probabilistic sampler processor, so only 30% of traces sent through the application reach the target backend.

Each call to the Calendar application results in metrics, traces, and logs being forwarded to the OpenTelemetry Collector, then to the Datadog Exporter, and finally to the Datadog backend.

Exploring observability data in Datadog

Use the Datadog UI to explore the Calendar application’s observability data.

Note: It may take a few minutes for your trace data to appear.

Runtime and infrastructure metrics

View runtime and infrastructure metrics to visualize, monitor, and measure the performance of your applications, hosts, containers, and processes.

  1. Go to APM > Service Catalog.

  2. Hover over the calendar-otel service and select Full Page.

  3. Scroll to the bottom panel and select:

    • Infrastructure Metrics to see your Docker container metrics, such as CPU and memory usage.
    • JVM Metrics to see runtime metrics, such as heap usage and thread count.
    View Infrastructure metrics and JVM Runtime metrics for the Calendar application

Logs

View logs to monitor and troubleshoot application and system operations.

  1. Go to Logs.
  2. If you have other logs in the list, add @service.name:calendar-otel to the Search for field to only see logs from the Calendar application.
  3. Select a log from the list to see more details.
View Logs for the Calendar application

Traces

View traces and spans to observe the status and performance of requests processed by your application.

  1. Go to APM > Traces.

  2. Find the Service section in the filter menu, and select the calendar-otel facet to display all calendar-otel traces:

    View Traces for the Calendar application

  3. Explore your calendar-otel traces.

    To start, click on a trace to open the trace side panel and find more details about the trace and its spans. For example, the Flame Graph captures how much time was spent in each component of the Calendar execution path:

    View the Flame Graph for a Calendar application trace

  4. Notice that you can select Infrastructure, Metrics, or Logs in the bottom panel to correlate your trace with other observability data.

    Correlate a Calendar application trace with logs

Further reading

Additional helpful documentation, links, and articles:

OpenTelemetry DocumentationOPENTELEMETRY more more Datadog OpenTelemetry DocumentationDOCUMENTATION more more Custom Instrumentation with OpenTelemetryDOCUMENTATION more more Monitor runtime metrics from OTel-instrumented apps in Datadog APMBLOG more more Correlate Datadog RUM events with traces from OTel-instrumented applicationsBLOG more more Send metrics and traces from OpenTelemetry Collector to Datadog via Datadog ExporterBLOG more more Forward logs from the OpenTelemetry Collector with the Datadog ExporterBLOG more more