Try 'Introduction to OpenTelemetry with Datadog' in the Learning Center

Learn how to configure OpenTelemetry to export metrics, traces, and logs to Datadog, and explore the collected data within the platform.

ENROLL NOW

Overview

OpenTelemetry is an open source observability framework that provides IT teams with standardized protocols and tools for collecting and routing observability data from software applications. OpenTelemetry provides a consistent format for instrumentingInstrumentation is the process of adding code to your application to capture and report observability data to Datadog, such as traces, metrics, and logs., generating, gathering, and exporting application observability data—namely metrics, logs, and traces—to monitoring platforms for analysis and insight.

This guide demonstrates how to configure a sample OpenTelemetry application to send observability data to Datadog using the OpenTelemetry SDK, OpenTelemetry Collector, and Datadog Exporter. This guide also shows you how to explore this data in the Datadog UI.

Follow this guide to:

1. Instrument the application with the OpenTelemetry API.
2. Configure the application to send observability data to Datadog.
3. Correlate observability data with unified service tagging.
4. Run the application to generate observability data.
5. Explore observability data in the Datadog UI.

Prerequisites

To complete this guide, you need the following:

1. Create a Datadog account if you haven’t yet.
2. Set up your Datadog API key:
   a. Find or create your Datadog API key.
   b. Export your Datadog API key to an environment variable:
   
   Copy

   export DD_API_KEY=<Your API Key>

3. Get the sample Calendar application.
   a. Clone the opentelemetry-examples repository to your device:
   Copy

   git clone https://github.com/DataDog/opentelemetry-examples.git

   b. Navigate to the /calendar directory:
   Copy

   cd opentelemetry-examples/apps/rest-services/java/calendar

4. Install Docker Compose.
5. (Optional) Use Linux to send infrastructure metrics.

The Calendar application uses OpenTelemetry tools to generate and collect metrics, logs, and traces. The following steps explain how to get this observability data into Datadog.

Instrumenting the application

The Calendar sample application is already partially instrumented:

1. Go to the main CalendarService.java file located at: ./src/main/java/com/otel/service/CalendarService.java.

2. The following code instruments getDate() using the OpenTelemetry annotations and API:

   CalendarService.java

   @WithSpan(kind = SpanKind.CLIENT)
   public String getDate() {
       Span span = Span.current();
       span.setAttribute("peer.service", "random-date-service");
       ...
   }

When the Calendar application runs, the getDate() call generates traces and spans.

Configuring the application

OTLP Receiver

The Calendar application is already configured to send data from the OpenTelemetry SDK to the OpenTelemetry Protocol (OTLP) receiver in the OpenTelemetry Collector.

1. Go to the Collector configuration file located at: ./src/main/resources/otelcol-config.yaml.

2. The following lines configure the OTLP Receiver to receive metrics, traces, and logs:

   otelcol-config.yaml

   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
   ...
   service:
     pipelines:
       traces:
         receivers: [otlp]
       metrics:
         receivers: [otlp]
       logs:
         receivers: [otlp]

Datadog Exporter

The Datadog Exporter sends data collected by the OTLP Receiver to the Datadog backend.

1. Go to the otelcol-config.yaml file.

2. The following lines configure the Datadog Exporter to send observability data to Datadog:

   otelcol-config.yaml

   exporters:
     datadog:
       traces:
         span_name_as_resource_name: true
         trace_buffer: 500
       hostname: "otelcol-docker"
       api:
         key: ${DD_API_KEY}
         site: datadoghq.com
      
   connectors:
       datadog/connector:
   
   service:
     pipelines:
       metrics:
         receivers: [otlp, datadog/connector] # <- update this line
         exporters: [datadog]
       traces:
         exporters: [datadog, datadog/connector]
       logs:
         exporters: [datadog]

3. Set exporters.datadog.api.site to your Datadog site. Otherwise, it defaults to US1.

This configuration allows the Datadog Exporter to send runtime metrics, traces, and logs to Datadog. However, sending infrastructure metrics requires additional configuration.

OpenTelemetry Collector

In this example, configure your OpenTelemetry Collector to send infrastructure metrics.

To collect container metrics, configure the Docker stats receiver in your Datadog Exporter:

1. Add a docker_stats block to the receivers section of otel-config.yaml:

   otelcol-config.yaml

   receivers:
         otlp:
           protocols:
             grpc:
               endpoint: 0.0.0.0:4317
             http:
               endpoint: 0.0.0.0:4318
         # add the following block
         docker_stats: 
           endpoint: unix:///var/run/docker.sock # default; if this is not the Docker socket path, update to the correct path
           metrics:
             container.network.io.usage.rx_packets:
               enabled: true
             container.network.io.usage.tx_packets:
               enabled: true
             container.cpu.usage.system:
               enabled: true
             container.memory.rss:
               enabled: true
             container.blockio.io_serviced_recursive:
               enabled: true
             container.uptime:
               enabled: true
             container.memory.hierarchical_memory_limit:
               enabled: true

2. Update service.pipelines.metrics.receivers to include docker_stats:

   otelcol-config.yaml

   service:
     pipelines:
       metrics:
         receivers: [otlp, datadog/connector, docker_stats] # <- update this line

This configuration allows the Calendar application to send container metrics to Datadog for you to explore in Datadog.

Sending observability data with OTLP

The Calendar application uses the OpenTelemetry logging exporter in its Logback configuration to send logs with OpenTelemetry Layer Processor (OTLP).

1. Go to the Calendar application’s Logback XML configuration file at /src/main/resources/logback.xml.

2. The following lines define the OpenTelemetry appender:

   logback.xml

   <appender name="OpenTelemetry" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
       <immediateFlush>true</immediateFlush>
       <captureExperimentalAttributes>true</captureExperimentalAttributes>
       <captureKeyValuePairAttributes>true</captureKeyValuePairAttributes>
     </appender>

3. The <appender-ref ref="OpenTelemetry"/> line references the OpenTelemetry appender in the root level configuration:

   logback.xml

   <root level="INFO">
     <appender-ref ref="console"/>
     <appender-ref ref="OpenTelemetry"/>
   </root>

Additionally, environment variables configure the OpenTelemetry environment to export logs, metrics, and traces:

1. Go to the Calendar application’s Docker Compose file at ./deploys/docker/docker-compose-otel.yml.
2. The OTEL_LOGS_EXPORTER=otlp configuration allows the logs to be sent with OTLP.
3. The OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317 configuration allows the metrics and traces to be sent with OTLP.

Correlating observability data

Unified service tagging ties observability data together in Datadog so you can navigate across metrics, traces, and logs with consistent tags.

The Calendar application is already configured with unified service tagging:

1. Go to the Calendar application’s Docker Compose file at ./deploys/docker/docker-compose-otel.yml.

2. The following lines enable the correlation between application traces and other observability data:

   docker-compose-otel.yml

   environment:
     - OTEL_SERVICE_NAME=calendar-otel
     - OTEL_RESOURCE_ATTRIBUTES=deployment.environment=docker,host.name=otelcol-docker,service.version=<IMAGE_TAG>

Running the application

To start generating and forwarding observability data to Datadog, you need to run the Calendar application with the OpenTelemetry SDK:

1. Run the application from the calendar/ folder:

   Copy

   docker compose -f deploys/docker/docker-compose-otel.yml up

   This command creates a Docker container with the OpenTelemetry Collector and the Calendar service.

2. To test that the Calendar application is running correctly, execute the following command from another terminal window:

   Copy

   curl localhost:9090/calendar 

3. Verify that you receive a response like:

   Copy

   {"date":"2022-12-30"}

4. Run the curl command several times to ensure at least one trace exports to the Datadog backend.

   The Calendar application uses the probabilistic sampler processor, so only 30% of traces sent through the application reach the target backend.

Each call to the Calendar application results in metrics, traces, and logs being forwarded to the OpenTelemetry Collector, then to the Datadog Exporter, and finally to the Datadog backend.

Exploring observability data in Datadog

Use the Datadog UI to explore the Calendar application’s observability data.

Note: It may take a few minutes for your trace data to appear.

Runtime and infrastructure metrics

View runtime and infrastructure metrics to visualize, monitor, and measure the performance of your applications, hosts, containers, and processes.

1. Go to APM > Service Catalog.

2. Hover over the calendar-otel service and select Full Page.

3. Scroll to the bottom panel and select:

   • Infrastructure Metrics to see your Docker container metrics, such as CPU and memory usage.
   • JVM Metrics to see runtime metrics, such as heap usage and thread count.

Logs

View logs to monitor and troubleshoot application and system operations.

1. Go to Logs.
2. If you have other logs in the list, add @service.name:calendar-otel to the Search for field to only see logs from the Calendar application.
3. Select a log from the list to see more details.

Traces

View traces and spans to observe the status and performance of requests processed by your application.

1. Go to APM > Traces.

2. Find the Service section in the filter menu, and select the calendar-otel facet to display all calendar-otel traces:

3. Explore your calendar-otel traces.

   To start, click on a trace to open the trace side panel and find more details about the trace and its spans. For example, the Flame Graph captures how much time was spent in each component of the Calendar execution path:

4. Notice that you can select Infrastructure, Metrics, or Logs in the bottom panel to correlate your trace with other observability data.

Further reading

お役に立つドキュメント、リンクや記事:

OpenTelemetry DocumentationEXTERNAL SITE Datadog OpenTelemetry DocumentationDOCUMENTATION Custom Instrumentation with OpenTelemetryDOCUMENTATION Monitor runtime metrics from OTel-instrumented apps in Datadog APMBLOG Correlate Datadog RUM events with traces from OTel-instrumented applicationsBLOG Send metrics and traces from OpenTelemetry Collector to Datadog via Datadog ExporterBLOG Forward logs from the OpenTelemetry Collector with the Datadog ExporterBLOG