OTLP Ingestion by the Datadog Agent

OTLP Ingest in the Agent is a way to send telemetry data directly from applications instrumented with OpenTelemetry SDKs to Datadog Agent. Since versions 6.32.0 and 7.32.0, the Datadog Agent can ingest OTLP traces and OTLP metrics through gRPC or HTTP.

OTLP Ingest in the Agent allows you to use observability features in the Datadog Agent. Because the application is instrumented with OpenTelemetry SDK, some Datadog library specific features aren’t available for the ingested data including Application Security Management, Continuous Profiler, Runtime Metrics, and Ingestion Rules.

To get started, you first instrument your application with OpenTelemetry SDKs. Then, export the telemetry data in OTLP format to the Datadog Agent. Configuring this varies depending on the kind of infrastructure your service is deployed on, as described on the page below. Although the aim is to be compatible with the latest OTLP version, the OTLP Ingest in the Agent is not compatible with all OTLP versions. Verify OTLP version compatibility through the Agent changelog.

Read the OpenTelemetry instrumentation documentation to understand how to point your instrumentation to the Agent. The receiver section described below follows the OpenTelemetry Collector OTLP receiver configuration schema.

OTel SDKs/Libraries, Datadog Trace Library, Datadog Integrations -> Datadog Agent -> Datadog
Note: The supported setup is an ingesting Agent deployed on every OpenTelemetry-data generating host. You cannot send OpenTelemetry telemetry from collectors or instrumented apps running one host to an Agent on a different host. But, provided the Agent is local to the collector or SDK instrumented app, you can set up multiple pipelines.

Enabling OTLP Ingestion on the Datadog Agent

OTLP ingestion is off by default, and you can turn it on by updating your datadog.yaml file configuration or by setting environment variables. The following datadog.yaml configurations enable endpoints on the default ports.

For gRPC, default port 4317:

otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: localhost:4317

For HTTP, default port 4318:

otlp_config:
  receiver:
    protocols:
      http:
        endpoint: localhost:4318

Alternatively, configure the endpoints by providing the port through the environment variables:

  • For gRPC (localhost:4317): DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT
  • For HTTP (localhost:4318): DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT

These must be passed to both the core Agent and trace Agent processes. If running in a containerized environment, use 0.0.0.0 instead of localhost to ensure the server is available on non-local interfaces.

Configure either gRPC or HTTP for this feature. Here is an example application that shows configuration for both.

  1. Follow the Datadog Docker Agent setup.

  2. For the Datadog Agent container, set the following endpoint environment variables and expose the corresponding port:

    • For gPRC: Set DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT to 0.0.0.0:4317 and expose port 4317.
    • For HTTP: Set DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT to 0.0.0.0:4318 and expose port 4318.
  1. Follow the Kubernetes Agent setup.

  2. Configure the following environment variables in both the trace Agent container and the core Agent container:

    For gPRC:

    name: DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT # enables gRPC receiver on port 4317
    value: "0.0.0.0:4317"
    

    For HTTP:

    name: DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT # enables HTTP receiver on port 4318
    value: "0.0.0.0:4318"
    
  3. Map the container ports 4317 or 4318 to the host port for the core Agent container:

    For gPRC:

    ports:
      - containerPort: 4317
        hostPort: 4317
        name: traceportgrpc
        protocol: TCP
    

    For HTTP

    ports:
      - containerPort: 4318
        hostPort: 4318
        name: traceporthttp
        protocol: TCP
    
  1. Follow the Kubernetes Agent setup.

  2. Enable the OTLP endpoints in the Agent by editing the datadog.otlp section of the values.yaml file:

    For gRPC:

    otlp:
     receiver:
       protocols:
         grpc:
           enabled: true
    

    For HTTP:

    otlp:
     receiver:
       protocols:
         http:
           enabled: true
    

This enables each protocol in the default port (4317 for OTLP/gRPC and 4318 for OTLP/HTTP).

  1. Follow the Kubernetes Agent setup.

  2. Enable the preferred protocol:

    For gRPC:

    --set "datadog.otlp.receiver.protocols.grpc.enabled=true"
    

    For HTTP:

    --set "datadog.otlp.receiver.protocols.http.enabled=true"
    

This enables each protocol in the default port (4317 for OTLP/gRPC and 4318 for OTLP/HTTP).

There are many other environment variables and settings supported in the Datadog Agent. To get an overview of them all, see the configuration template.

Sending OpenTelemetry traces and metrics to Datadog Agent

  1. For the application container, set OTEL_EXPORTER_OTLP_ENDPOINT environment variable to point to the Datadog Agent container. For example:

    OTEL_EXPORTER_OTLP_ENDPOINT=http://<datadog-agent>:4318.
    
  2. Both containers must be defined in the same bridge network, which is handled automatically if you use Docker Compose. Otherwise, follow the Docker example in Tracing Docker Applications to set up a bridge network with the correct ports.

  1. In the application deployment file, configure the endpoint that the OpenTelemetry client sends traces to with the OTEL_EXPORTER_OTLP_ENDPOINT environment variable:

    For gPRC:

    env:
     - name: HOST_IP
       valueFrom:
         fieldRef:
           fieldPath: status.hostIP
     - name: OTEL_EXPORTER_OTLP_ENDPOINT
       value: "http://$(HOST_IP):4317" # sends to gRPC receiver on port 4317
    

    For HTTP:

    env:
     - name: HOST_IP
       valueFrom:
         fieldRef:
           fieldPath: status.hostIP
     - name: OTEL_EXPORTER_OTLP_ENDPOINT
       value: "http://$(HOST_IP):4318" # sends to HTTP receiver on port 4318
    
Check the documentation of your OTLP Library. Some of them must send traces to /v1/traces instead of the / root path.

Out-of-the-box dashboards

Datadog provides out-of-the-box dashboards that you can copy and customize. To use Datadog’s out-of-the-box OpenTelemetry dashboards:

  1. Install the OpenTelemetry integration.

  2. Go to Dashboards > Dashboards list and search for opentelemetry:

    The Dashboards list, showing two OpenTelemetry out-of-the-box dashboards: Host Metrics and Collector Metrics.

The Host Metrics dashboard is for data collected from the host metrics receiver. The Collector Metrics dashboard is for any other types of metrics collected, depending on which metrics receiver you choose to enable.

Further Reading

Additional helpful documentation, links, and articles: