Install the Datadog Distribution of OTel Collector

Docs > OpenTelemetry in Datadog > Send Data to Datadog > Datadog Distribution of OpenTelemetry Collector > Install the Datadog Distribution of OTel Collector

Overview

Follow this guide to install the Datadog Distribution of OpenTelemetry (DDOT) Collector using Helm.

Need additional OpenTelemetry components? If you need components beyond those included in the default package, follow Use Custom OpenTelemetry Components to extend the Datadog Agent's capabilities. For a list of components included by default, see OpenTelemetry Collector components.

Requirements

To complete this guide, you need the following:

Datadog account:

Create a Datadog account if you don’t have one.
Find or create your Datadog API key.
Find or create your Datadog application key.

Software: Install and set up the following on your machine:

A Kubernetes cluster (v1.29+)
- Note: EKS Fargate environments are not supported
Helm (v3+)
Docker
kubectl

Install the Datadog Agent with OpenTelemetry Collector

Select installation method

Choose one of the following installation methods:

Datadog Operator: A Kubernetes-native approach that automatically reconciles and maintains your Datadog setup. It reports deployment status, health, and errors in its Custom Resource status, and it limits the risk of misconfiguration thanks to higher-level configuration options.
Helm chart: A straightforward way to deploy Datadog Agent. It provides versioning, rollback, and templating capabilities, making deployments consistent and easier to replicate.

Install the Datadog Operator

You can install the Datadog Operator in your cluster using the Datadog Operator Helm chart:

helm repo add datadog https://helm.datadoghq.com
helm repo update
helm install datadog-operator datadog/datadog-operator

Add the Datadog Helm Repository

To add the Datadog repository to your Helm repositories:

helm repo add datadog https://helm.datadoghq.com
helm repo update

Set up Datadog API and application keys

Get the Datadog API and application keys.

Store the keys as a Kubernetes secret:

kubectl create secret generic datadog-secret \
  --from-literal api-key=<DD_API_KEY> \
  --from-literal app-key=<DD_APP_KEY>

Replace <DD_API_KEY> and <DD_APP_KEY> with your actual Datadog API and application keys.

Configure the Datadog Agent

After deploying the Datadog Operator, create the DatadogAgent resource that triggers the deployment of the Datadog Agent, Cluster Agent and Cluster Checks Runners (if used) in your Kubernetes cluster. The Datadog Agent deploys as a DaemonSet, running a pod on every node of your cluster.

Use the datadog-agent.yaml file to specify your DatadogAgent deployment configuration.

datadog-agent.yaml

   apiVersion: datadoghq.com/v2alpha1
   kind: DatadogAgent
   metadata:
     name: datadog
   spec:
     global:
       clusterName: <CLUSTER_NAME>
       site: <DATADOG_SITE>
       credentials:
         apiSecret:
           secretName: datadog-secret
           keyName: api-key
         appSecret:
           secretName: datadog-secret
           keyName: app-key

Replace <CLUSTER_NAME> with a name for your cluster.
Replace <DATADOG_SITE> with your Datadog site. Your site is . (Ensure the correct DATADOG SITE is selected on the right.)

Use the Datadog Agent image tag with embedded DDOT Collector:

datadog-agent.yaml

  ...
  override:
    # Node Agent configuration
    nodeAgent:
      image:
        name: "gcr.io/datadoghq/agent:7.65.0-full"
        pullPolicy: Always

By default, the Agent image is pulled from Google Artifact Registry (gcr.io/datadoghq). If Artifact Registry is not accessible in your deployment region, use another registry.

Enable the OpenTelemetry Collector:

datadog-agent.yaml

  ...
  override:
    # Node Agent configuration
    nodeAgent:
      image:
        name: "gcr.io/datadoghq/agent:7.65.0-full"
        pullPolicy: Always
      containers:
        otel-agent:
          env:
            - name: DD_OTELCOLLECTOR_ENABLED
              value: "true"
  ...
  # Enable Features
  features:
    otelCollector:
      enabled: true

The Datadog Operator automatically binds the OpenTelemetry Collector to ports 4317 (named otel-grpc) and 4318 (named otel-http) by default.

(Optional) Enable additional Datadog features:

Enabling these features may incur additional charges. Review the pricing page and talk to your Customer Success Manager before proceeding.

datadog-agent.yaml

  # Enable Features
  features:
  ...
    apm:
      enabled: true
    orchestratorExplorer:
      enabled: true
    processDiscovery:
      enabled: true
    liveProcessCollection:
      enabled: true
    usm:
      enabled: true
    clusterChecks:
      enabled: true

When enabling additional Datadog features, always use the Datadog or OpenTelemetry Collector configuration files instead of relying on Datadog environment variables.

Use a YAML file to specify the Helm chart parameters for the Datadog Agent chart.

Create an empty datadog-values.yaml file:

touch datadog-values.yaml

Unspecified parameters use defaults from values.yaml.

Configure the Datadog API and application key secrets:

datadog-values.yaml

datadog:
  site: <DATADOG_SITE>
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret

Set <DATADOG_SITE> to your Datadog site. Otherwise, it defaults to datadoghq.com, the US1 site.

Use the Datadog Agent image tag with embedded DDOT Collector:

datadog-values.yaml

agents:
  image:
    repository: gcr.io/datadoghq/agent
    tag: 7.65.0-full
    doNotCheckTag: true
...

By default, the Agent image is pulled from Google Artifact Registry (gcr.io/datadoghq). If Artifact Registry is not accessible in your deployment region, use another registry.

Enable the OpenTelemetry Collector and configure the essential ports:

datadog-values.yaml

datadog:
  ...
  otelCollector:
    enabled: true
    ports:
      - containerPort: "4317" # default port for OpenTelemetry gRPC receiver.
        hostPort: "4317"
        name: otel-grpc
      - containerPort: "4318" # default port for OpenTelemetry HTTP receiver
        hostPort: "4318"
        name: otel-http

Set the hostPort to expose the container port to the external network. This enables configuring the OTLP exporter to point to the IP address of the node where the Datadog Agent is assigned.

If you don’t want to expose the port, you can use the Agent service instead:

Remove the hostPort entries from your datadog-values.yaml file.

In your application’s deployment file (deployment.yaml), configure the OTLP exporter to use the Agent service:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: 'http://<SERVICE_NAME>.<SERVICE_NAMESPACE>.svc.cluster.local'
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: 'grpc'

(Optional) Enable additional Datadog features:

Enabling these features may incur additional charges. Review the pricing page and talk to your Customer Success Manager before proceeding.

datadog-values.yaml

datadog:
  ...
  apm:
    portEnabled: true
    peer_tags_aggregation: true
    compute_stats_by_span_kind: true
    peer_service_aggregation: true
  orchestratorExplorer:
    enabled: true
  processAgent:
    enabled: true
    processCollection: true

When enabling additional Datadog features, always use the Datadog or OpenTelemetry Collector configuration files instead of relying on Datadog environment variables.

(Optional) Collect pod labels and use them as tags to attach to metrics, traces, and logs:

Custom metrics may impact billing. See the custom metrics billing page for more information.

datadog-values.yaml

datadog:
  ...
  podLabelsAsTags:
    app: kube_app
    release: helm_release

Completed datadog-values.yaml file

Your datadog-values.yaml file should look something like this:

datadog-values.yaml

agents:
  image:
    repository: gcr.io/datadoghq/agent
    tag: 7.65.0-full
    doNotCheckTag: true

datadog:
  site: datadoghq.com
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret

  otelCollector:
    enabled: true
    ports:
      - containerPort: "4317"
        hostPort: "4317"
        name: otel-grpc
      - containerPort: "4318"
        hostPort: "4318"
        name: otel-http
  apm:
    portEnabled: true
    peer_tags_aggregation: true
    compute_stats_by_span_kind: true
    peer_service_aggregation: true
  orchestratorExplorer:
    enabled: true
  processAgent:
    enabled: true
    processCollection: true

  podLabelsAsTags:
    app: kube_app
    release: helm_release
   

Configure the OpenTelemetry Collector

The Datadog Operator provides a sample OpenTelemetry Collector configuration that you can use as a starting point. If you need to modify this configuration, the Datadog Operator supports two ways of providing a custom Collector configuration:

Inline configuration: Add your custom Collector configuration directly in the features.otelCollector.conf.configData field.
ConfigMap-based configuration: Store your Collector configuration in a ConfigMap and reference it in the features.otelCollector.conf.configMap field. This approach allows you to keep Collector configuration decoupled from the DatadogAgent resource.

Inline Collector configuration

In the snippet below, the Collector configuration is placed directly under the features.otelCollector.conf.configData parameter:

datadog-agent.yaml

  ...
  # Enable Features
  features:
    otelCollector:
      enabled: true
      ports:
        - containerPort: 4317
          hostPort: 4317
          name: otel-grpc
        - containerPort: 4318
          hostPort: 4318
          name: otel-http
      conf:
        configData: |-
          receivers:
            prometheus:
              config:
                scrape_configs:
                  - job_name: "otelcol"
                    scrape_interval: 10s
                    static_configs:
                      - targets:
                          - 0.0.0.0:8888
            otlp:
              protocols:
                grpc:
                  endpoint: 0.0.0.0:4317
                http:
                  endpoint: 0.0.0.0:4318
          exporters:
            debug:
              verbosity: detailed
            datadog:
              api:
                key: ${env:DD_API_KEY}
                site: ${env:DD_SITE}
          processors:
            infraattributes:
              cardinality: 2
            batch:
              timeout: 10s
          connectors:
            datadog/connector:
              traces:
          service:
            pipelines:
              traces:
                receivers: [otlp]
                processors: [infraattributes, batch]
                exporters: [debug, datadog, datadog/connector]
              metrics:
                receivers: [otlp, datadog/connector, prometheus]
                processors: [infraattributes, batch]
                exporters: [debug, datadog]
              logs:
                receivers: [otlp]
                processors: [infraattributes, batch]
                exporters: [debug, datadog]          

When you apply the datadog-agent.yaml file containing this DatadogAgent resource, the Operator automatically mounts the Collector configuration into the Agent DaemonSet.

Completed datadog-agent.yaml file with inlined Collector config

Completed datadog-agent.yaml with inline Collector configuration should look something like this:

datadog-agent.yaml

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    clusterName: <CLUSTER_NAME>
    site: <DATADOG_SITE>
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
      appSecret:
        secretName: datadog-secret
        keyName: app-key

  override:
    # Node Agent configuration
    nodeAgent:
      image:
        name: "gcr.io/datadoghq/agent:7.65.0-full"
        pullPolicy: Always
      containers:
        otel-agent:
          env:
            - name: DD_OTELCOLLECTOR_ENABLED
              value: "true"

  # Enable Features
  features:
    apm:
      enabled: true
    orchestratorExplorer:
      enabled: true
    processDiscovery:
      enabled: true
    liveProcessCollection:
      enabled: true
    usm:
      enabled: true
    clusterChecks:
      enabled: true
    otelCollector:
      enabled: true
      ports:
        - containerPort: 4317
          hostPort: 4317
          name: otel-grpc
        - containerPort: 4318
          hostPort: 4318
          name: otel-http
      conf:
        configData: |-
          receivers:
            prometheus:
              config:
                scrape_configs:
                  - job_name: "datadog-agent"
                    scrape_interval: 10s
                    static_configs:
                      - targets:
                          - 0.0.0.0:8888
            otlp:
              protocols:
                grpc:
                  endpoint: 0.0.0.0:4317
                http:
                  endpoint: 0.0.0.0:4318
          exporters:
            debug:
              verbosity: detailed
            datadog:
              api:
                key: ${env:DD_API_KEY}
                site: ${env:DD_SITE}
          processors:
            infraattributes:
              cardinality: 2
            batch:
              timeout: 10s
          connectors:
            datadog/connector:
              traces:
          service:
            pipelines:
              traces:
                receivers: [otlp]
                processors: [infraattributes, batch]
                exporters: [debug, datadog, datadog/connector]
              metrics:
                receivers: [otlp, datadog/connector, prometheus]
                processors: [infraattributes, batch]
                exporters: [debug, datadog]
              logs:
                receivers: [otlp]
                processors: [infraattributes, batch]
                exporters: [debug, datadog]          

ConfigMap-based Collector Configuration

For more complex or frequently updated configurations, storing Collector configuration in a ConfigMap can simplify version control.

Create a ConfigMap that contains your Collector configuration:

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-config-map
  namespace: system
data:
  # must be named otel-config.yaml
  otel-config.yaml: |-
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: "datadog-agent"
              scrape_interval: 10s
              static_configs:
                - targets:
                    - 0.0.0.0:8888
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      debug:
        verbosity: detailed
      datadog:
        api:
          key: ${env:DD_API_KEY}
          site: ${env:DD_SITE}
    processors:
      infraattributes:
        cardinality: 2
      batch:
        timeout: 10s
    connectors:
      datadog/connector:
        traces:
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [infraattributes, batch]
          exporters: [debug, datadog, datadog/connector]
        metrics:
          receivers: [otlp, datadog/connector, prometheus]
          processors: [infraattributes, batch]
          exporters: [debug, datadog]
        logs:
          receivers: [otlp]
          processors: [infraattributes, batch]
          exporters: [debug, datadog]    

The field for Collector config in the ConfigMap must be called otel-config.yaml.

Reference the otel-agent-config-map ConfigMap in your DatadogAgent resource using features.otelCollector.conf.configMap parameter:

datadog-agent.yaml

  ...
  # Enable Features
  features:
    otelCollector:
      enabled: true
      ports:
        - containerPort: 4317
          hostPort: 4317
          name: otel-grpc
        - containerPort: 4318
          hostPort: 4318
          name: otel-http
      conf:
        configMap:
          name: otel-agent-config-map

The Operator automatically mounts otel-config.yaml from the ConfigMap into the Agent’s OpenTelemetry Collector DaemonSet.

Completed datadog-agent.yaml file with Collector config in the ConfigMap

Completed datadog-agent.yaml with Collector configuration defined as ConfigMap should look something like this:

datadog-agent.yaml

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    clusterName: <CLUSTER_NAME>
    site: <DATADOG_SITE>
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
      appSecret:
        secretName: datadog-secret
        keyName: app-key

  override:
    # Node Agent configuration
    nodeAgent:
      image:
        name: "gcr.io/datadoghq/agent:7.65.0-full"
        pullPolicy: Always

  # Enable Features
  features:
    apm:
      enabled: true
    orchestratorExplorer:
      enabled: true
    processDiscovery:
      enabled: true
    liveProcessCollection:
      enabled: true
    usm:
      enabled: true
    clusterChecks:
      enabled: true
    otelCollector:
      enabled: true
      ports:
        - containerPort: 4317
          hostPort: 4317
          name: otel-grpc
        - containerPort: 4318
          hostPort: 4318
          name: otel-http
      conf:
        configMap:
          name: otel-agent-config-map
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-config-map
  namespace: system
data:
  # must be named otel-config.yaml
  otel-config.yaml: |-
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: "datadog-agent"
              scrape_interval: 10s
              static_configs:
                - targets:
                    - 0.0.0.0:8888
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      debug:
        verbosity: detailed
      datadog:
        api:
          key: ${env:DD_API_KEY}
          site: ${env:DD_SITE}
    processors:
      infraattributes:
        cardinality: 2
      batch:
        timeout: 10s
    connectors:
      datadog/connector:
        traces:
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [infraattributes, batch]
          exporters: [debug, datadog, datadog/connector]
        metrics:
          receivers: [otlp, datadog/connector, prometheus]
          processors: [infraattributes, batch]
          exporters: [debug, datadog]
        logs:
          receivers: [otlp]
          processors: [infraattributes, batch]
          exporters: [debug, datadog]    

The Datadog Helm chart provides a sample OpenTelemetry Collector configuration that you can use as a starting point. This section walks you through the predefined pipelines and included OpenTelemetry components.

This is the full OpenTelemetry Collector configuration in otel-config.yaml:

otel-config.yaml

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "otelcol"
          scrape_interval: 10s
          static_configs:
            - targets: ["0.0.0.0:8888"]
  otlp:
    protocols:
      grpc:
         endpoint: 0.0.0.0:4317
      http:
         endpoint: 0.0.0.0:4318
exporters:
  debug:
    verbosity: detailed
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: ${env:DD_SITE}
processors:
  infraattributes:
    cardinality: 2
  batch:
    timeout: 10s
connectors:
  datadog/connector:
    traces:
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [infraattributes, batch]
      exporters: [datadog, datadog/connector]
    metrics:
      receivers: [otlp, datadog/connector, prometheus]
      processors: [infraattributes, batch]
      exporters: [datadog]
    logs:
      receivers: [otlp]
      processors: [infraattributes, batch]
      exporters: [datadog]

Key components

To send telemetry data to Datadog, the following components are defined in the configuration:

Diagram depicting the Agent deployment pattern

Datadog connector

The Datadog connector computes Datadog APM trace metrics.

otel-config.yaml

connectors:
  datadog/connector:
    traces:

Datadog exporter

The Datadog exporter exports traces, metrics, and logs to Datadog.

otel-config.yaml

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: ${env:DD_SITE}

Note: If key is not specified or set to a secret, or if site is not specified, the system uses values from the core Agent configuration. By default, the core Agent sets site to datadoghq.com (US1).

Prometheus receiver

The Prometheus receiver collects health metrics from the OpenTelemetry Collector for the metrics pipeline.

otel-config.yaml

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "otelcol"
          scrape_interval: 10s
          static_configs:
            - targets: ["0.0.0.0:8888"]

For more information, see the Collector Health Metrics documentation.

Deploy the Agent with the OpenTelemetry Collector

Deploy the Datadog Agent with the configuration file:

kubectl apply -f datadog-agent.yaml

This deploys the Datadog Agent as a DaemonSet with the DDOT OpenTelemetry Collector. The Collector runs on the same host as your application, following the Agent deployment pattern. The Gateway deployment pattern is not supported.

To install or upgrade the Datadog Agent with OpenTelemetry Collector in your Kubernetes environment, use one of the following Helm commands:

For default OpenTelemetry Collector configuration:

helm upgrade -i <RELEASE_NAME> datadog/datadog -f datadog-values.yaml

For custom OpenTelemetry Collector configuration:

helm upgrade -i <RELEASE_NAME> datadog/datadog \
  -f datadog-values.yaml \
  --set-file datadog.otelCollector.config=otel-config.yaml

This command allows you to specify your own otel-config.yaml file.

Replace <RELEASE_NAME> with the Helm release name you are using.

You may see warnings during the deployment process. These warnings can be ignored.

This Helm chart deploys the Datadog Agent with OpenTelemetry Collector as a DaemonSet. The Collector is deployed on the same host as your application, following the Agent deployment pattern. The Gateway deployment pattern is not supported.

Deployment diagram

Send your telemetry to Datadog

To send your telemetry data to Datadog:

Instrument your application
Configure the application
Correlate observability data
Run your application

Instrument the application

Instrument your application using the OpenTelemetry API.

Example application instrumented with the OpenTelemetry API

As an example, you can use the Calendar sample application that’s already instrumented for you. The following code instruments the CalendarService.getDate() method using the OpenTelemetry annotations and API:

CalendarService.java

@WithSpan(kind = SpanKind.CLIENT)
public String getDate() {
    Span span = Span.current();
    span.setAttribute("peer.service", "random-date-service");
    ...
}

Configure the application

Your application container must send data to the DDOT Collector on the same host. Since the Collector runs as a DaemonSet, you need to specify the local host as the OTLP endpoint.

If the OTEL_EXPORTER_OTLP_ENDPOINT environment variable is not already set, add it to your application’s Deployment manifest file:

deployment.yaml

env:
  ...
  - name: HOST_IP
    valueFrom:
     fieldRef:
        fieldPath: status.hostIP
  - name: OTLP_GRPC_PORT
    value: "4317"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: 'http://$(HOST_IP):$(OTLP_GRPC_PORT)'
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: 'grpc'
   

Correlate observability data

Unified service tagging ties observability data together in Datadog so you can navigate across metrics, traces, and logs with consistent tags.

In containerized environments, env, service, and version are set through the OpenTelemetry Resource Attributes environment variables or Kubernetes labels on your deployments and pods. The DDOT detects this tagging configuration and applies it to the data it collects from containers.

To get the full range of unified service tagging, add both the environment variables and the deployment/pod labels:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    tags.datadoghq.com/env: "<ENV>"
    tags.datadoghq.com/service: "<SERVICE>"
    tags.datadoghq.com/version: "<VERSION>"
...
template:
  metadata:
    labels:
      tags.datadoghq.com/env: "<ENV>"
      tags.datadoghq.com/service: "<SERVICE>"
      tags.datadoghq.com/version: "<VERSION>"
  containers:
  -  ...
     env:
      - name: OTEL_SERVICE_NAME
        value: "<SERVICE>"
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: >-
          service.name=$(OTEL_SERVICE_NAME),
          service.version=<VERSION>,
          deployment.environment.name=<ENV>          

Run the application

Redeploy your application to apply the changes made in the deployment manifest. Once the updated configuration is active, Unified Service Tagging will be fully enabled for your metrics, traces, and logs.

Explore observability data in Datadog

Use Datadog to explore the observability data for your application.

Fleet automation

Explore your Datadog Agent and Collector configuration.

Live container monitoring

Monitor your container health using Live Container Monitoring capabilities.

Monitor your container health from the Containers page.

Infrastructure node health

View runtime and infrastructure metrics to visualize, monitor, and measure the performance of your nodes.

View runtime and infrastructure metrics from the Host List.

Logs

View logs to monitor and troubleshoot application and system operations.

Traces

View traces and spans to observe the status and performance of requests processed by your application, with infrastructure metrics correlated in the same trace.