--- title: Monitor Kubernetes with OpenTelemetry description: >- Send Kubernetes resource data and infrastructure metrics to Datadog with OpenTelemetry. breadcrumbs: Docs > Containers > Kubernetes > Monitor Kubernetes with OpenTelemetry --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Monitor Kubernetes with OpenTelemetry ## Overview{% #overview %} Use OpenTelemetry to send Kubernetes data to Datadog without installing the Datadog Agent. Choose the setup that matches the data you need: | Goal | Components | Setup | | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | View pods, deployments, and other resource data in Kubernetes Explorer | One cluster Collector with the `k8sobjects` receiver | [Send Kubernetes resources over OTLP](https://docs.datadoghq.com/containers/monitoring/kubernetes_explorer.md?tab=opentelemetrycollector#enable-kubernetes-explorer) | | Populate Datadog's out-of-the-box Kubernetes dashboards | `kube-state-metrics`, one cluster Collector, and one node Collector per node | Follow this guide | The full setup below collects Kubernetes infrastructure metrics for the [Kubernetes - Overview](https://app.datadoghq.com/dash/integration/86/kubernetes---overview) dashboard and resource data for [Kubernetes Explorer](https://docs.datadoghq.com/containers/monitoring/kubernetes_explorer.md). It does not instrument your applications. {% image source="https://docs.dd-static.net/images/opentelemetry/collector_exporter/kubernetes_metrics.3f85284d72929c386b5ba4d3f15df3ca.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/opentelemetry/collector_exporter/kubernetes_metrics.3f85284d72929c386b5ba4d3f15df3ca.png?auto=format&fit=max&w=850&dpr=2 2x" alt="The 'Kubernetes - Overview' dashboard, showing metrics for containers, including status and resource usage of your cluster and its containers." /%} The setup uses three components: - **[`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics)** generates metrics about Kubernetes objects, such as deployments, nodes, and pods. - **A cluster Collector**, running as a single-replica Deployment, collects cluster-wide metrics and resource data for Explorer. - **A node Collector**, running as a DaemonSet, collects metrics from each node, such as CPU and memory usage. The cluster Collector scrapes `kube-state-metrics` with its Prometheus receiver. You do not need to install a Prometheus server. These metrics populate the dashboards linked from Kubernetes Explorer. The `k8sobjects` receiver supplies Explorer's resource data. ## Setup{% #setup %} These steps deploy new Collectors in the `default` namespace. If you already collect Kubernetes metrics, review your existing configuration before deploying additional Collectors to avoid duplicate collection. ### Prerequisites{% #prerequisites %} - [Helm](https://helm.sh/docs/intro/install/) and `kubectl`, with permission to deploy workloads and create RBAC resources in the cluster. - A [Datadog API key](https://docs.datadoghq.com/account_management/api-app-keys.md#api-keys) and your [Datadog site](https://docs.datadoghq.com/getting_started/site.md). Use OpenTelemetry Collector [Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/opentelemetry-collector-0.156.2/charts/opentelemetry-collector) v0.156.2 or later and OpenTelemetry Collector Contrib v0.159.0 or later. The commands below pin the Collector image to v0.159.0. The `k8sobjects` receiver used for Explorer can increase Kubernetes API server load. Datadog recommends Kubernetes 1.33 or later and testing on smaller clusters before expanding collection. See [Kubernetes Explorer limitations](https://docs.datadoghq.com/containers/monitoring/kubernetes_explorer.md?tab=opentelemetrycollector#limitations). {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="warning" %} Kubernetes Explorer with OpenTelemetry is not available for {% placeholder "user-datadog-site-name" /%}. {% /alert %} {% /callout %} ### Installation{% #installation %} #### 1. Install kube-state-metrics Add the `prometheus-community` Helm repository and install `kube-state-metrics`: ```sh helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install kube-state-metrics prometheus-community/kube-state-metrics \ --namespace default ``` The reference configuration scrapes `kube-state-metrics.default.svc:8080`. If you use a different service name or namespace, update the Prometheus receiver target in `cluster-collector.yaml`. #### 2. Create a Datadog secret Set your API key and site, then create a secret in the Collectors' namespace: ```sh export DD_API_KEY="" export DD_SITE="" kubectl create secret generic datadog-secret \ --namespace default \ --from-literal="api-key=$DD_API_KEY" \ --from-literal="dd-site=$DD_SITE" ``` #### 3. Configure and install the Collectors 1. Add the OpenTelemetry Helm chart repository: ```sh helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update ``` 1. Download [cluster-collector.yaml](https://github.com/DataDog/opentelemetry-examples/blob/main/guides/kubernetes/configuration/cluster-collector.yaml) and [daemonset-collector.yaml](https://github.com/DataDog/opentelemetry-examples/blob/main/guides/kubernetes/configuration/daemonset-collector.yaml) into the same directory. These Helm values files are maintained in the `opentelemetry-examples` repository: ```sh CONFIG_URL="https://raw.githubusercontent.com/DataDog/opentelemetry-examples/main/guides/kubernetes/configuration" curl -fsSLo cluster-collector.yaml "$CONFIG_URL/cluster-collector.yaml" curl -fsSLo daemonset-collector.yaml "$CONFIG_URL/daemonset-collector.yaml" ``` 1. In both files, replace the `datadog/exporter` block under `config.exporters` with the recommended OTLP HTTP exporter: ```yaml exporters: otlp_http: endpoint: https://otlp.${env:DD_SITE} logs_endpoint: https://otlp.${env:DD_SITE}/api/v2/otlplogs headers: dd-api-key: ${env:DD_API_KEY} dd-otel-metric-config: >- { "resource_attributes_as_tags": true, "instrumentation_scope_metadata_as_tags": true } compression: zstd compression_params: level: 3 sending_queue: batch: sizer: bytes min_size: 2097152 max_size: 4194304 ``` Replace each `datadog/exporter` entry in a pipeline's `exporters` list with `otlp_http`. In the cluster Collector, do not include the Datadog Exporter's `orchestrator_explorer` option; Datadog recognizes resource data from the `k8sobjects` receiver when it arrives over OTLP. 1. Update trace processing in `daemonset-collector.yaml`. The reference file also supports application traces: - If the node Collector does not receive application traces, remove `datadog/connector`, the `traces` and `traces/sampling` pipelines, and `datadog/connector` from the `metrics` pipeline's receivers. - If the node Collector receives application traces, use the [recommended Collector configuration](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter.md?tab=kubernetesmanifestreference#2-configure-and-deploy-the-collector) to replace `datadog/connector` with the upstream `forward/traces_sample` and `span_metrics` connectors. Keep the `datadog` extension and its entry under `service.extensions`. The extension reports Collector metadata used for host enrichment; it does not export telemetry. 1. Make sure both Collectors report the same cluster name: - To detect it automatically, keep `k8s_api` and your provider's detector under `resourcedetection.detectors`, and remove the other cloud-provider detectors. Configure the provider detector and its permissions for [EKS](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#amazon-eks), [AKS](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#azure-aks), or [GKE](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#gcp-metadata). - Otherwise, set `resourcedetection.detectors` to `[k8s_api]`. Uncomment `resource/add-cluster-name` and replace `` with the same value in both files. In each pipeline that uses `resourcedetection`, add `resource/add-cluster-name` immediately after it. Keep the other processors in place. 1. Run the following commands from the directory containing the values files: ```sh # Install the node Collector (DaemonSet) helm install otel-daemon-collector open-telemetry/opentelemetry-collector \ --namespace default \ -f daemonset-collector.yaml \ --set image.repository=otel/opentelemetry-collector-contrib \ --set image.tag=0.159.0 # Install the cluster Collector (Deployment) helm install otel-cluster-collector open-telemetry/opentelemetry-collector \ --namespace default \ -f cluster-collector.yaml \ --set image.repository=otel/opentelemetry-collector-contrib \ --set image.tag=0.159.0 ``` ### Verify the setup{% #verify-the-setup %} 1. Check that the Collector and `kube-state-metrics` pods are running and ready: ```sh kubectl get pods --namespace default \ -l 'app.kubernetes.io/instance in (otel-daemon-collector,otel-cluster-collector,kube-state-metrics)' ``` 1. Open the [Kubernetes - Overview](https://app.datadoghq.com/dash/integration/86/kubernetes---overview) dashboard and select your cluster. Check for node resource usage and Kubernetes object metrics. 1. Open [Kubernetes Explorer](https://app.datadoghq.com/orchestration/overview) and filter by your cluster name. Check that resources such as pods and deployments appear. If data is missing, check the Collector logs for export errors. Verify that the secret contains an API key for the selected Datadog site. ## Correlate traces with infrastructure metrics (optional){% #correlating-traces-with-infrastructure-metrics %} For applications that already send traces, use [unified service tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md?tab=kubernetes#opentelemetry) to correlate application telemetry with infrastructure metrics. Set the same resource attributes on both: - `service.name` maps to the Datadog `service` tag. - `service.version` maps to the Datadog `version` tag. - `deployment.environment.name` maps to the Datadog `env` tag. ### Application configuration{% #application-configuration %} Set the following environment variables in your application's container specification to tag outgoing telemetry: ```yaml spec: containers: - name: my-container env: - name: OTEL_SERVICE_NAME value: "" - name: OTEL_RESOURCE_ATTRIBUTES value: "service.version=,deployment.environment.name=" ``` ### Infrastructure configuration{% #infrastructure-configuration %} Add the corresponding annotations to your Kubernetes `Deployment` metadata. The `k8sattributes` processor in the Collector uses these annotations to enrich infrastructure metrics with service context. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app annotations: # Use resource.opentelemetry.io/ for the k8sattributes processor resource.opentelemetry.io/service.name: "" resource.opentelemetry.io/service.version: "" resource.opentelemetry.io/deployment.environment.name: "" spec: template: metadata: annotations: resource.opentelemetry.io/service.name: "" resource.opentelemetry.io/service.version: "" resource.opentelemetry.io/deployment.environment.name: "" # ... rest of the manifest ``` ## Data collected{% #data-collected %} This integration collects metrics using several OpenTelemetry receivers. ### kube-state-metrics (using Prometheus receiver){% #kube-state-metrics-using-prometheus-receiver %} Metrics scraped from the `kube-state-metrics` endpoint provide information about the state of Kubernetes API objects. ### Kubelet stats receiver{% #kubelet-stats-receiver %} The `kubeletstatsreceiver` collects metrics from the Kubelet on each node, focusing on pod, container, and volume resource usage. | OTEL | DESCRIPTION | FILTER | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | k8s.node.filesystem.usage | Node filesystem usage | | k8s.pod.cpu.usage | Pod's CPU usage, measured in cpus. Range from 0 to the number of allocatable CPUs. Note: CPU usage of the specific Pod on all available CPU cores, averaged over the sample window | | k8s.pod.filesystem.usage | Pod filesystem usage | | k8s.pod.memory.rss | Pod memory RSS. Note: The amount of anonymous and swap cache memory (includes transparent hugepages). This metric is derived from the [MemoryStats.RSSBytes](https://pkg.go.dev/k8s.io/kubelet@v0.34.0/pkg/apis/stats/v1alpha1#MemoryStats) field of the [PodStats.Memory](https://pkg.go.dev/k8s.io/kubelet@v0.34.0/pkg/apis/stats/v1alpha1#PodStats) of the Kubelet's stats API. | | k8s.pod.memory.usage | Memory usage of the Pod. Note: Total memory usage of the Pod | | k8s.pod.network.errors | Pod network errors | `direction`: `receive` | | k8s.pod.network.errors | Pod network errors | `direction`: `transmit` | | k8s.pod.network.io | Network bytes for the Pod. | `direction`: `receive` | | k8s.pod.network.io | Network bytes for the Pod. | `direction`: `transmit` | ### Kubernetes cluster receiver{% #kubernetes-cluster-receiver %} The `k8sclusterreceiver` collects cluster-level metrics, such as the status and count of nodes, pods, and other objects. | OTEL | DESCRIPTION | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | k8s.container.cpu_limit | Maximum resource limit set for the container. See [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core) for details | | k8s.container.cpu_request | Resource requested for the container. See [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core) for details | | k8s.container.ephemeralstorage_limit | Maximum resource limit set for the container. See [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core) for details | | k8s.container.ephemeralstorage_request | Resource requested for the container. See [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core) for details | | k8s.container.memory_limit | Maximum resource limit set for the container. See [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core) for details | | k8s.container.memory_request | Resource requested for the container. See [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#resourcerequirements-v1-core) for details | ### Count connector{% #count-connector %} The [count connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/countconnector) generates object-count metrics by counting the number of metric series that pass through the pipeline. It produces the following metrics: | OTEL | DESCRIPTION | | ----------------- | ------------------------------ | | k8s.job.count | The current count of jobs. | | k8s.node.count | The current count of nodes. | | k8s.service.count | The current count of services. | ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Send OpenTelemetry Data to Datadog](https://docs.datadoghq.com/opentelemetry/setup.md) - [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md) - [Example Collector Configurations](https://github.com/DataDog/opentelemetry-examples/tree/main/guides/kubernetes)