Live Containers Configuration
Kubernetes resources
The Datadog Agent and Cluster Agent can be configured to retrieve Kubernetes resources for Live Containers. This feature allows you to monitor the state of pods, deployments, and other Kubernetes concepts in a specific namespace or availability zone, view resource specifications for failed pods within a deployment, correlate node activity with related logs, and more.
Kubernetes resources for Live Containers requires Agent version >= 7.27.0 and Cluster Agent version >= 1.11.0. For older versions of the Datadog Agent and Cluster Agent, see Live Containers Legacy Configuration.
Note: For Kubernetes version 1.25 and above, the minimal Cluster Agent version required is 7.40.0.
If you are using the official Datadog Helm Chart:
Use chart version >= 2.10.0. Ensure the Agent and Cluster Agent versions are hardcoded with the minimum versions required (or above) in your Helm chart values.yaml file.
Enable the Process Agent. You can do this by modifying your datadog-values.yaml
file to include:
datadog:
# (...)
processAgent:
enabled: true
Deploy a new release.
In some setups, the Process Agent and Cluster Agent cannot automatically detect a Kubernetes cluster name. If this happens, the feature does not start, and the following warning displays in the Cluster Agent log: Orchestrator explorer enabled but no cluster name set: disabling
. In this case, you must set datadog.clusterName
to your cluster name in values.yaml.
Cluster Agent version >= 1.11.0 is required before configuring the DaemonSet. The Cluster Agent must be running, and the Agent must be able to communicate with it. See the Cluster Agent Setup for configuration.
Set the Cluster Agent container with the following environment variable:
- name: DD_ORCHESTRATOR_EXPLORER_ENABLED
value: "true"
Set the Cluster Agent ClusterRole with the following RBAC permissions.
Note in particular that for the apps
and batch
apiGroups, Live Containers need
permissions to collect common Kubernetes resources (pods
, services
,
nodes
, etc.), which should be already in the RBAC if you followed Cluster
Agent Setup. But if they are missing, ensure they are
added (after deployments
, replicasets
):
ClusterRole:
- apiGroups: # To create the datadog-cluster-id ConfigMap
- ""
resources:
- configmaps
verbs:
- create
- get
- update
...
- apiGroups: # Required to get the kube-system namespace UID and generate a cluster ID
- ""
resources:
- namespaces
verbs:
- get
...
- apiGroups: # To collect new resource types
- "apps"
resources:
- deployments
- replicasets
- daemonsets
- statefulsets
verbs:
- list
- get
- watch
- apiGroups:
- "batch"
resources:
- cronjobs
- jobs
verbs:
- list
- get
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- list
- get
- watch
- apiGroups:
- rbac.authorization.k8s.io
resources:
- roles
- rolebindings
- clusterroles
- clusterrolebindings
verbs:
- list
- get
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- list
- watch
...
These permissions are needed to create a datadog-cluster-id
ConfigMap in the same Namespace as the Agent DaemonSet and the Cluster Agent Deployment, as well as to collect supported Kubernetes resources.
If the cluster-id
ConfigMap isn’t created by the Cluster Agent, the Agent pod cannot collect resources. In such a case, update the Cluster Agent permissions and restart its pods to let it create the ConfigMap, and then restart the Agent pod.
The Process Agent, which runs in the Agent DaemonSet, must be enabled and running (it doesn’t need to run the process collection), and configured with the following options:
- name: DD_ORCHESTRATOR_EXPLORER_ENABLED
value: "true"
In some setups, the Process Agent and Cluster Agent cannot automatically detect a Kubernetes cluster name. If this happens, the feature does not start, and the following warning displays in the Cluster Agent log: Orchestrator explorer enabled but no cluster name set: disabling
. In this case, add the following options in the env
section of both the Cluster Agent and the Process Agent:
- name: DD_CLUSTER_NAME
value: "<YOUR_CLUSTER_NAME>"
Resource collection compatibility matrix
The following table presents the list of collected resources and the minimal Agent, Cluster Agent, and Helm chart versions for each.
Resource | Minimal Agent version | Minimal Cluster Agent version* | Minimal Helm chart version | Minimal Kubernetes version |
---|
ClusterRoleBindings | 7.27.0 | 1.19.0 | 2.30.9 | 1.14.0 |
ClusterRoles | 7.27.0 | 1.19.0 | 2.30.9 | 1.14.0 |
Clusters | 7.27.0 | 1.12.0 | 2.10.0 | 1.17.0 |
CronJobs | 7.27.0 | 7.40.0 | 2.15.5 | 1.16.0 |
DaemonSets | 7.27.0 | 1.14.0 | 2.16.3 | 1.16.0 |
Deployments | 7.27.0 | 1.11.0 | 2.10.0 | 1.16.0 |
Ingresses | 7.27.0 | 1.22.0 | 2.30.7 | 1.21.0 |
Jobs | 7.27.0 | 1.13.1 | 2.15.5 | 1.16.0 |
Namespaces | 7.27.0 | 7.41.0 | 2.30.9 | 1.17.0 |
Nodes | 7.27.0 | 1.11.0 | 2.10.0 | 1.17.0 |
PersistentVolumes | 7.27.0 | 1.18.0 | 2.30.4 | 1.17.0 |
PersistentVolumeClaims | 7.27.0 | 1.18.0 | 2.30.4 | 1.17.0 |
Pods | 7.27.0 | 1.11.0 | 2.10.0 | 1.17.0 |
ReplicaSets | 7.27.0 | 1.11.0 | 2.10.0 | 1.16.0 |
RoleBindings | 7.27.0 | 1.19.0 | 2.30.9 | 1.14.0 |
Roles | 7.27.0 | 1.19.0 | 2.30.9 | 1.14.0 |
ServiceAccounts | 7.27.0 | 1.19.0 | 2.30.9 | 1.17.0 |
Services | 7.27.0 | 1.11.0 | 2.10.0 | 1.17.0 |
Statefulsets | 7.27.0 | 1.15.0 | 2.20.1 | 1.16.0 |
Note: After version 1.22, Cluster Agent version numbering follows Agent release numbering, starting with version 7.39.0.
You can add custom tags to Kubernetes resources to ease filtering inside the Kubernetes resources view.
Additional tags are added through the DD_ORCHESTRATOR_EXPLORER_EXTRA_TAGS
environment variable.
Note: These tags only show up in the Kubernetes resources view.
If you are using the official Helm chart, add the environment variable on both the Process Agent and the Cluster Agent by setting agents.containers.processAgent.env
and clusterAgent.env
in [values.yaml][1].
agents:
containers:
processAgent:
env:
- name: "DD_ORCHESTRATOR_EXPLORER_EXTRA_TAGS"
value: "tag1:value1 tag2:value2"
clusterAgent:
env:
- name: "DD_ORCHESTRATOR_EXPLORER_EXTRA_TAGS"
value: "tag1:value1 tag2:value2"
Then, deploy a new release.
Set the environment variable on both the Process Agent and Cluster Agent containers:
- name: DD_ORCHESTRATOR_EXPLORER_EXTRA_TAGS
value: "tag1:value1 tag2:value2"
Include or exclude containers
It is possible to include and/or exclude containers from real-time collection:
- Exclude containers either by passing the environment variable
DD_CONTAINER_EXCLUDE
or by adding container_exclude:
in your datadog.yaml
main configuration file. - Include containers either by passing the environment variable
DD_CONTAINER_INCLUDE
or by adding container_include:
in your datadog.yaml
main configuration file.
Both arguments take an image name as value. Regular expressions are also supported.
For example, to exclude all Debian images except containers with a name starting with frontend, add these two configuration lines in your datadog.yaml
file:
container_exclude: ["image:debian"]
container_include: ["name:frontend.*"]
Note: For Agent 5, instead of including the above in the datadog.conf
main configuration file, explicitly add a datadog.yaml
file to /etc/datadog-agent/
, as the Process Agent requires all configuration options here. This configuration only excludes containers from real-time collection, not from Autodiscovery.
To prevent the leaking of sensitive data, you can scrub sensitive words in container YAML files. Container scrubbing is enabled by default for Helm charts, and some default sensitive words are provided:
password
passwd
mysql_pwd
access_token
auth_token
api_key
apikey
pwd
secret
credentials
stripetoken
You can set additional sensitive words by providing a list of words to the environment variable DD_ORCHESTRATOR_EXPLORER_CUSTOM_SENSITIVE_WORDS
. This adds to, and does not overwrite, the default words.
Note: The additional sensitive words must be in lowercase, as the Agent compares the text with the pattern in lowercase. This means password
scrubs MY_PASSWORD
to MY_*******
, while PASSWORD
does not.
You need to setup this environment variable for the following agents:
- process-agent
- cluster-agent
env:
- name: DD_ORCHESTRATOR_EXPLORER_CUSTOM_SENSITIVE_WORDS
value: "customword1 customword2 customword3"
For example, because password
is a sensitive word, the scrubber changes <MY_PASSWORD>
in any of the following to a string of asterisks, ***********
:
password <MY_PASSWORD>
password=<MY_PASSWORD>
password: <MY_PASSWORD>
password::::== <MY_PASSWORD>
However, the scrubber does not scrub paths that contain sensitive words. For example, it does not overwrite /etc/vaultd/secret/haproxy-crt.pem
with /etc/vaultd/******/haproxy-crt.pem
even though secret
is a sensitive word.
Further reading
Additional helpful documentation, links, and articles: