Datadog Live Containers enable real-time visibility into all containers across your environment.
Taking inspiration from bedrock tools like htop and ctop, live containers give you complete coverage of your container infrastructure in a continuously updated table with resource metrics at two-second resolution and faceted search. Coupled with integrations for Docker, Kubernetes, ECS, and other container technologies, plus built-in tagging of dynamic components, the live container view provides a detailed overview of your containers’ health, resource consumption, and deployment in real time:
After deploying the Docker Agent, no other configuration is necessary.
Note: To collect container information in the standard install rather than with the Docker Agent, the
dd-agent user must have permissions to access docker.sock.
It is possible to include and/or exclude containers from real-time collection:
datadog.yamlmain configuration file.
datadog.yamlmain configuration file.
Both arguments take an image name as value; regular expressions are also supported.
For example, to exclude all Debian images except containers with a name starting with frontend, add these two configuration lines in your
ac_exclude: ["image:debian"] ac_include: ["name:frontend.*"]
Note: For Agent 5, instead of including the above in the
datadog.conf main configuration file, explicitly add a
datadog.yaml file to
/etc/datadog-agent/, as the Process Agent requires all configuration options here. This configuration only excludes containers from real-time collection, not from Autodiscovery.
Containers are, by their nature, extremely high cardinality objects. Datadog’s flexible string search matches substrings in the container name, ID, or image fields.
To combine multiple string searches into a complex query, you can use any of the following Boolean operators:
||Intersection: both terms are in the selected events (if nothing is added, AND is taken by default)||java AND elasticsearch|
||Union: either term is contained in the selected events||java OR python|
||Exclusion: the following term is NOT in the event. You may use the word
||java NOT elasticsearch
equivalent: java !elasticsearch
Use parentheses to group operators together. For example,
(NOT (elasticsearch OR kafka) java) OR python.
Containers are tagged with all existing host-level tags, as well as with metadata associated with individual containers.
All containers are tagged by
image_name, including integrations with popular orchestrators, such as ECS and Kubernetes, which provide further container-level tags. Additionally, each container is decorated with Docker, ECS, or Kubernetes icons so you can tell which are being orchestrated at a glance.
ECS containers are tagged by:
Kubernetes containers are tagged by:
The screenshot below displays a system that has been filtered down to a Kubernetes cluster of 9 nodes. RSS and CPU utilization on containers is reported compared to the provisioned limits on the containers, when they exist. Here, it is apparent that the containers in this cluster are over-provisioned. You could use tighter limits and bin packing to achieve better utilization of resources.
Container environments are dynamic and can be hard to follow. The following screenshot displays a view that has been pivotted by
host—and, to reduce system noise, filtered to
kube_namespace:default. You can see what services are running where, and how saturated key metrics are:
You could pivot by ECS
ecs_task_version to understand changes to resource utilization between updates.
Use the scatter plot analytic to compare two metrics with one another in order to better understand the performance of your containers.
To access the scatter plot analytic in the Containers page click on the Show Summary graph button and select the “Scatter Plot” tab:
By default, the graph groups by the
short_image tag key. The size of each dot represents the number of containers in that group, and clicking on a dot displays the individual containers and hosts that contribute to the group.
The query at the top of the scatter plot analytic allows you to control your scatter plot analytic:
While actively working with the containers page, metrics are collected at a 2-second resolution. This is important for highly volatile metrics such as CPU. In the background, for historical context, metrics are collected at 10s resolution.
This feature does not support Windows containers at this time.
Real-time (2s) data collection is turned off after 30 minutes. To resume real-time collection, refresh the page.
RBAC settings can restrict Kubernetes metadata collection. Refer to the RBAC entites for the Datadog Agent.
In Kubernetes the
health value is the containers’ readiness probe, not its liveness probe.
Additional helpful documentation, links, and articles: