--- title: Kubernetes log collection description: >- Configure log collection from containerized applications running on Kubernetes using the Datadog Agent breadcrumbs: Docs > Containers > Kubernetes > Kubernetes log collection --- # Kubernetes log collection This page discusses collecting logs from Kubernetes log files. When your containerized applications write their logs to standard output and error (`stdout`/`stderr`), the container runtime and Kubernetes automatically manage the logs for you. The default pattern is that [Kubernetes stores these log streams as files](https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-location-node) on the host in the `/var/log/pods` folder and subfolders for each Pod and container. The Datadog Agent can collect these Kubernetes log files for these containers using the instructions below. This option scales well for the ephemeral nature of the Pods that Kubernetes creates, and is more resource-efficient than collecting logs from the Docker socket. Datadog recommends this method for log collection in Kubernetes. Alternatively, the Datadog Agent can also collect logs by repeated requests to the Docker API through the Docker socket. However, this requires Docker as the container runtime for your Kubernetes cluster. This is also more resource-intensive than using log files. To see how to collect logs using the Docker socket, see [Log collection with Docker socket](https://docs.datadoghq.com/agent/faq/log-collection-with-docker-socket.md). If your containerized applications are writing to log files stored in the container, this can complicate log collection. See log collection from a file. ## Setup{% #setup %} ### Log collection{% #log-collection %} Before you start collecting application logs, ensure that you are running the Datadog Agent in your Kubernetes cluster. To configure log collection manually in the DaemonSet, see [DaemonSet Log Collection](https://docs.datadoghq.com/containers/guide/kubernetes_daemonset.md#log-collection). Otherwise, follow the instructions below: {% tab title="Datadog Operator" %} Update your `datadog-agent.yaml` manifest with: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: global: credentials: apiKey: features: logCollection: enabled: true containerCollectAll: true ``` Then apply the new configuration: ```shell kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml ``` See the sample [manifest with logs, metrics, and APM collection enabled](https://github.com/DataDog/datadog-operator/blob/main/examples/datadogagent/datadog-agent-with-logs-apm.yaml) for an additional example. You can set `features.logCollection.containerCollectAll` to `true` to collect logs from all discovered containers by default. When set to `false` (default), you need to specify Autodiscovery log configurations to enable log collection. For more information, see Log discovery - Filtering. {% /tab %} {% tab title="Helm" %} To enable log collection with Helm, update your [datadog-values.yaml](https://github.com/DataDog/helm-charts/blob/master/charts/datadog/values.yaml) file with the following log collection configuration. Then, upgrade your Datadog Helm chart: ```yaml datadog: logs: enabled: true containerCollectAll: true ``` You can set `datadog.logs.containerCollectAll` to `true` to collect logs from all discovered containers by default. When set to `false` (default), you need to specify Autodiscovery log configurations to enable log collection. For more information, see Log discovery - Filtering. {% /tab %} ### Unprivileged{% #unprivileged %} {% tab title="Datadog Operator" %} (Optional) To run an unprivileged installation, add the following to the [DatadogAgent custom resource](https://github.com/DataDog/datadog-operator/blob/main/docs/configuration.v2alpha1.md#override): ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: global: credentials: apiKey: features: logCollection: enabled: true containerCollectAll: true override: nodeAgent: securityContext: runAsUser: supplementalGroups: - ``` - Replace `` with the UID to run the Agent - Replace `` with the group ID that owns the Docker or containerd socket. {% /tab %} {% tab title="Helm" %} (Optional) To run an unprivileged installation, add the following in the `values.yaml` file: ```yaml datadog: securityContext: runAsUser: supplementalGroups: - ``` - Replace `` with the UID to run the Agent. - Replace `` with the group ID that owns the Docker or containerd socket. {% /tab %} {% alert level="danger" %} **Warning for unprivileged installations**When running an unprivileged installation, the Agent needs to be able to read log files in `/var/log/pods`.If you are using the containerd runtime, the log files in `/var/log/pods` are readable by members of the `root` group. With the above instructions, the Agent runs with the `root` group. No action is required.If you are using the Docker runtime, the log files in `/var/log/pods` are symbolic links to `/var/lib/docker/containers`, which is traversable only by the `root` user. Consequently, with the Docker runtime, it is not possible for a non-`root` Agent to read logs in `/var/log/pods`. The Docker socket must be mounted in the Agent container, so that it can get Pod logs through the Docker daemon.To collect logs from `/var/log/pods` when the Docker socket is mounted, set the environment variable `DD_LOGS_CONFIG_K8S_CONTAINER_USE_FILE` (or `logs_config.k8s_container_use_file` in `datadog.yaml`) to `true`. This forces the Agent to use file collection mode. {% /alert %} ## Log discovery{% #log-discovery %} The Datadog Agent in Kubernetes is deployed by a DaemonSet (managed by the Datadog Operator or Helm). This DaemonSet schedules one replica of the Agent Pod on each node of the cluster. Each Agent Pod is then responsible for reporting the logs of the other Pods and containers on its respective node. When the "Container Collect All" feature is enabled, the Agent reports the logs from every discovered container with a default set of tags. ### Filtering{% #filtering %} When "Container Collect All" is enabled you can configure which containers you want to collect logs from. This can be useful to prevent the collection of the Datadog Agent logs, if desired. You can do this by passing configurations to the Datadog Agent to control what it pulls, or by passing configurations to the Kubernetes Pod to exclude certain logs more explicitly. When filtering out logs through methods like `DD_CONTAINER_EXCLUDE_LOGS` or `ad.datadoghq.com/logs_exclude`, the Agent ignores log collection regardless of explicitly defined log collection configurations in [Autodiscovery annotations](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-annotations) or [Autodiscovery configuration files](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-configuration-files). When "Container Collect All" is disabled (default) you don't need to add any filtering because everything is excluded by default. To include collection for only selected pods, you can enable the log configuration by [Autodiscovery annotations](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-annotations) or [Autodiscovery configuration files](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-configuration-files) for the desired pods. See [Container Discovery Management](https://docs.datadoghq.com/agent/guide/autodiscovery-management.md) to learn more about filtering. ### Tagging{% #tagging %} The Datadog Agent tags the logs from the Kubernetes containers with the default [Kubernetes tags](https://docs.datadoghq.com/containers/kubernetes/tag.md), as well as any custom extracted tags. When "Container Collect All" is enabled, the Agent reports the logs for a container with a `source` and `service` tag matching the container short image name. For example, the logs from a container using the `gcr.io/owner/example-image:latest` container image would have `example-image` as the `source`, `service`, and `short_image` tag value. The `service` tag can also be set by the [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md?tab=kubernetes) Pod label `tags.datadoghq.com/service: ""`. For more information about `source` and `service` attributes, see [Reserved Attributes](https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention.md). The `source` tag can be important for your logs, as the [out of box log pipelines](https://docs.datadoghq.com/logs/log_configuration/pipelines.md?tab=source#integration-pipelines) are filtered using this tag. However, these pipelines can be completely customized as desired. You can see the steps in the Integration Logs section below for customizing the tags on your logs further. ## Integration logs{% #integration-logs %} [Autodiscovery](https://docs.datadoghq.com/getting_started/containers/autodiscovery.md) enables you to use templates to configure log collection (and other capabilities) on containers. This can be used to enable log collection, customize tagging, and add advanced collection rules. To configure log collection for an integration with Autodiscovery you can either: - Specify a log configuration as Autodiscovery Annotations on a given Pod, to configure the rules for a given container *(Recommended)* - Specify a log configuration as a configuration file, to configure the rules for each matching container by image At minimum, these log configurations require a `source` and `service` tag. You may want to match the `source` tag to one of Datadog's [out-of-the-box log pipelines](https://docs.datadoghq.com/logs/log_configuration/pipelines.md?tab=source#integration-pipelines) to help automatically enrich your logs. You can also find a [library of pipelines in Datadog](https://app.datadoghq.com/logs/pipelines/pipeline/library). ### Autodiscovery annotations{% #autodiscovery-annotations %} With Autodiscovery, the Agent automatically searches all Pod annotations for integration templates. To apply a specific configuration to a given container, add the annotation `ad.datadoghq.com/.logs` to your Pod with the JSON formatted log configuration. **Note**: Autodiscovery annotations identify containers by name, **not** image. It tries to match `` to the `.spec.containers[i].name`, not `.spec.containers[i].image`. {% alert level="info" %} If you define your Kubernetes Pods directly (with `kind:Pod`), add each Pod's annotations in its `metadata` section, as shown in the following section.If you define your Kubernetes Pods indirectly (with replication controllers, ReplicaSets, or Deployments), add Pod annotations to the Pod template under `.spec.template.metadata`. {% /alert %} #### Configure a single container{% #configure-a-single-container %} To configure log collection for a given container within your Pod, add the following annotations to your Pod: ```yaml apiVersion: v1 kind: Pod # (...) metadata: name: '' annotations: ad.datadoghq.com/.logs: '[]' # (...) spec: containers: - name: '' # (...) ``` #### Example log Autodiscovery annotations{% #example-log-autodiscovery-annotations %} The following Pod annotation defines the integration template for an example container. It is defined within the Pod template's annotations, rather than on the Deployment itself. This log configuration sets all the logs from the `app` container with the tags `source:java`, `service:example-app`, and the extra tag `foo:bar`. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: example labels: app: example-app spec: selector: matchLabels: app: example-app template: metadata: labels: app: example-app annotations: ad.datadoghq.com/app.logs: '[{"source":"java", "service":"example-app", "tags":["foo:bar"]}]' spec: containers: - name: app image: owner/example-image:latest ``` #### Configure two different containers{% #configure-two-different-containers %} To apply two different integration templates to two different containers within your Pod, `` and ``, add the following annotations to your Pod: ```yaml apiVersion: v1 kind: Pod # (...) metadata: name: '' annotations: ad.datadoghq.com/.logs: '[]' # (...) ad.datadoghq.com/.logs: '[]' spec: containers: - name: '' # (...) - name: '' # (...) ``` ### Autodiscovery configuration files{% #autodiscovery-configuration-files %} You can provide the Datadog Agent with configuration files to have the Agent run a specified integration when it discovers a container using the matching image identifier. This allows you to create a generic log configuration that applies to a set of container images. {% tab title="Datadog Operator" %} You can customize logs collection per integration with an override in the `override.nodeAgent.extraConfd.configDataMap`. This method creates the ConfigMap and mounts the desired configuration file onto the Agent container. ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: #(...) override: nodeAgent: extraConfd: configDataMap: .yaml: |- ad_identifiers: - logs: - source: example-source service: example-service ``` The `` should match the container short image name that you want this to apply towards. See the sample manifest [with ConfigMap mapping](https://github.com/DataDog/datadog-operator/blob/main/examples/datadogagent/datadog-agent-with-extraconfd.yaml) for an additional example. {% /tab %} {% tab title="Helm" %} You can customize logs collection per integration within `datadog.confd`. This method creates the ConfigMap and mounts the desired configuration file onto the Agent container. ```yaml datadog: #(...) confd: .yaml: |- ad_identifiers: - logs: - source: example-source service: example-service ``` The `` should match the container short image name that you want this to apply towards. {% /tab %} {% tab title="Key-value store" %} The following etcd commands create a Redis integration template with a custom `password` parameter and tags logs with the correct `source` and `service` attributes: ``` etcdctl mkdir /datadog/check_configs/redis etcdctl set /datadog/check_configs/redis/logs '[{"source": "redis", "service": "redis", "tags": ["env:prod"]}]' ``` Notice that each of the three values is a list. Autodiscovery assembles list items into the integration configurations based on shared list indexes. In this case, it composes the first (and only) check configuration from `check_names[0]`, `init_configs[0]`, and `instances[0]`. Unlike auto-conf files, **key-value stores may use the short OR long image name as container identifiers**, for example, `redis` OR `redis:latest`. Autodiscovery can use [Consul](https://docs.datadoghq.com/integrations/consul.md), Etcd, and Zookeeper as integration template sources. To use a key-value store, configure it in the Agent `datadog.yaml` configuration file and mount this file inside the containerized Agent. Alternatively, pass your key-value store as environment variables to the containerized Agent. #### In `datadog.yaml`{% #in-datadogyaml %} In the `datadog.yaml` file, set the `` address and `` of your key-value store: ```yaml config_providers: - name: etcd polling: true template_dir: /datadog/check_configs template_url: ':' username: password: - name: consul polling: true template_dir: datadog/check_configs template_url: ':' ca_file: ca_path: cert_file: key_file: username: password: token: - name: zookeeper polling: true template_dir: /datadog/check_configs template_url: ':' username: password: ``` Then [restart the Agent](https://docs.datadoghq.com/agent/configuration/agent-commands.md) to apply the configuration change. #### In environment variables{% #in-environment-variables %} With the key-value store enabled as a template source, the Agent looks for templates under the key `/datadog/check_configs`. Autodiscovery expects a key-value hierarchy like this: ```yaml /datadog/ check_configs/ / - logs: [""] ... ``` **Note**: To apply a specific configuration to a given container, Autodiscovery identifies containers by **image** when using the key-value stores by trying to match `` to `.spec.containers[0].image`. {% /tab %} To match a log configuration to a set of containers with more granularity than the container short image name, see [Autodiscovery Container Identifiers](https://docs.datadoghq.com/containers/guide/ad_identifiers.md). ## Advanced log collection{% #advanced-log-collection %} Use Autodiscovery log labels to apply advanced log collection processing logic, for example: - [Filter logs before sending them to Datadog](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md?tab=kubernetes#filter-logs). - [Scrub sensitive data from your logs](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md?tab=kubernetes#scrub-sensitive-data-from-your-logs). - [Proceed to multi-line aggregation](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md?tab=kubernetes#multi-line-aggregation). ### From a container local log file{% #from-a-container-local-log-file %} Datadog recommends that you use the `stdout` and `stderr` output streams for containerized applications, so that you can more automatically set up log collection. However, the Agent can also directly collect logs from a file based on an annotation. To collect these logs, use `ad.datadoghq.com/.logs` with a `type: file` and `path` configuration. Logs collected from files with such an annotation are automatically tagged with the same set of tags as logs coming from the container itself. Datadog recommends that you use the `stdout` and `stderr` output streams for containerized applications, so that you can automatically set up log collection. For more information, see the Recommended configurations. These file paths are **relative** to the Agent container. Therefore, the directory containing the log file needs to be mounted into both the application and Agent container so the Agent can have proper visibility. For example, you can do this with a shared `hostPath` volume. The Pod below is emitting logs into the file `/var/log/example/app.log`. This is done in the `/var/log/example` directory, where a volume and volumeMount have set this as a `hostPath`. ```yaml apiVersion: v1 kind: Pod metadata: name: logger annotations: ad.datadoghq.com/busybox.logs: | [{ "type": "file", "path": "/var/log/example/app.log", "source": "example-source", "service": "example-service" }] spec: containers: - name: busybox image: busybox command: [ "/bin/sh", "-c", "--" ] args: [ "while true; do sleep 1; echo `date` example file log >> /var/log/example/app.log; done;" ] volumeMounts: - name: applogs mountPath: /var/log/example volumes: - name: applogs hostPath: path: /var/log/example ``` The equivalent volume and volumeMount path need to be set in the Agent container so it can read that same log file. ```yaml containers: - name: agent # (...) volumeMounts: - mountPath: /var/log/example name: applogs # (...) volumes: - name: applogs hostPath: path: /var/log/example # (...) ``` #### Recommended configurations{% #recommended-configurations %} - This strategy can work for a given pod, but can become cumbersome with multiple apps using this strategy. You can also run into issues if multiple replicas are using the same log path. If possible, Datadog recommends taking advantage of the [Autodiscovery template variable](https://docs.datadoghq.com/containers/guide/template_variables.md) `%%kube_pod_name%%`. For example, you can set your `path` to reference this variable: `"path": "/var/log/example/%%kube_pod_name%%/app.log"`. Your application pod then needs to write its log files with respect to this new path as well. You can use the [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) to help your application determine its Pod name. - When using this kind of annotation with a container, `stdout` and `stderr` logs are not collected automatically from the container. If collection from both the container output streams and file are needed, explicitly enable this in the annotation. For example: ```yaml ad.datadoghq.com/.logs: | [ {"type":"file","path":"/var/log/example/app.log","source":"file","service":"example-service"}, {"source":"container","service":"example-service"} ] ``` - When using this kind of combination, `source` and `service` have no default value for logs collected from a file and should be explicitly set in the annotation. ## Troubleshooting{% #troubleshooting %} For troubleshooting steps, see [Container Log Collection Troubleshooting](https://docs.datadoghq.com/containers/troubleshooting/log-collection.md?tab=datadogoperator). ## Further Reading{% #further-reading %} - [Monitor logs from Amazon EKS on Fargate with Datadog](https://www.datadoghq.com/blog/eks-fargate-logs-datadog) - [Collect your application traces](https://docs.datadoghq.com/agent/kubernetes/apm.md) - [Collect your Prometheus metrics](https://docs.datadoghq.com/agent/kubernetes/prometheus.md) - [Collect automatically your applications metrics and logs](https://docs.datadoghq.com/agent/kubernetes/integrations.md) - [Limit data collection to a subset of containers only](https://docs.datadoghq.com/agent/guide/autodiscovery-management.md) - [Assign tags to all data emitted by a container](https://docs.datadoghq.com/agent/kubernetes/tag.md) - [Container Log Collection Troubleshooting](https://docs.datadoghq.com/containers/troubleshooting/log-collection.md)