---
title: Kubernetes log collection
description: >-
  Configure log collection from containerized applications running on Kubernetes
  using the Datadog Agent
breadcrumbs: Docs > Containers > Kubernetes > Kubernetes log collection
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Kubernetes log collection

This page discusses collecting logs from Kubernetes log files.

When your containerized applications write their logs to standard output and error (`stdout`/`stderr`), the container runtime and Kubernetes automatically manage the logs for you. The default pattern is that [Kubernetes stores these log streams as files](https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-location-node) on the host in the `/var/log/pods` folder and subfolders for each Pod and container.

The Datadog Agent can collect these Kubernetes log files for these containers using the instructions below. This option scales well for the ephemeral nature of the Pods that Kubernetes creates, and is more resource-efficient than collecting logs from the Docker socket. Datadog recommends this method for log collection in Kubernetes.

Alternatively, the Datadog Agent can also collect logs by repeated requests to the Docker API through the Docker socket. However, this requires Docker as the container runtime for your Kubernetes cluster. This is also more resource-intensive than using log files. To see how to collect logs using the Docker socket, see [Log collection with Docker socket](https://docs.datadoghq.com/agent/faq/log-collection-with-docker-socket.md). If your containerized applications are writing to log files stored in the container, this can complicate log collection. See log collection from a file.

## Setup{% #setup %}

### Log collection{% #log-collection %}

Before you start collecting application logs, ensure that you are running the Datadog Agent in your Kubernetes cluster.

To configure log collection manually in the DaemonSet, see [DaemonSet Log Collection](https://docs.datadoghq.com/containers/guide/kubernetes_daemonset.md#log-collection). Otherwise, follow the instructions below:

{% tab title="Datadog Operator" %}
Update your `datadog-agent.yaml` manifest with:

```yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>

  features:
    logCollection:
      enabled: true
      containerCollectAll: true
```

Then apply the new configuration:

```shell
kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml
```

See the sample [manifest with logs, metrics, and APM collection enabled](https://github.com/DataDog/datadog-operator/blob/main/examples/datadogagent/datadog-agent-with-logs-apm.yaml) for an additional example. You can set `features.logCollection.containerCollectAll` to `true` to collect logs from all discovered containers by default. When set to `false` (default), you need to specify Autodiscovery log configurations to enable log collection. For more information, see Log discovery - Filtering.
{% /tab %}

{% tab title="Helm" %}
To enable log collection with Helm, update your [datadog-values.yaml](https://github.com/DataDog/helm-charts/blob/master/charts/datadog/values.yaml) file with the following log collection configuration. Then, upgrade your Datadog Helm chart:

```yaml
datadog:
  logs:
    enabled: true
    containerCollectAll: true
```

You can set `datadog.logs.containerCollectAll` to `true` to collect logs from all discovered containers by default. When set to `false` (default), you need to specify Autodiscovery log configurations to enable log collection. For more information, see Log discovery - Filtering.
{% /tab %}

### Unprivileged{% #unprivileged %}

{% tab title="Datadog Operator" %}
(Optional) To run an unprivileged installation, add the following to the [DatadogAgent custom resource](https://github.com/DataDog/datadog-operator/blob/main/docs/configuration.v2alpha1.md#override):

```yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>

  features:
    logCollection:
      enabled: true
      containerCollectAll: true

  override:
    nodeAgent:
      securityContext:
        runAsUser: <USER_ID>
        supplementalGroups:
          - <DOCKER_GROUP_ID>
```

- Replace `<USER_ID>` with the UID to run the Agent
- Replace `<DOCKER_GROUP_ID>` with the group ID that owns the Docker or containerd socket.

{% /tab %}

{% tab title="Helm" %}
(Optional) To run an unprivileged installation, add the following in the `values.yaml` file:

```yaml
datadog:
  securityContext:
    runAsUser: <USER_ID>
    supplementalGroups:
      - <DOCKER_GROUP_ID>
```

- Replace `<USER_ID>` with the UID to run the Agent.
- Replace `<DOCKER_GROUP_ID>` with the group ID that owns the Docker or containerd socket.

{% /tab %}

{% alert level="danger" %}
**Warning for unprivileged installations**  When running an unprivileged installation, the Agent needs to be able to read log files in `/var/log/pods`.  If you are using the containerd runtime, the log files in `/var/log/pods` are readable by members of the `root` group. With the above instructions, the Agent runs with the `root` group. No action is required.  If you are using the Docker runtime, the log files in `/var/log/pods` are symbolic links to `/var/lib/docker/containers`, which is traversable only by the `root` user. Consequently, with the Docker runtime, it is not possible for a non-`root` Agent to read logs in `/var/log/pods`. The Docker socket must be mounted in the Agent container, so that it can get Pod logs through the Docker daemon.  To collect logs from `/var/log/pods` when the Docker socket is mounted, set the environment variable `DD_LOGS_CONFIG_K8S_CONTAINER_USE_FILE` (or `logs_config.k8s_container_use_file` in `datadog.yaml`) to `true`. This forces the Agent to use file collection mode.
{% /alert %}

## Log discovery{% #log-discovery %}

The Datadog Agent in Kubernetes is deployed by a DaemonSet (managed by the Datadog Operator or Helm). This DaemonSet schedules one replica of the Agent Pod on each node of the cluster. Each Agent Pod is then responsible for reporting the logs of the other Pods and containers on its respective node. When the "Container Collect All" feature is enabled, the Agent reports the logs from every discovered container with a default set of tags.

### Filtering{% #filtering %}

When "Container Collect All" is enabled you can configure which containers you want to collect logs from. This can be useful to prevent the collection of the Datadog Agent logs, if desired. You can do this by passing configurations to the Datadog Agent to control what it pulls, or by passing configurations to the Kubernetes Pod to exclude certain logs more explicitly.

Exclusions such as `DD_CONTAINER_EXCLUDE_LOGS` or `ad.datadoghq.com/logs_exclude` take precedence over explicit log collection configuration, including configuration from the [`DatadogInstrumentation` CRD](https://docs.datadoghq.com/containers/guide/configure-autodiscovery-with-the-datadoginstrumentation-crd.md), [Autodiscovery annotations](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-annotations), and [Autodiscovery configuration files](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-configuration-files).

When "Container Collect All" is disabled (the default), the Agent collects logs only from containers that have an explicit log configuration. To collect logs from selected workloads, configure them with the [`DatadogInstrumentation` CRD](https://docs.datadoghq.com/containers/guide/configure-autodiscovery-with-the-datadoginstrumentation-crd.md), [Autodiscovery annotations](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-annotations), or [Autodiscovery configuration files](https://docs.datadoghq.com/containers/kubernetes/log.md?tab=helm#autodiscovery-configuration-files).

See [Container Discovery Management](https://docs.datadoghq.com/agent/guide/autodiscovery-management.md) to learn more about filtering.

### Tagging{% #tagging %}

The Datadog Agent tags the logs from the Kubernetes containers with the default [Kubernetes tags](https://docs.datadoghq.com/containers/kubernetes/tag.md), as well as any custom extracted tags. When "Container Collect All" is enabled, the Agent reports the logs for a container with a `source` and `service` tag matching the container short image name. For example, the logs from a container using the `gcr.io/owner/example-image:latest` container image would have `example-image` as the `source`, `service`, and `short_image` tag value.

The `service` tag can also be set by the [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md?tab=kubernetes) Pod label `tags.datadoghq.com/service: "<SERVICE>"`. For more information about `source` and `service` attributes, see [Reserved Attributes](https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention.md).

The `source` tag can be important for your logs, as the [out of box log pipelines](https://docs.datadoghq.com/logs/log_configuration/pipelines.md?tab=source#integration-pipelines) are filtered using this tag. However, these pipelines can be completely customized as desired. You can see the steps in the Integration Logs section below for customizing the tags on your logs further.

## Integration logs{% #integration-logs %}

[Autodiscovery](https://docs.datadoghq.com/getting_started/containers/autodiscovery.md) enables you to use templates to configure log collection and other capabilities on containers. Use one of the following methods to configure log collection:

- `DatadogInstrumentation` CRD (recommended)
- Autodiscovery annotations
- Autodiscovery configuration files

Setting a `source` and `service` tag on these log configurations is strongly recommended. Match the `source` tag to one of Datadog's [out-of-the-box log pipelines](https://docs.datadoghq.com/logs/log_configuration/pipelines.md?tab=source#integration-pipelines) so your logs are automatically enriched; you can also find a [library of pipelines in Datadog](https://app.datadoghq.com/logs/pipelines/pipeline/library). The `service` tag powers [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md?tab=kubernetes), linking your logs with metrics and traces from the same service. If `source` and `service` are omitted, the Agent falls back to the `service` tag from Unified Service Tagging (when set), and otherwise to the container's short image name.

### `DatadogInstrumentation` CRD{% #datadoginstrumentation-crd %}

Use a `DatadogInstrumentation` resource to configure log collection without modifying application pod templates. This method requires Agent log collection to be enabled. Create the resource in the same namespace as the target workload.

The following example collects logs from the `app` container in the `example` Deployment:

```yaml
apiVersion: datadoghq.com/v1alpha1
kind: DatadogInstrumentation
metadata:
  name: java-logs
  namespace: <WORKLOAD_NAMESPACE>
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example
  config:
    logs:
      - containerName: app
        source: java
        service: example-app
        tags:
          - foo:bar
```

Save the resource as `java-logs.yaml`, apply it, and check its status:

```shell
kubectl apply -f java-logs.yaml
kubectl describe datadoginstrumentation java-logs -n <WORKLOAD_NAMESPACE>
```

#### Configure multiple containers{% #configure-multiple-containers %}

To collect logs from both the `app` and `proxy` containers in the same Deployment, replace the `logs` section under `spec.config` with:

```yaml
logs:
  - containerName: app
    source: java
    service: example-app
    tags:
      - foo:bar
  - containerName: proxy
    source: nginx
    service: example-proxy
```

Apply the updated resource to change log collection without restarting application pods.

### Autodiscovery annotations{% #autodiscovery-annotations %}

With Autodiscovery, the Agent automatically searches all Pod annotations for integration templates.

To apply a specific configuration to a given container, add the annotation `ad.datadoghq.com/<CONTAINER_NAME>.logs` to your Pod with the JSON formatted log configuration.

**Note**: Autodiscovery annotations identify containers by name, **not** image. It tries to match `<CONTAINER_NAME>` to the `.spec.containers[i].name`, not `.spec.containers[i].image`.

{% alert level="info" %}
If you define your Kubernetes Pods directly (with `kind:Pod`), add each Pod's annotations in its `metadata` section, as shown in the following section.  If you define your Kubernetes Pods indirectly (with replication controllers, ReplicaSets, or Deployments), add Pod annotations to the Pod template under `.spec.template.metadata`.
{% /alert %}

#### Configure a single container{% #configure-a-single-container %}

To configure log collection for a given container within your Pod, add the following annotations to your Pod:

```yaml
apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/<CONTAINER_NAME>.logs: '[<LOG_CONFIG>]'
    # (...)
spec:
  containers:
    - name: '<CONTAINER_NAME>'
# (...)
```

#### Example log Autodiscovery annotations{% #example-log-autodiscovery-annotations %}

The following Pod annotation defines the integration template for an example container. It is defined within the Pod template's annotations, rather than on the Deployment itself. This log configuration sets all the logs from the `app` container with the tags `source:java`, `service:example-app`, and the extra tag `foo:bar`.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  labels:
    app: example-app
spec:
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
      annotations:
        ad.datadoghq.com/app.logs: '[{"source":"java", "service":"example-app", "tags":["foo:bar"]}]'
    spec:
      containers:
        - name: app
          image: owner/example-image:latest
```

#### Configure two different containers{% #configure-two-different-containers %}

To apply two different integration templates to two different containers within your Pod, `<CONTAINER_NAME_1>` and `<CONTAINER_NAME_2>`, add the following annotations to your Pod:

```yaml
apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/<CONTAINER_NAME_1>.logs: '[<LOG_CONFIG_1>]'
    # (...)
    ad.datadoghq.com/<CONTAINER_NAME_2>.logs: '[<LOG_CONFIG_2>]'
spec:
  containers:
    - name: '<CONTAINER_NAME_1>'
    # (...)
    - name: '<CONTAINER_NAME_2>'
# (...)
```

### Autodiscovery configuration files{% #autodiscovery-configuration-files %}

You can provide the Datadog Agent with configuration files to have the Agent run a specified integration when it discovers a container using the matching image identifier. This allows you to create a generic log configuration that applies to a set of container images.

{% tab title="Datadog Operator" %}
You can customize logs collection per integration with an override in the `override.nodeAgent.extraConfd.configDataMap`. This method creates the ConfigMap and mounts the desired configuration file onto the Agent container.

```yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      extraConfd:
        configDataMap:
          <INTEGRATION_NAME>.yaml: |-
            ad_identifiers:
            - <CONTAINER_IMAGE>

            logs:
            - source: example-source
              service: example-service
```

The `<CONTAINER_IMAGE>` should match the container short image name that you want this to apply towards. See the sample manifest [with ConfigMap mapping](https://github.com/DataDog/datadog-operator/blob/main/examples/datadogagent/datadog-agent-with-extraconfd.yaml) for an additional example.
{% /tab %}

{% tab title="Helm" %}
You can customize logs collection per integration within `datadog.confd`. This method creates the ConfigMap and mounts the desired configuration file onto the Agent container.

```yaml
datadog:
  #(...)
  confd:
    <INTEGRATION_NAME>.yaml: |-
      ad_identifiers:
      - <CONTAINER_IMAGE>
      logs:
      - source: example-source
        service: example-service
```

The `<CONTAINER_IMAGE>` should match the container short image name that you want this to apply towards.
{% /tab %}

{% tab title="Key-value store" %}
The following etcd commands create a Redis integration template with a custom `password` parameter and tags logs with the correct `source` and `service` attributes:

```
etcdctl mkdir /datadog/check_configs/redis
etcdctl set /datadog/check_configs/redis/logs '[{"source": "redis", "service": "redis", "tags": ["env:prod"]}]'
```

Notice that each of the three values is a list. Autodiscovery assembles list items into the integration configurations based on shared list indexes. In this case, it composes the first (and only) check configuration from `check_names[0]`, `init_configs[0]`, and `instances[0]`.

Unlike auto-conf files, **key-value stores may use the short OR long image name as container identifiers**, for example, `redis` OR `redis:latest`.

Autodiscovery can use [Consul](https://docs.datadoghq.com/integrations/consul.md), Etcd, and Zookeeper as integration template sources.

To use a key-value store, configure it in the Agent `datadog.yaml` configuration file and mount this file inside the containerized Agent. Alternatively, pass your key-value store as environment variables to the containerized Agent.

#### In `datadog.yaml`{% #in-datadogyaml %}

In the `datadog.yaml` file, set the `<KEY_VALUE_STORE_IP>` address and `<KEY_VALUE_STORE_PORT>` of your key-value store:

```yaml
config_providers:
  - name: etcd
    polling: true
    template_dir: /datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    username:
    password:

  - name: consul
    polling: true
    template_dir: datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    ca_file:
    ca_path:
    cert_file:
    key_file:
    username:
    password:
    token:

  - name: zookeeper
    polling: true
    template_dir: /datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    username:
    password:
```

Then [restart the Agent](https://docs.datadoghq.com/agent/configuration/agent-commands.md) to apply the configuration change.

#### In environment variables{% #in-environment-variables %}

With the key-value store enabled as a template source, the Agent looks for templates under the key `/datadog/check_configs`. Autodiscovery expects a key-value hierarchy like this:

```yaml
/datadog/
  check_configs/
    <CONTAINER_IMAGE>/
      - logs: ["<LOGS_CONFIG>"]
    ...
```

**Note**: To apply a specific configuration to a given container, Autodiscovery identifies containers by **image** when using the key-value stores by trying to match `<CONTAINER_IMAGE>` to `.spec.containers[0].image`.
{% /tab %}

To match a log configuration to a set of containers with more granularity than the container short image name, see [Autodiscovery Container Identifiers](https://docs.datadoghq.com/containers/guide/ad_identifiers.md).

## Advanced log collection{% #advanced-log-collection %}

Use `spec.config.logs` in a `DatadogInstrumentation` resource or Autodiscovery log annotations to apply advanced log processing, for example:

- [Filter logs before sending them to Datadog](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md?tab=annotations#filter-logs).
- [Scrub sensitive data from your logs](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md?tab=annotations#scrub-sensitive-data-from-your-logs).
- [Proceed to multi-line aggregation](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md?tab=annotations#manually-aggregate-multi-line-logs).

### From a container local log file{% #from-a-container-local-log-file %}

Datadog recommends that you use the `stdout` and `stderr` output streams for containerized applications, so that you can more automatically set up log collection.

To collect logs from a file, set `type: file` and `path` in a `DatadogInstrumentation` log entry or an Autodiscovery log annotation. The Agent adds the same container tags as it does for logs from the container output streams. For additional guidance, see the Recommended configurations section.

These file paths are **relative** to the Agent container. Therefore, the directory containing the log file needs to be mounted into both the application and Agent container so the Agent can have proper visibility.

For example, use a shared `hostPath` volume for an application that writes to `/var/log/example/app.log`:

{% tab title="DatadogInstrumentation CRD" %}
Apply the following resource in the namespace of the targeted workload. The `logger` workload must also write its logs to `/var/log/example` on a `hostPath` volume, as shown in the **Annotations** tab:

```yaml
apiVersion: datadoghq.com/v1alpha1
kind: DatadogInstrumentation
metadata:
  name: logger-logs
  namespace: <WORKLOAD_NAMESPACE>
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: logger
  config:
    logs:
      - containerName: busybox
        type: file
        path: /var/log/example/app.log
        source: example-source
        service: example-service
```

{% /tab %}

{% tab title="Annotations" %}

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: logger
  annotations:
    ad.datadoghq.com/busybox.logs: |
      [{
          "type": "file",
          "path": "/var/log/example/app.log",
          "source": "example-source",
          "service": "example-service"
      }]
spec:
  containers:
   - name: busybox
     image: busybox
     command: [ "/bin/sh", "-c", "--" ]
     args: [ "while true; do sleep 1; echo `date` example file log >> /var/log/example/app.log; done;" ]
     volumeMounts:
     - name: applogs
       mountPath: /var/log/example
  volumes:
     - name: applogs
       hostPath:
         path: /var/log/example
```

{% /tab %}

For either method, mount the same host directory into the Agent container so it can read the log file:

```yaml
  containers:
  - name: agent
    # (...)
    volumeMounts:
    - mountPath: /var/log/example
      name: applogs
    # (...)
  volumes:
  - name: applogs
    hostPath:
      path: /var/log/example
    # (...)
```

#### Recommended configurations{% #recommended-configurations %}

- This strategy can work for a given pod, but can become cumbersome with multiple apps using this strategy. You can also run into issues if multiple replicas are using the same log path. If possible, Datadog recommends taking advantage of the [Autodiscovery template variable](https://docs.datadoghq.com/containers/guide/template_variables.md) `%%kube_pod_name%%`. For example, you can set your `path` to reference this variable: `"path": "/var/log/example/%%kube_pod_name%%/app.log"`. Your application pod then needs to write its log files with respect to this new path as well. You can use the [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) to help your application determine its Pod name.

- File log configurations replace automatic collection of `stdout` and `stderr` for that container. To collect both, add a second log entry for the container output streams. For `DatadogInstrumentation`, use the following `spec.config.logs` entries:

  ```yaml
  logs:
    - containerName: busybox
      type: file
      path: /var/log/example/app.log
      source: file
      service: example-service
    - containerName: busybox
      source: container
      service: example-service
  ```

For annotations, use:

  ```yaml
  ad.datadoghq.com/<CONTAINER_NAME>.logs: |
    [
      {"type":"file","path":"/var/log/example/app.log","source":"file","service":"example-service"},
      {"source":"container","service":"example-service"}
    ]
  ```

- Set `source` and `service` explicitly for file log entries because they have no default values.

## Troubleshooting{% #troubleshooting %}

For troubleshooting steps, see [Container Log Collection Troubleshooting](https://docs.datadoghq.com/containers/troubleshooting/log-collection.md?tab=datadogoperator).

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Collect your application traces](https://docs.datadoghq.com/agent/kubernetes/apm.md)
- [Collect your Prometheus metrics](https://docs.datadoghq.com/agent/kubernetes/prometheus.md)
- [Collect automatically your applications metrics and logs](https://docs.datadoghq.com/agent/kubernetes/integrations.md)
- [Limit data collection to a subset of containers only](https://docs.datadoghq.com/agent/guide/autodiscovery-management.md)
- [Assign tags to all data emitted by a container](https://docs.datadoghq.com/agent/kubernetes/tag.md)
- [Container Log Collection Troubleshooting](https://docs.datadoghq.com/containers/troubleshooting/log-collection.md)
- [Monitor logs from Amazon EKS on Fargate with Datadog](https://www.datadoghq.com/blog/eks-fargate-logs-datadog)
- [Manage Kubernetes observability with DatadogInstrumentation](https://www.datadoghq.com/blog/manage-kubernetes-observability-with-datadoginstrumentation/)
- [Monitoring Container Apps - Logs](https://www.datadoghq.com/architecture/monitoring-container-apps-logs/)
