This page discusses collecting logs from Kubernetes log files.

When your containerized applications write their logs to standard output and error (stdout/stderr), the container runtime and Kubernetes automatically manage the logs for you. The default pattern is that Kubernetes stores these log streams as files on the host in the /var/log/pods folder and subfolders for each Pod and container.

The Datadog Agent can collect these Kubernetes log files for these containers using the instructions below. This option scales well for the ephemeral nature of the Pods that Kubernetes creates, and is more resource-efficient than collecting logs from the Docker socket. Datadog recommends this method for log collection in Kubernetes.

Alternatively, the Datadog Agent can also collect logs by repeated requests to the Docker API through the Docker socket. However, this requires Docker as the container runtime for your Kubernetes cluster. This is also more resource-intensive than using log files. To see how to collect logs using the Docker socket, see Log collection with Docker socket. If your containerized applications are writing to log files stored in the container, this can complicate log collection. See log collection from a file.

Setup

Log collection

Before you start collecting application logs, ensure that you are running the Datadog Agent in your Kubernetes cluster.

To configure log collection manually in the DaemonSet, see DaemonSet Log Collection. Otherwise, follow the instructions below:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>

  features:
    logCollection:
      enabled: true
      containerCollectAll: true

kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml

datadog:
  logs:
    enabled: true
    containerCollectAll: true

Unprivileged

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>

  features:
    logCollection:
      enabled: true
      containerCollectAll: true

  override:
    nodeAgent:
      securityContext:
        runAsUser: <USER_ID>
        supplementalGroups:
          - <DOCKER_GROUP_ID>

datadog:
  securityContext:
    runAsUser: <USER_ID>
    supplementalGroups:
      - <DOCKER_GROUP_ID>

Log discovery

The Datadog Agent in Kubernetes is deployed by a DaemonSet (managed by the Datadog Operator or Helm). This DaemonSet schedules one replica of the Agent Pod on each node of the cluster. Each Agent Pod is then responsible for reporting the logs of the other Pods and containers on its respective node. When the “Container Collect All” feature is enabled, the Agent reports the logs from every discovered container with a default set of tags.

Filtering

You can configure which containers you want to collect logs from. This can be useful to prevent the collection of the Datadog Agent logs, if desired. You can do this by passing configurations to the Datadog Agent to control what it pulls, or by passing configurations to the Kubernetes Pod to exclude certain logs more explicitly.

See Container Discovery Management to learn more.

Tagging

The Datadog Agent tags the logs from the Kubernetes containers with the default Kubernetes tags, as well as any custom extracted tags. When “Container Collect All” is enabled, the Agent reports the logs for a container with a source and service tag matching the container short image name. For example, the logs from a container using the gcr.io/owner/example-image:latest container image would have example-image as the source, service, and short_image tag value.

The service tag can also be set by the Unified Service Tagging Pod label tags.datadoghq.com/service: "<SERVICE>". For more information about source and service attributes, see Reserved Attributes.

The source tag can be important for your logs, as the out of box log pipelines are filtered using this tag. However, these pipelines can be completely customized as desired. You can see the steps in the Integration Logs section below for customizing the tags on your logs further.

Integration logs

Autodiscovery enables you to use templates to configure log collection (and other capabilities) on containers. This can be used to enable log collection, customize tagging, and add advanced collection rules. To configure log collection for an integration with Autodiscovery you can either:

• Specify a log configuration as Autodiscovery Annotations on a given Pod, to configure the rules for a given container (Recommended)
• Specify a log configuration as a configuration file, to configure the rules for each matching container by image

At minimum, these log configurations require a source and service tag. You may want to match the source tag to one of Datadog’s out-of-the-box log pipelines to help automatically enrich your logs. You can also find a library of pipelines in Datadog.

Autodiscovery annotations

With Autodiscovery, the Agent automatically searches all Pod annotations for integration templates.

To apply a specific configuration to a given container, add the annotation ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs to your Pod with the JSON formatted log configuration.

Note: Autodiscovery annotations identify containers by name, not image. It tries to match <CONTAINER_IDENTIFIER> to the .spec.containers[i].name, not .spec.containers[i].image.

Configure a single container

To configure log collection for a given <CONTAINER_IDENTIFIER> within your Pod, add the following annotations to your Pod:

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs: '[<LOG_CONFIG>]'
    # (...)
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'
# (...)

Example log Autodiscovery annotations

The following Pod annotation defines the integration template for an example container. It is defined within the Pod template’s annotations, rather than on the Deployment itself. This log configuration sets all the logs from the app container with the tags source:java, service:example-app, and the extra tag foo:bar.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  labels:
    app: example-app
spec:
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
      annotations:
        ad.datadoghq.com/app.logs: '[{"source":"java", "service":"example-app", "tags":["foo:bar"]}]'
    spec:
      containers:
        - name: app
          image: owner/example-image:latest

Configure two different containers

To apply two different integration templates to two different containers within your Pod, <CONTAINER_IDENTIFIER_1> and <CONTAINER_IDENTIFIER_2>, add the following annotations to your Pod:

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER_1>.logs: '[<LOG_CONFIG_1>]'
    # (...)
    ad.datadoghq.com/<CONTAINER_IDENTIFIER_2>.logs: '[<LOG_CONFIG_2>]'
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER_1>'
    # (...)
    - name: '<CONTAINER_IDENTIFIER_2>'
# (...)

Autodiscovery configuration files

You can provide the Datadog Agent with configuration files to have the Agent run a specified integration when it discovers a container using the matching image identifier. This allows you to create a generic log configuration that applies to a set of container images.

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      extraConfd:
        configDataMap:
          <INTEGRATION_NAME>.yaml: |-
            ad_identifiers:
            - <INTEGRATION_AUTODISCOVERY_IDENTIFIER>            
        
            logs:
            - source: example-source
              service: example-service

datadog:
  #(...)
  confd:
    <INTEGRATION_NAME>.yaml: |-
      ad_identifiers:
      - <INTEGRATION_AUTODISCOVERY_IDENTIFIER>
      
      logs:
      - source: example-source
        service: example-service      

etcdctl mkdir /datadog/check_configs/redis
etcdctl set /datadog/check_configs/redis/logs '[{"source": "redis", "service": "redis", "tags": ["env:prod"]}]'

config_providers:
  - name: etcd
    polling: true
    template_dir: /datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    username:
    password:

  - name: consul
    polling: true
    template_dir: datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    ca_file:
    ca_path:
    cert_file:
    key_file:
    username:
    password:
    token:

  - name: zookeeper
    polling: true
    template_dir: /datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    username:
    password:

/datadog/
  check_configs/
    <CONTAINER_IDENTIFIER>/
      - logs: ["<LOGS_CONFIG>"]
    ...

Advanced log collection

Use Autodiscovery log labels to apply advanced log collection processing logic, for example:

• Filter logs before sending them to Datadog.
• Scrub sensitive data from your logs.
• Proceed to multi-line aggregation.

From a container local log file

Datadog recommends that you use the stdout and stderr output streams for containerized applications, so that you can more automatically set up log collection.

However, the Agent can also directly collect logs from a file based on an annotation. To collect these logs, use ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs with a type: file and path configuration. Logs collected from files with such an annotation are automatically tagged with the same set of tags as logs coming from the container itself. Datadog recommends that you use the stdout and stderr output streams for containerized applications, so that you can automatically set up log collection. For more information, see the Recommended configurations.

These file paths are relative to the Agent container. Therefore, the directory containing the log file needs to be mounted into both the application and Agent container so the Agent can have proper visibility.

For example, you can do this with a shared hostPath volume. The Pod below is emitting logs into the file /var/log/example/app.log. This is done in the /var/log/example directory, where a volume and volumeMount have set this as a hostPath.

apiVersion: v1
kind: Pod
metadata:
  name: logger
  annotations:
    ad.datadoghq.com/busybox.logs: |
      [{
          "type": "file",
          "path": "/var/log/example/app.log",
          "source": "example-source",
          "service": "example-service"
      }]      
spec:
  containers:
   - name: busybox
     image: busybox
     command: [ "/bin/sh", "-c", "--" ]
     args: [ "while true; do sleep 1; echo `date` example file log >> /var/log/example/app.log; done;" ]
     volumeMounts:
     - name: applogs
       mountPath: /var/log/example
  volumes:
     - name: applogs
       hostPath:
         path: /var/log/example

The equivalent volume and volumeMount path need to be set in the Agent container so it can read that same log file.

  containers:
  - name: agent
    # (...)
    volumeMounts:
    - mountPath: /var/log/example
      name: applogs
    # (...)
  volumes:
  - name: applogs
    hostPath:
      path: /var/log/example
    # (...)

Recommended configurations

• This strategy can work for a given pod, but can become cumbersome with multiple apps using this strategy. You can also run into issues if multiple replicas are using the same log path. If possible, Datadog recommends taking advantage of the Autodiscovery template variable %%kube_pod_name%%. For example, you can set your path to reference this variable: "path": "/var/log/example/%%kube_pod_name%%/app.log". Your application pod then needs to write its log files with respect to this new path as well. You can use the Downward API to help your application determine its Pod name.

• When using this kind of annotation with a container, stdout and stderr logs are not collected automatically from the container. If collection from both the container output streams and file are needed, explicitly enable this in the annotation. For example:

  ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs: |
    [
      {"type":"file","path":"/var/log/example/app.log","source":"file","service":"example-service"},
      {"source":"container","service":"example-service"}
    ]  
  

• When using this kind of combination, source and service have no default value for logs collected from a file and should be explicitly set in the annotation.

Troubleshooting

Short lived containers

By default the Agent looks every 5 seconds for new containers.

For Agent v6.12+, short lived container logs (stopped or crashed) are automatically collected when using the K8s file log collection method (through /var/log/pods). This also includes the collection init container logs.

Missing tags on new containers or Pods

When sending logs to Datadog from newly created containers or Pods, the Datadog Agent’s internal tagger may not yet have the related container/pod tags. As a result, tags may be missing from these logs.

To remediate this issue, you can use the environment variable DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION to configure a duration (in seconds) for the Datadog Agent to wait before it begins to send logs from newly created containers and Pods. The default value is 0.

spec:
  override:
    nodeAgent:
      env:
        - name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
          value: "5"

datadog:
  env:
    - name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
      value: "5"

Missing host-level tags on new hosts or nodes

Host-level tags are those seen in the infrastructure list for a given host, and are sourced from either a cloud provider or the Datadog Agent. Common host-level tags include kube_cluster_name, region, instance-type, and autoscaling-group.

When sending logs to Datadog from a newly created host or node, it can take a few minutes for host-level tags to be inherited. As a result, host-level tags may be missing from these logs.

To remediate this issue, you can use the environment variable DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION to configure a duration (in minutes). For this duration, the Datadog Agent manually attaches the host-level tags that it knows about to each sent log. After this duration, the Agent reverts to relying on tag inheritance at intake.

spec:
  override:
    nodeAgent:
      env:
        - name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
          value: "10m"

datadog:
  env:
    - name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
      value: "10m"

Further Reading

Additional helpful documentation, links, and articles:

Collect your application tracesDOCUMENTATION Collect your Prometheus metricsDOCUMENTATION Collect automatically your applications metrics and logsDOCUMENTATION Limit data collection to a subset of containers onlyDOCUMENTATION Assign tags to all data emitted by a containerDOCUMENTATION