このページでは、Kubernetes ログファイルからのログの収集について説明します。

When your containerized applications write their logs to standard output and error (stdout/stderr), the container runtime and Kubernetes automatically manage the logs for you. The default pattern is that Kubernetes stores these log streams as files on the host in the /var/log/pods folder and subfolders for each Pod and container.

The Datadog Agent can collect these Kubernetes log files for these containers using the instructions below. This option scales well for the ephemeral nature of the Pods that Kubernetes creates, and is more resource-efficient than collecting logs from the Docker socket. Datadog recommends this method for log collection in Kubernetes.

Alternatively, the Datadog Agent can also collect logs by repeated requests to the Docker API through the Docker socket. However, this requires Docker as the container runtime for your Kubernetes cluster. This is also more resource-intensive than using log files. To see how to collect logs using the Docker socket, see Log collection with Docker socket. If your containerized applications are writing to log files stored in the container, this can complicate log collection. See log collection from a file.

セットアップ

ログ収集

Before you start collecting application logs, ensure that you are running the Datadog Agent in your Kubernetes cluster.

To configure log collection manually in the DaemonSet, see DaemonSet Log Collection. Otherwise, follow the instructions below:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>

  features:
    logCollection:
      enabled: true
      containerCollectAll: true

kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml

datadog:
  logs:
    enabled: true
    containerCollectAll: true

非特権

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>

  features:
    logCollection:
      enabled: true
      containerCollectAll: true

  override:
    nodeAgent:
      securityContext:
        runAsUser: <USER_ID>
        supplementalGroups:
          - <DOCKER_GROUP_ID>

datadog:
  securityContext:
    runAsUser: <USER_ID>
    supplementalGroups:
      - <DOCKER_GROUP_ID>

Log discovery

The Datadog Agent in Kubernetes is deployed by a DaemonSet (managed by the Datadog Operator or Helm). This DaemonSet schedules one replica of the Agent Pod on each node of the cluster. Each Agent Pod is then responsible for reporting the logs of the other Pods and containers on its respective node. When the “Container Collect All” feature is enabled, the Agent reports the logs from every discovered container with a default set of tags.

フィルタリング

You can configure which containers you want to collect logs from. This can be useful to prevent the collection of the Datadog Agent logs, if desired. You can do this by passing configurations to the Datadog Agent to control what it pulls, or by passing configurations to the Kubernetes Pod to exclude certain logs more explicitly.

See Container Discovery Management to learn more.

タグ付け

The Datadog Agent tags the logs from the Kubernetes containers with the default Kubernetes tags, as well as any custom extracted tags. When “Container Collect All” is enabled, the Agent reports the logs for a container with a source and service tag matching the container short image name. For example, the logs from a container using the gcr.io/owner/example-image:latest container image would have example-image as the source, service, and short_image tag value.

The service tag can also be set by the Unified Service Tagging Pod label tags.datadoghq.com/service: "<SERVICE>". For more information about source and service attributes, see Reserved Attributes.

The source tag can be important for your logs, as the out of box log pipelines are filtered using this tag. However, these pipelines can be completely customized as desired. You can see the steps in the Integration Logs section below for customizing the tags on your logs further.

インテグレーションログ

Autodiscovery enables you to use templates to configure log collection (and other capabilities) on containers. This can be used to enable log collection, customize tagging, and add advanced collection rules. To configure log collection for an integration with Autodiscovery you can either:

• Specify a log configuration as Autodiscovery Annotations on a given Pod, to configure the rules for a given container (Recommended)
• Specify a log configuration as a configuration file, to configure the rules for each matching container by image

At minimum, these log configurations require a source and service tag. You may want to match the source tag to one of Datadog’s out-of-the-box log pipelines to help automatically enrich your logs. You can also find a library of pipelines in Datadog.

オートディスカバリーアノテーション

With Autodiscovery, the Agent automatically searches all Pod annotations for integration templates.

To apply a specific configuration to a given container, add the annotation ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs to your Pod with the JSON formatted log configuration.

Note: Autodiscovery annotations identify containers by name, not image. It tries to match <CONTAINER_IDENTIFIER> to the .spec.containers[i].name, not .spec.containers[i].image.

単一コンテナの構成

To configure log collection for a given <CONTAINER_IDENTIFIER> within your Pod, add the following annotations to your Pod:

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<ポッド名>'
  annotations:
    ad.datadoghq.com/<コンテナ識別子>.logs: '[<ログ_コンフィグ>]'
    # (...)
spec:
  containers:
    - name: '<コンテナ識別子>'
# (...)

Example log Autodiscovery annotations

The following Pod annotation defines the integration template for an example container. It is defined within the Pod template’s annotations, rather than on the Deployment itself. This log configuration sets all the logs from the app container with the tags source:java, service:example-app, and the extra tag foo:bar.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  labels:
    app: example-app
spec:
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
      annotations:
        ad.datadoghq.com/app.logs: '[{"source":"java", "service":"example-app", "tags":["foo:bar"]}]'
    spec:
      containers:
        - name: app
          image: owner/example-image:latest

2 つの異なるコンテナの構成

To apply two different integration templates to two different containers within your Pod, <CONTAINER_IDENTIFIER_1> and <CONTAINER_IDENTIFIER_2>, add the following annotations to your Pod:

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<ポッド名>'
  annotations:
    ad.datadoghq.com/<コンテナ識別子_1>.logs: '[<ログ_コンフィグ_1>]'
    # (...)
    ad.datadoghq.com/<コンテナ識別子_2>.logs: '[<ログ_コンフィグ_2>]'
spec:
  containers:
    - name: '<コンテナ識別子_1>'
    # (...)
    - name: '<コンテナ識別子_2>'
# (...)

オートディスカバリーコンフィギュレーションファイル

You can provide the Datadog Agent with configuration files to have the Agent run a specified integration when it discovers a container using the matching image identifier. This allows you to create a generic log configuration that applies to a set of container images.

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      extraConfd:
        configDataMap:
          <INTEGRATION_NAME>.yaml: |-
            ad_identifiers:
            - <INTEGRATION_AUTODISCOVERY_IDENTIFIER>

            logs:
            - source: example-source
              service: example-service            

datadog:
  #(...)
  confd:
    <INTEGRATION_NAME>.yaml: |-
      ad_identifiers:
      - <INTEGRATION_AUTODISCOVERY_IDENTIFIER>

      logs:
      - source: example-source
        service: example-service      

etcdctl mkdir /datadog/check_configs/redis
etcdctl set /datadog/check_configs/redis/logs '[{"source": "redis", "service": "redis", "tags": ["env:prod"]}]'

config_providers:
  - name: etcd
    polling: true
    template_dir: /datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    username:
    password:

  - name: consul
    polling: true
    template_dir: datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    ca_file:
    ca_path:
    cert_file:
    key_file:
    username:
    password:
    token:

  - name: zookeeper
    polling: true
    template_dir: /datadog/check_configs
    template_url: '<KV_STORE_IP>:<KV_STORE_PORT>'
    username:
    password:

/datadog/
  check_configs/
    <コンテナ識別子>/
      - logs: ["<ログ_コンフィグ>"]
    ...

高度なログの収集

オートディスカバリーログラベルを使用し、高度なログ収集の処理ロジックを適用します。たとえば、

• Datadog へ送信する前にログを絞り込む。
• ログの機密データのスクラビング。
• 複数行の集約の実行。

From a container local log file

Datadog recommends that you use the stdout and stderr output streams for containerized applications, so that you can more automatically set up log collection.

However, the Agent can also directly collect logs from a file based on an annotation. To collect these logs, use ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs with a type: file and path configuration. Logs collected from files with such an annotation are automatically tagged with the same set of tags as logs coming from the container itself.

These file paths are relative to the Agent container. Therefore, the directory containing the log file needs to be mounted into both the application and Agent container so the Agent can have proper visibility.

例えば、共有の hostPath ボリュームを使用してこれを行うことができます。下記の Pod は /var/log/example/app.log というファイルにログを出力しています。これは /var/log/example ディレクトリで行われ、ボリュームと volumeMount がこれを hostPath として設定しています。

apiVersion: v1
kind: Pod
metadata:
  name: logger
  annotations:
    ad.datadoghq.com/busybox.logs: |
      [{
          "type": "file",
          "path": "/var/log/example/app.log",
          "source": "example-source",
          "service": "example-service"
      }]      
spec:
  containers:
   - name: busybox
     image: busybox
     command: [ "/bin/sh", "-c", "--" ]
     args: [ "while true; do sleep 1; echo `date` example file log >> /var/log/example/app.log; done;" ]
     volumeMounts:
     - name: applogs
       mountPath: /var/log/example
  volumes:
     - name: applogs
       hostPath:
         path: /var/log/example

Agent コンテナに同等のボリュームと VolumeMount パスを設定し、同じログファイルを読み込むことができるようにする必要があります。

  containers:
  - name: agent
    # (...)
    volumeMounts:
    - mountPath: /var/log/example
      name: applogs
    # (...)
  volumes:
  - name: applogs
    hostPath:
      path: /var/log/example
    # (...)

Note: This strategy can work for a given pod, but can become cumbersome with multiple apps using this strategy, as well run into issues if multiple replicas are using the same log path. If possible, Datadog recommends taking advantage of the Autodiscovery template variable %%kube_pod_name%%. For example, you can set your path to reference this variable: "path": "/var/log/example/%%kube_pod_name%%/app.log".

Your application pod then needs to write its log files with respect to this new path as well. You can use the Downward API to help your application determine its Pod name.

Note: When using this kind of annotation with a container, stdout and stderr logs are not collected automatically from the container. If collection from both the container output streams and file are needed, explicitly enable this in the annotation. For example:

ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs: |
  [
    {"type":"file","path":"/var/log/example/app.log","source":"file","service":"example-service"},
    {"source":"container","service":"example-service"}
  ]  

この種の組み合わせを使用する場合、source と service にはファイルから収集されたログのデフォルト値がないため、アノテーションで明示的に設定する必要があります。

トラブルシューティング

存続期間が短いコンテナ

デフォルトでは、Agent は 5 秒ごとに新しいコンテナを探します。

Agent v6.12+ では、K8s ファイルログ収集方法 (/var/log/pods 経由) を使用している場合、存続期間の短いコンテナのログ (停止またはクラッシュ) が自動的に収集されます。これには、収集初期化コンテナログも含まれます。

Missing tags on new containers or Pods

When sending logs to Datadog from newly created containers or Pods, the Datadog Agent’s internal tagger may not yet have the related container/pod tags. As a result, tags may be missing from these logs.

To remediate this issue, you can use the environment variable DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION to configure a duration (in seconds) for the Datadog Agent to wait before it begins to send logs from newly created containers and Pods. The default value is 0.

spec:
  override:
    nodeAgent:
      env:
        - name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
          value: "5"

datadog:
  env:
    - name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
      value: "5"

新しいホストまたはノードでホストレベルのタグが欠落している場合

ホストレベルタグは、特定のホストのインフラストラクチャーリストに表示されるタグで、クラウドプロバイダーまたは Datadog Agent のいずれかから供給されます。代表的なホストレベルタグは、kube_cluster_name、region、instance-type、autoscaling-group などです。

When sending logs to Datadog from a newly created host or node, it can take a few minutes for host-level tags to be inherited. As a result, host-level tags may be missing from these logs.

To remediate this issue, you can use the environment variable DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION to configure a duration (in minutes). For this duration, the Datadog Agent manually attaches the host-level tags that it knows about to each sent log. After this duration, the Agent reverts to relying on tag inheritance at intake.

spec:
  override:
    nodeAgent:
      env:
        - name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
          value: "10m"

datadog:
  env:
    - name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
      value: "10m"

その他の参考資料

お役に立つドキュメント、リンクや記事:

アプリケーショントレースの収集ドキュメント Prometheus メトリクスの収集ドキュメント アプリケーションのメトリクスとログを自動で収集ドキュメント データ収集をコンテナのサブセットのみに制限ドキュメント コンテナから送信された全データにタグを割り当てドキュメント