Kubernetes Audit Logs

Kubernetes Audit Logs

Linux Mac OS Windows OS Supported


Collect Kubernetes audit logs to track everything that happens inside your Kubernetes clusters, including every call made to the Kubernetes API by any service. This includes the control plane (built-in controllers, the scheduler), node daemons (the kubelet, kube-proxy, and others), cluster services (e.g., the cluster autoscaler), users making kubectl requests, and even the Kubernetes API itself.

With the Kubernetes audit logs integration, you can diagnose permission issues, identify RBAC policies that need to be updated, and track slow API requests that are impacting your whole cluster. Deep dive into these topics with the Datadog talk at KubeCon 2019.


This integration is available for Agent >6.0


For more information about setting up Kubernetes audit logs, refer to the official Kubernetes documentation.

To enable audit logs in Kubernetes:

  1. Audit logs are disabled by default in Kubernetes. To enable them in your API server configuration, specify an audit policy file path:

  2. Create the policy file at /etc/kubernetes/audit-policies/policy.yaml to specify the types of API requests you want to capture in your audit logs. Audit policy rules are evaluated in order. The API server follows the first matching rule it finds for each type of operation or resource. Example of an audit policy:

# /etc/kubernetes/audit-policies/policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
    # do not log requests to the following
    - level: None
          - '/healthz*'
          - '/logs'
          - '/metrics'
          - '/swagger*'
          - '/version'

    # limit level to Metadata so token is not included in the spec/status
    - level: Metadata
          - RequestReceived
          - group: authentication.k8s.io
                - tokenreviews

    # extended audit of auth delegation
    - level: RequestResponse
          - RequestReceived
          - group: authorization.k8s.io
                - subjectaccessreviews

    # log changes to pods at RequestResponse level
    - level: RequestResponse
          - RequestReceived
          # core API group; add third-party API services and your API services if needed
          - group: ''
            resources: ['pods']
            verbs: ['create', 'patch', 'update', 'delete']

    # log everything else at Metadata level
    - level: Metadata
          - RequestReceived

This example policy file configures the API server to log at the highest level of detail for certain types of cluster-changing operations (update, patch, create, delete). It also tracks requests to the subjectaccessreviews resource at the highest level to help troubleshoot authentication delegation issues.

You may want to reduce the level of verbosity to Metadata for endpoints that contain sensitive data (e.g., tokenreviews resource). Datadog also omits the RequestReceived stage from logs.

In the last section, for everything that was not explicitly configured by the previous rules, the policy is configured to log at Metadata level. As audit logs might be verbose, you can choose to exclude less critical actions/verbs (e.g., operations that don’t change the cluster state like list, watch, and get).

Log collection

  1. Install the Agent on your Kubernetes environment.

  2. Log collection is disabled by default. Enable it in the env section of your DaemonSet:

        # (...)
        - name: DD_LOGS_ENABLED
          value: 'true'
  3. Mount the audit log directory as well as a directory that the Agent uses to store a pointer to know which log was last sent from that file. To do this, add the following in the volumeMounts section of the daemonset:

     # (...)
          # (...)
          - name: pointdir
            mountPath: /opt/datadog-agent/run
          - name: auditdir
            mountPath: /var/log/kubernetes/apiserver
          - name: dd-agent-config
            mountPath: /conf.d/kubernetes_audit.d
      # (...)
        # (...)
        - hostPath:
            path: /opt/datadog-agent/run
          name: pointdir
        - hostPath:
            path: /var/log/kubernetes/apiserver
          name: auditdir
        - name: dd-agent-config
              name: dd-agent-config
                - key: kubernetes-audit-log
                  path: conf.yaml
      # (...)

    This also mounts the conf.d folder which is used to configure the Agent to collect logs from the audit log file.

  4. Configure the Agent to collect logs from that file with a ConfigMap:

    kind: ConfigMap
    apiVersion: v1
        name: dd-agent-config
        namespace: default
        kubernetes-audit-log: |-
              - type: file
                path: /var/log/kubernetes/apiserver/audit.log
                source: kubernetes.audit
                service: audit        


Run the Agent’s status subcommand and look for Logs under the Checks section.


Need help? Contact Datadog support.

Further Reading