Logging is here!

Docker Log collection

Overview

Datadog Agent version 6 and greater can collect logs from containers. Two installations are possible:

  • on the host where the Agent is external to the Docker environment
  • or by deploying its containerized version in the Docker environment

You can then choose to collect all the logs from all your environment container or to filter by container image name or container label to cherry pick what logs should be collected.

Setup

Option 1: Host installation

Install the latest version of the Agent 6 on your host.

The Agent can both collect logs from files on the host or from container stdout/stderr.

To collect logs from all your container without filtering by image or label add the following at the end of docker.d/conf.yaml in your agent’s conf.d directory:

logs:
    - type: docker
      service: docker

Important note: Integration pipelines and processors will not be installed automatically as the source tag is not set. The integrations setup is described below and it automatically installs integration pipelines that parse your logs and extract all the relevant information from them.

Option 2: Container installation

As explained above, the Agent also has a containerized installation.

First let’s create one directories on the host that we will later mount on the containerized Agent:

  • /opt/datadog-agent/run: to make sure we do not lose any logs from containers during restarts or network issues we store on the host the last line that was collected for each container in this directory

To run a Docker container which embeds the Datadog Agent to monitor your host use the following command:

docker run -d --name datadog-agent \
           -e DD_API_KEY=<YOUR_API_KEY> \
           -e DD_LOGS_ENABLED=true \
           -e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
           -v /var/run/docker.sock:/var/run/docker.sock:ro \
           -v /proc/:/host/proc/:ro \
           -v /opt/datadog-agent/run:/opt/datadog-agent/run:rw \
           -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
           datadog/agent:latest

Important notes:

The command related to log collection are the following:

  • -e DD_LOGS_ENABLED=true: this parameter enables the log collection when set to true. The Agent now looks for log instructions in configuration files.
  • -e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true: this parameter add a log configuration that enabled log collection for all containers (see Option 1 below)
  • -v /opt/datadog-agent/run:/opt/datadog-agent/run:rw: mount the directory we created to store pointer on each container logs to make sure we do not lose any.

Activate Log Collection

Option 1: Configuration file

Set the source attribute to start using integrations. This allows Datadog to identify the log source for each container.

For containerized installation, begin by creating the configuration directory on the host directly:

  • /opt/datadog-agent/conf.d: this is the folder in which integration instructions are provided. Any configuration file added there is automatically picked up by the containerized Agent when restarted. To learn more about this topic refer to the dedicated documentation on how to enable integrations with the docker agent.

Then mount the directory by adding -v /opt/datadog-agent/conf.d:/conf.d:ro in the run command of the containerized agent.

To set the source for a given container filtered by image or label or name, update the log section in an integration or custom .yaml file in your agent’s conf.d directory or the directory we just created for containerized installation.

#Log section
logs:

    # - type : (mandatory) type of log input source (tcp / udp / file / docker)
    #   port / path : (mandatory) Set port if type is tcp or udp. Set path if type is file
    #   service : (mandatory) name of the service owning the log
    #   source : (mandatory) attribute that defines which integration is sending the logs
    #   sourcecategory : (optional) Multiple value attribute. Can be used to refine the source attribtue
    #   tags: (optional) add tags to each logs collected

  - type: docker
    image: <IMAGE_NAME>    #or label: <MY_LABEL> or name: <CONTAINER_NAME>
    source: <SOURCE>
    sourcecategory: <SOURCE_CATEGORY>
    service: <SERVICE>

When filtering on the container image, both exact container image name or short names are supported. Suppose you have one container running library/httpd:latest the following filtering match this image name:

  • - image: httpd
  • - image: library/httpd
  • - image: httpd:latest
  • - image: library/httpd:latest

For more examples of configuration files or Agent capabilities (such as filtering, redacting, multiline, …) read the advanced log collection functions.

Option 2: Autodiscovery

Since version 6.2 of the Datadog Agent, you can configure log collection directly in the container labels.

Autodiscovery expects labels to look like these format, depending on the file type:

  • Dockerfile: LABEL "com.datadoghq.ad.logs"='[<LOGS_CONFIG>]'
  • docker-compose.yaml:
  labels: 
    com.datadoghq.ad.logs: '[<LOGS_CONFIG>]'
  • run command: -l com.datadoghq.ad.logs='[<LOGS_CONFIG>]'

Docker Example: NGINX Dockerfile:

The following Dockerfile launches an NGINX container with Autodiscovery enabled:

FROM nginx

EXPOSE 8080
COPY nginx.conf /etc/nginx/nginx.conf
LABEL "com.datadoghq.ad.check_names"='["nginx"]'
LABEL "com.datadoghq.ad.init_configs"='[{}]'
LABEL "com.datadoghq.ad.instances"='[{"nginx_status_url": "http://%%host%%/nginx_status:%%port%%"}]'
LABEL "com.datadoghq.ad.logs"='[{"source": "nginx", "service": "webapp"}]'

Check our Autodiscovery Guidefor more information about Autodiscovery setup and examples.

Further Reading