Host Agent Log collection

Host Agent Log collection

Log collection requires the Datadog Agent v6.0+. Older versions of the Agent do not include the log collection interface. If you are not using the Agent already, follow the Agent installation instructions.

Activate log collection

Collecting logs is not enabled by default in the Datadog Agent. If you are running the Agent in a Kubernetes or Docker environment, see the dedicated Kubernetes Log Collection or Docker Log Collection documentation.

To enable log collection with an Agent running on your host, change logs_enabled:false to logs_enabled:true in the Agent’s main configuration file (datadog.yaml).

datadog.yaml

## @param logs_enabled - boolean - optional - default: false
## @env DD_LOGS_ENABLED - boolean - optional - default: false
## Enable Datadog Agent log collection by setting logs_enabled to true.
logs_enabled: false

## @param logs_config - custom object - optional
## Enter specific configurations for your Log collection.
## Uncomment this parameter and the one below to enable them.
## See https://docs.datadoghq.com/agent/logs/
logs_config:

  ## @param container_collect_all - boolean - optional - default: false
  ## @env DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL - boolean - optional - default: false
  ## Enable container log collection for all the containers (see ac_exclude to filter out containers)
  container_collect_all: false

  ## @param logs_dd_url - string - optional
  ## @env DD_LOGS_CONFIG_DD_URL - string - optional
  ## Define the endpoint and port to hit when using a proxy for logs. The logs are forwarded in TCP
  ## therefore the proxy must be able to handle TCP connections.
  logs_dd_url: <ENDPOINT>:<PORT>

  ## @param logs_no_ssl - boolean - optional - default: false
  ## @env DD_LOGS_CONFIG_LOGS_NO_SSL - optional - default: false
  ## Disable the SSL encryption. This parameter should only be used when logs are
  ## forwarded locally to a proxy. It is highly recommended to then handle the SSL encryption
  ## on the proxy side.
  logs_no_ssl: false

  ## @param processing_rules - list of custom objects - optional
  ## @env DD_LOGS_CONFIG_PROCESSING_RULES - list of custom objects - optional
  ## Global processing rules that are applied to all logs. The available rules are
  ## "exclude_at_match", "include_at_match" and "mask_sequences". More information in Datadog documentation:
  ## https://docs.datadoghq.com/agent/logs/advanced_log_collection/#global-processing-rules
  processing_rules:
    - type: <RULE_TYPE>
      name: <RULE_NAME>
      pattern: <RULE_PATTERN>

  ## @param use_http - boolean - optional - default: false
  ## @env DD_LOGS_CONFIG_USE_HTTP - boolean - optional - default: false
  ## By default, logs are sent through TCP, use this parameter
  ## to send logs in HTTPS batches to port 443
  use_http: true

  ## @param use_tcp - boolean - optional - default: false
  ## @env DD_USE_TCP - boolean - optional - default: false
  ## By default, logs are sent through HTTP if possible, use this parameter
  ## to send logs in TCP
  use_tcp: true

  ## @param use_compression - boolean - optional - default: false
  ## @env DD_LOGS_CONFIG_USE_COMPRESSION - boolean - optional - default: false
  ## This parameter is available when sending logs with HTTPS. If enabled, the Agent
  ## compresses logs before sending them.
  use_compression: true

  ## @param compression_level - integer - optional - default: 6
  ## @env DD_LOGS_CONFIG_COMPRESSION_LEVEL - boolean - optional - default: false
  ## The compression_level parameter accepts values from 0 (no compression)
  ## to 9 (maximum compression but higher resource usage).
  compression_level: 6

  ## @param batch_wait - integer - optional - default: 5
  ## @env DD_LOGS_CONFIG_BATCH_WAIT - integer - optional - default: 5
  ## The maximum time the Datadog Agent waits to fill each batch of logs before sending.
  batch_wait: 5

Starting with Agent v6.19+/v7.19+, HTTPS transport is the default transport used. For more details on how to enforce HTTPS/TCP transport, refer to the Agent transport documentation.

To send logs with environment variables, configure the following:

  • DD_LOGS_ENABLED=true

After activating log collection, the Agent is ready to forward logs to Datadog. Next, configure the Agent on where to collect logs from.

Custom log collection

Datadog Agent v6 can collect logs and forward them to Datadog from files, the network (TCP or UDP), journald, and Windows channels:

  1. Create a new <CUSTOM_LOG_SOURCE>.d/ folder in the conf.d/ directory at the root of your Agent’s configuration directory.
  2. Create a new conf.yaml file in this new folder.
  3. Add a custom log collection configuration group with the parameters below.
  4. Restart your Agent to take into account this new configuration.
  5. Run the Agent’s status subcommand and look for <CUSTOM_LOG_SOURCE> under the Checks section.

Below are examples of custom log collection setup:

To gather logs from your <APP_NAME> application stored in <PATH_LOG_FILE>/<LOG_FILE_NAME>.log create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: file
    path: "<PATH_LOG_FILE>/<LOG_FILE_NAME>.log"
    service: "<APP_NAME>"
    source: "<SOURCE>"

To gather logs from your <APP_NAME> application that forwards its logs to TCP port 10518, create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: tcp
    port: 10518
    service: "<APP_NAME>"
    source: "<CUSTOM_SOURCE>"

If you are using Serilog, Serilog.Sinks.Network is an option for connecting with UDP.

In the Agent version 7.31.0+, the TCP connection stays open indefinitely even when idle.

Note: The Agent supports raw string, JSON, and Syslog formatted logs. If you are sending logs in batch, use line break characters to separate your logs.

To gather logs from journald, create a journald.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: journald
    path: /var/log/journal/

Refer to the journald integration documentation for more details regarding the setup for containerized environments and units filtering.

To send Windows events as logs to Datadog, add the channels to conf.d/win32_event_log.d/conf.yaml manually or use the Datadog Agent Manager.

To see your channel list, run the following command in a PowerShell:

Get-WinEvent -ListLog *

To see the most active channels, run the following command in a PowerShell:

Get-WinEvent -ListLog * | sort RecordCount -Descending

Then add the channels to your win32_event_log.d/conf.yaml configuration file:

logs:
  - type: windows_event
    channel_path: "<CHANNEL_1>"
    source: "<CHANNEL_1>"
    service: "<SERVICE>"
    sourcecategory: windowsevent

  - type: windows_event
    channel_path: "<CHANNEL_2>"
    source: "<CHANNEL_2>"
    service: "<SERVICE>"
    sourcecategory: windowsevent

Edit the <CHANNEL_X> parameters with the Windows channel name you want to collect events from. Set the corresponding source parameter to the same channel name to benefit from the integration automatic processing pipeline setup.

Finally, restart the Agent.

List of all available parameters for log collection:

ParameterRequiredDescription
typeYesThe type of log input source. Valid values are: tcp, udp, file, windows_event, docker, or journald.
portYesIf type is tcp or udp, set the port for listening to logs.
pathYesIf type is file or journald, set the file path for gathering logs.
channel_pathYesIf type is windows_event, list the Windows event channels for collecting logs.
serviceYesThe name of the service owning the log. If you instrumented your service with Datadog APM, this must be the same service name. Check the unified service tagging instructions when configuring service across multiple data types.
sourceYesThe attribute that defines which integration is sending the logs. If the logs do not come from an existing integration, then this field may include a custom source name. However, it is recommended that you match this value to the namespace of any related custom metrics you are collecting, for example: myapp from myapp.request.count.
include_unitsNoIf type is journald, list of the specific journald units to include.
exclude_pathsNoIf type is file, and path contains a wildcard character, list the matching file or files to exclude from log collection. This is available for Agent version >= 6.18.
exclude_unitsNoIf type is journald, list of the specific journald units to exclude.
sourcecategoryNoThe attribute used to define the category a source attribute belongs to, for example: source:postgres, sourcecategory:database or source: apache, sourcecategory: http_web_access.
start_positionNoIf type is file, set the position for the Agent to start reading the file. Valid values are beginning and end (default: end). If path contains a wildcard character, beginning is not supported. Added in Agent v6.19/v7.19
encodingNoIf type is file, set the encoding for the Agent to read the file. Set it to utf-16-le for UTF16 little-endian and utf-16-be for UTF16 big-endian. Any other value will be ignored and the Agent will read the file as UTF8. Added in Agent v6.23/v7.23
tagsNoA list of tags added to each log collected (learn more about tagging).

Further Reading