Host Agent Log collection
Security Monitoring is now available Security Monitoring is now available

Host Agent Log collection

Log collection requires the Datadog Agent v6.0+. Older versions of the Agent do not include the log collection interface. If you are not using the Agent already, follow the Agent installation instructions.

Activate log collection

Collecting logs is disabled by default in the Datadog Agent. To lean how to enable it with the host Agent, see the instructions below. If you are running the Agent in a Kuberentes or Docker environment, see the dedicated Kubernetes Log Collection or Docker Log Collection documentation.

To enable log collection with an Agent running on your host, update the Agent’s main configuration file (datadog.yaml) with:

logs_enabled: true

Starting with Agent v6.19+/v7.19+, HTTPS transport is the default transport used. For more details on how to enforce HTTPS/TCP transport, refer to the Agent transport documentation.

To send logs with environment variables, configure the following:

  • DD_LOGS_ENABLED=true

After activating log collection, the Agent is ready to forward logs to Datadog. Next, configure the Agent on where to collect logs from.

Enabling log collection from integrations

To collect logs for a given integration, uncomment the logs section in that integration’s conf.yaml file and configure it for your environment. If you are running the Agent in a Kubernetes or Docker environment, see the dedicated Kubernetes Log Collection or Docker Log Collection documentation.

Consult the list of supported integrations that include out of the box log configurations.

Custom log collection

Datadog Agent v6 can collect logs and forward them to Datadog from files, the network (TCP or UDP), journald, and Windows channels:

  1. Create a new <CUSTOM_LOG_SOURCE>.d/ folder in the conf.d/ directory at the root of your Agent’s configuration directory.
  2. Create a new conf.yaml file in this new folder.
  3. Add a custom log collection configuration group with the parameters below.
  4. Restart your Agent to take into account this new configuration.
  5. Run the Agent’s status subcommand and look for <CUSTOM_LOG_SOURCE> under the Checks section.

Below are examples of custom log collection setup:

To gather logs from your <APP_NAME> application stored in <PATH_LOG_FILE>/<LOG_FILE_NAME>.log create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: file
    path: "<PATH_LOG_FILE>/<LOG_FILE_NAME>.log"
    service: "<APP_NAME>"
    source: "<SOURCE>"

Note: When tailing files for logs, the Datadog Agent v6 for Windows requires the log files have UTF8 encoding.

To gather logs from your <APP_NAME> application that forwards its logs with TCP over port 10518, create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: tcp
    port: 10518
    service: "<APP_NAME>"
    source: "<CUSTOM_SOURCE>"

If you are using Serilog, Serilog.Sinks.Network is an option for connecting with UDP.

Note: The Agent supports raw string, JSON, and Syslog formatted logs. If you are sending logs in batch, use line break characters to separate your logs.

To gather logs from journald, create a journald.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: journald
    path: /var/log/journal/

Refer to the journald integration documentation for more details regarding the setup for containerized environments and units filtering.

To send Windows events as logs to Datadog, add the channels to conf.d/win32_event_log.d/conf.yaml manually or use the Datadog Agent Manager.

To see your channel list, run the following command in a PowerShell:

Get-WinEvent -ListLog *

To see the most active channels, run the following command in a PowerShell:

Get-WinEvent -ListLog * | sort RecordCount -Descending

Then add the channels to your win32_event_log.d/conf.yaml configuration file:

logs:
  - type: windows_event
    channel_path: "<CHANNEL_1>"
    source: "<CHANNEL_1>"
    service: "<SERVICE>"
    sourcecategory: windowsevent

  - type: windows_event
    channel_path: "<CHANNEL_2>"
    source: "<CHANNEL_2>"
    service: "<SERVICE>"
    sourcecategory: windowsevent

Edit the <CHANNEL_X> parameters with the Windows channel name you want to collect events from. Set the corresponding source parameter to the same channel name to benefit from the integration automatic processing pipeline setup.

Finally, restart the Agent.

List of all available parameters for log collection:

ParameterRequiredDescription
typeYesThe type of log input source. Valid values are: tcp, udp, file, windows_event, docker, or journald.
portYesIf type is tcp or udp, set the port for listening to logs.
pathYesIf type is file or journald, set the file path for gathering logs.
channel_pathYesIf type is windows_event, list the Windows event channels for collecting logs.
serviceYesThe name of the service owning the log. If you instrumented your service with Datadog APM, this must be the same service name.
sourceYesThe attribute that defines which integration is sending the logs. If the logs do not come from an existing integration, then this field may include a custom source name. However, it is recommended that you match this value to the namespace of any related custom metrics you are collecting, for example: myapp from myapp.request.count.
include_unitsNoIf type is journald, list of the specific journald units to include.
exclude_pathNoIf type is file, and path contains a wildcard character, list the matching file that should be excluded from log collection. It is available for Agent version >= 6.18.
exclude_unitsNoIf type is journald, list of the specific journald units to exclude.
sourcecategoryNoA multiple value attribute used to refine the source attribute, for example: source:mongodb, sourcecategory:db_slow_logs.
start_positionNoIf type is file, set the position for the Agent to start reading the file. Valid values are beginning and end (default: end). If path contains a wildcard character, beginning is not supported. Added in Agent v6.19/v7.19
tagsNoA list of tags added to each log collected (learn more about tagging).

Further Reading