Agent Log collection
Datadog's Research Report: The State of Serverless Report: The State of Serverless

Agent Log collection

Log collection requires the Datadog Agent v6.0+. Older versions of the Agent do not include the log collection interface. If you are not using the Agent already, follow the Agent installation instructions.

Activate log collection

Collecting logs is disabled by default in the Datadog Agent.

Datadog recommends that you send compressed logs over HTTPS with the Datadog Agent v6.14+, enable log collection in the Agent’s main configuration file (datadog.yaml):

logs_enabled: true
logs_config:
  use_http: true
  use_compression: true

To send logs with environment variables, configure the following:

  • DD_LOGS_ENABLED
  • DD_LOGS_CONFIG_USE_HTTP
  • DD_LOGS_CONFIG_USE_COMPRESSION

For more details about the compression perfomances and batching size, refer to the HTTPS section.

To send logs over HTTPS with the Datadog Agent v6.14+, enable log collection in the Agent’s main configuration file (datadog.yaml):

logs_enabled: true
logs_config:
  use_http: true

Use DD_LOGS_CONFIG_USE_HTTP to configure this through environment variable.

For more details about the compression perfomances and batching size, refer to the HTTPS section.

TCP log forwarding is the default behaviour of the Datadog Agent. Enable log collection in the Agent’s main configuration file (datadog.yaml):

logs_enabled: true

By default, the Datadog Agent sends its logs to Datadog over TLS-encrypted TCP. This requires outbound communication over port 10516.

After activating log collection, the Agent is ready to forward logs to Datadog. Next, configure the Agent on where to collect logs from.

Enabling log collection from integrations

To collect logs for a given integration, uncomment the logs section in that integration’s conf.yaml file and configure it for your environment.

Consult the list of supported integrations that include out of the box log configurations.

If you’re using Kubernetes, make sure to enable log collection in your DaemonSet setup. If you’re using Docker, enable log collection for the containerized Agent. For more information about log collection from containerized environments, refer to the Container Log Collection documentation. If an integration does not support logs by default, use the custom log collection.

Custom log collection

Datadog Agent v6 can collect logs and forward them to Datadog from files, the network (TCP or UDP), journald, and Windows channels:

  1. Create a new <CUSTOM_LOG_SOURCE>.d/ folder in the conf.d/ directory at the root of your Agent’s configuration directory.
  2. Create a new conf.yaml file in this new folder.
  3. Add a custom log collection configuration group with the parameters below.
  4. Restart your Agent to take into account this new configuration.
  5. Run the Agent’s status subcommand and look for <CUSTOM_LOG_SOURCE> under the Checks section.

Below are examples of custom log collection setup:

To gather logs from your <APP_NAME> application stored in <PATH_LOG_FILE>/<LOG_FILE_NAME>.log create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: file
    path: "<PATH_LOG_FILE>/<LOG_FILE_NAME>.log"
    service: "<APP_NAME>"
    source: "<SOURCE>"

Note: When tailing files for logs, the Datadog Agent v6 for Windows requires the log files have UTF8 encoding.

To gather logs from your <APP_NAME> application that forwards its logs with TCP over port 10518, create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: tcp
    port: 10518
    service: "<APP_NAME>"
    source: "<CUSTOM_SOURCE>"

If you are using Serilog, Serilog.Sinks.Network is an option for connecting with UDP.

Note: The Agent supports raw string, JSON, and Syslog formatted logs. If you are sending logs in batch, use line break characters to separate your logs.

To gather logs from journald, create a journald.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: journald
    path: /var/log/journal/

Refer to the journald integration documentation for more details regarding the setup for containerized environments and units filtering.

To send Windows events as logs to Datadog, add the channels to conf.d/win32_event_log.d/conf.yaml manually or use the Datadog Agent Manager.

To see your channel list, run the following command in a PowerShell:

Get-WinEvent -ListLog *

To see the most active channels, run the following command in a PowerShell:

Get-WinEvent -ListLog * | sort RecordCount -Descending

Then add the channels to your win32_event_log.d/conf.yaml configuration file:

logs:
  - type: windows_event
    channel_path: "<CHANNEL_1>"
    source: "<CHANNEL_1>"
    service: "<SERVICE>"
    sourcecategory: windowsevent

  - type: windows_event
    channel_path: "<CHANNEL_2>"
    source: "<CHANNEL_2>"
    service: "<SERVICE>"
    sourcecategory: windowsevent

Edit the <CHANNEL_X> parameters with the Windows channel name you want to collect events from. Set the corresponding source parameter to the same channel name to benefit from the integration automatic processing pipeline setup.

Finally, restart the Agent.

List of all available parameters for log collection:

ParameterRequiredDescription
typeYesThe type of log input source. Valid values are: tcp, udp, file, windows_event, docker, or journald.
portYesIf type is tcp or udp, set the port for listening to logs.
pathYesIf type is file or journald, set the file path for gathering logs.
channel_pathYesIf type is windows_event, list the Windows event channels for collecting logs.
serviceYesThe name of the service owning the log. If you instrumented your service with Datadog APM, this must be the same service name.
sourceYesThe attribute that defines which integration is sending the logs. If the logs do not come from an existing integration, then this field may include a custom source name. However, it is recommended that you match this value to the namespace of any related custom metrics you are collecting, for example: myapp from myapp.request.count.
include_unitsNoIf type is journald, list of the specific journald units to include.
exclude_pathNoIf type is file, and path contains a wildcard character, list the matching file that should be excluded from log collection. It is available for Agent version >= 6.18.
exclude_unitsNoIf type is journald, list of the specific journald units to exclude.
sourcecategoryNoA multiple value attribute used to refine the source attribute, for example: source:mongodb, sourcecategory:db_slow_logs.
tagsNoA list of tags added to each log collected (learn more about tagging).

Send logs over HTTPS

Compressed HTTPS log forwarding is the recommended configuration because a 200 response is returned only if the logs have been written in the Datadog storage:

logs_enabled: true
logs_config:
  use_http: true
  use_compression: true
  compression_level: 6

The Agent sends HTTPS batches with the following limits:

  • Maximum content size per payload: 1MB
  • Maximum size for a single log: 256kB
  • Maximum array size if sending multiple logs in an array: 200 entries logs.

Log compression

The compression_level parameter (or DD_LOGS_CONFIG_COMPRESSION_LEVEL) accepts values from 0 (no compression) to 9 (maximum compression but higher resource usage). The default value is 6.

See the Datadog Agent overhead section for more information about Agent resource usage when compression is enabled.

Configure the batch wait time

The Agent waits up to 5 seconds to fill each batch (either in content size or number of logs). Therefore, in the worst case scenario (when very few logs are generated) switching to HTTPS might add a 5-second latency compared to TCP, which sends all logs in real time.

To change the maximum time the Datadog Agent waits to fill each batch, add the following in the Agent’s main configuration file (datadog.yaml):

logs_config:
  batch_wait: 2

Or use the DD_LOGS_CONFIG_BATCH_WAIT environment variable. The unit is seconds and must be an integer between 1 and 10.

HTTPS Proxy configuration

When logs are sent through HTTPS, use the same set of proxy settings as the other data types to send logs through a web proxy.

Further Reading