Network Performance Monitoring is now generally available! Network Monitoring is now available!

Agent Log collection

Log collection requires the Datadog Agent v6.0+. Older versions of the Agent do not include the log collection interface. If you are not using the Agent already, follow the Agent installation instructions.

Collecting logs is disabled by default in the Datadog Agent. Enable log collection in the Agent’s main configuration file (datadog.yaml):

logs_enabled: true

By default, the Datadog Agent sends its logs to Datadog over TLS-encrypted TCP. This requires outbound communication over port 10516.

See below to send logs over HTTPS.

Note: If you’re using Kubernetes, make sure to enable log collection in your DaemonSet setup. If you’re using Docker, enable log collection for the containerized Agent.

Enabling log collection from integrations

To collect logs for a given integration, uncomment the logs section in that integration’s conf.yaml file and configure it for your environment.

Consult the list of supported integrations that include out of the box log configurations.

If an integration does not support logs by default, use the custom log collection.

Custom log collection

Datadog Agent v6 can collect logs and forward them to Datadog from files, the network (TCP or UDP), journald, and Windows channels:

  1. Create a new <CUSTOM_LOG_SOURCE>.d/ folder in the conf.d/ directory at the root of your Agent’s configuration directory.
  2. Create a new conf.yaml file in this new folder.
  3. Add a custom log collection configuration group with the parameters below.
  4. Restart your Agent to take into account this new configuration.
  5. Run the Agent’s status subcommand and look for <CUSTOM_LOG_SOURCE> under the Checks section.

Below are examples of custom log collection setup:

To gather logs from your <APP_NAME> application stored in <PATH_LOG_FILE>/<LOG_FILE_NAME>.log create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: file
    path: <PATH_LOG_FILE>/<LOG_FILE_NAME>.log
    service: <APP_NAME>
    source: <SOURCE>

Note: When tailing files for logs, the Datadog Agent v6 for Windows requires the log files have UTF8 encoding.

To gather logs from your <APP_NAME> application that forwards its logs with TCP over port 10518, create a <APP_NAME>.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: tcp
    port: 10518
    service: <APP_NAME>
    source: <CUSTOM_SOURCE>

If you are using Serilog, Serilog.Sinks.Network is an option for connecting with UDP.

Note: The Agent supports raw string, JSON, and Syslog formatted logs. If you are sending logs in batch, use line break characters to separate your logs.

To gather logs from journald, create a journald.d/conf.yaml file at the root of your Agent’s configuration directory with the following content:

logs:
  - type: journald
    path: /var/log/journal/

Refer to the journald integration documentation for more details regarding the setup for containerized environments and units filtering.

To send Windows events as logs to Datadog, add the channels to conf.d/win32_event_log.d/conf.yaml manually or use the Datadog Agent Manager.

To see your channel list, run the following command in a PowerShell:

Get-WinEvent -ListLog *

To see the most active channels, run the following command in a PowerShell:

Get-WinEvent -ListLog * | sort RecordCount -Descending

Then add the channels to your win32_event_log.d/conf.yaml configuration file:

logs:
  - type: windows_event
    channel_path: <CHANNEL_1>
    source: <CHANNEL_1>
    service: <SERVICE>
    sourcecategory: windowsevent

  - type: windows_event
    channel_path: <CHANNEL_2>
    source: <CHANNEL_2>
    service: <SERVICE>
    sourcecategory: windowsevent

Edit the <CHANNEL_X> parameters with the Windows channel name you want to collect events from. Set the corresponding source parameter to the same channel name to benefit from the integration automatic processing pipeline setup.

Finally, restart the Agent.

List of all available parameters for log collection:

ParameterRequiredDescription
typeYesThe type of log input source. Valid values are: tcp, udp, file, windows_event, docker, or journald.
portYesIf type is tcp or udp, set the port for listening to logs.
pathYesIf type is file or journald, set the file path for gathering logs.
channel_pathYesIf type is windows_event, list the Windows event channels for collecting logs.
serviceYesThe name of the service owning the log. If you instrumented your service with Datadog APM, this must be the same service name.
sourceYesThe attribute that defines which integration is sending the logs. If the logs do not come from an existing integration, then this field may include a custom source name. However, it is recommended that you match this value to the namespace of any related custom metrics you are collecting, for example: myapp from myapp.request.count.
include_unitsNoIf type is journald, list of the specific journald units to include.
exclude_unitsNoIf type is journald, list of the specific journald units to exclude.
sourcecategoryNoA multiple value attribute used to refine the source attribute, for example: source:mongodb, sourcecategory:db_slow_logs.
tagsNoA list of tags added to each log collected (learn more about tagging).

Send logs over HTTPS

To send logs over HTTPS with the Datadog Agent 6.14+, add the following in the Agent’s main configuration file (datadog.yaml):

logs_config:
  use_http: true

Or set the DD_LOGS_CONFIG_USE_HTTP environment variable to true. Then restart the Agent to sends logs through HTTPS to agent-http-intake.logs.datadoghq.com (US site) or agent-http-intake.logs.datadoghq.eu (EU site) on port 443.

The Agent sends batches that have the following limits:

  • Maximum content size per payload: 1MB
  • Maximum size for a single log: 256kB
  • Maximum array size if sending multiple logs in an array: 200 entries logs.

The Agent waits up to 5 seconds to fill each batch (either in content size or number of logs). Therefore, in the worst case scenario (when very few logs are generated) switching to HTTPS might add a 5-second latency compared to TCP, which sends all logs in real time.

Configure the batch wait time

To change the maximum time the Datadog Agent waits to fill each batch, add the following in the Agent’s main configuration file (datadog.yaml):

logs_config:
  batch_wait: 2

Or use the DD_LOGS_CONFIG_BATCH_WAIT environment variable. The value is in seconds and must be an integer between 1 and 10.

HTTPS Proxy configuration

When logs are sent through HTTPS, use the same set of proxy settings as the other data types to send logs through a web proxy.

Further Reading