Log Collection and Integrations

Overview

Choose a configuration option below to begin ingesting your logs. If you are already using a log-shipper daemon, refer to the dedicated documentation for Rsyslog, Syslog-ng, NXlog, FluentD, or Logstash.

Consult the list of available Datadog log collection endpoints if you want to send your logs directly to Datadog.

Note: When sending logs in a JSON format to Datadog, there is a set of reserved attributes that have a specific meaning within Datadog. See the Reserved Attributes section to learn more.

Setup

  1. Install the Datadog Agent.
  2. To enable log collection, change logs_enabled: false to logs_enabled: true in your Agent’s main configuration file (datadog.yaml). See the Host Agent Log collection documentation for more information and examples.
  3. Once enabled, the Datadog Agent can be configured to tail log files or listen for logs sent over UDP/TCP, filter out logs or scrub sensitive data, and aggregate multi-line logs.
  1. Install the Datadog Agent.
  2. To enable log collection, change logs_enabled: false to logs_enabled: true in your Agent’s main configuration file (datadog.yaml). See the Host Agent Log collection documentation for more information and examples.
  3. Follow your application language installation instructions to configure a logger and start generating logs:
Java
Python
Go
Ruby
.Net
PHP
Node.js
PHP
PHP

Choose a container or orchestrator provider and follow their dedicated log collection instructions:

Docker
Kubernetes
Red Hat OpenShift
Amazon ECS
ECS Fargate

Notes:

Use the Datadog Forwarder, an AWS Lambda function that ships logs from your environment to Datadog. To enable log collection in your AWS serverless environment, refer to the Datadog Forwarder documentation.

Select your Cloud provider below to see how to automatically collect your logs and forward them to Datadog:

Docker
Kubernetes
Amazon ECS
Amazon ECS

Datadog integrations and log collection are tied together. You can use an integration’s default configuration file to enable dedicated processors, parsing, and facets in Datadog. To begin log collection with an integration:

  1. Select an integration from the Integrations page and follow the setup instructions.
  2. Follow the integration’s log collection instructions. This section covers how to uncomment the logs section in that integration’s conf.yaml file and configure it for your environment.

Additional configuration options

Logging endpoints

Datadog provides logging endpoints for both SSL-encrypted connections and unencrypted connections. Use the encrypted endpoint when possible. The Datadog Agent uses the encrypted endpoint to send logs to Datadog. More information is available in the Datadog security documentation.

Supported endpoints

Use the site selector dropdown on the right side of the page to see supported endpoints by Datadog site.

SiteTypeEndpointPortDescription
USHTTPShttp-intake.logs.datadoghq.com443Used by custom forwarder to send logs in JSON or plain text format over HTTPS. See the Logs HTTP API documentation.
USHTTPSagent-http-intake.logs.datadoghq.com443Used by the Agent to send logs in JSON format over HTTPS. See the Host Agent Log collection documentation.
USHTTPSlambda-http-intake.logs.datadoghq.com443Used by Lambda functions to send logs in raw, Syslog, or JSON format over HTTPS.
USHTTPSlogs.443Used by the Browser SDK to send logs in JSON format over HTTPS.
USTCPagent-intake.logs.datadoghq.com10514Used by the Agent to send logs without TLS.
USTCP and TLSagent-intake.logs.datadoghq.com10516Used by the Agent to send logs with TLS.
USTCP and TLSintake.logs.datadoghq.com443Used by custom forwarders to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection.
USTCP and TLSfunctions-intake.logs.datadoghq.com443Used by Azure functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. Note: This endpoint may be useful with other cloud providers.
USTCP and TLSlambda-intake.logs.datadoghq.com443Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection.

SiteTypeEndpointPortDescription
EUHTTPShttp-intake.logs.datadoghq.eu443Used by custom forwarder to send logs in JSON or plain text format over HTTPS. See the Logs HTTP API documentation.
EUHTTPSagent-http-intake.logs.datadoghq.eu443Used by the Agent to send logs in JSON format over HTTPS. See the Host Agent Log collection documentation.
EUHTTPSlambda-http-intake.logs.datadoghq.eu443Used by Lambda functions to send logs in raw, Syslog, or JSON format over HTTPS.
EUHTTPSlogs.443Used by the Browser SDK to send logs in JSON format over HTTPS.
EUTCP and TLSagent-intake.logs.datadoghq.eu443Used by the Agent to send logs in protobuf format over an SSL-encrypted TCP connection.
EUTCP and TLSfunctions-intake.logs.datadoghq.eu443Used by Azure functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. Note: This endpoint may be useful with other cloud providers.
EUTCP and TLSlambda-intake.logs.datadoghq.eu443Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection.

SiteTypeEndpointPortDescription
US3HTTPShttp-intake.logs.us3.datadoghq.com443Used by custom forwarder to send logs in JSON or plain text format over HTTPS. See the Logs HTTP API documentation.
US3HTTPSlambda-http-intake.logs.us3.datadoghq.com443Used by Lambda functions to send logs in raw, Syslog, or JSON format over HTTPS.
US3HTTPSagent-http-intake.logs.us3.datadoghq.com443Used by the Agent to send logs in JSON format over HTTPS. See the Host Agent Log collection documentation.
US3HTTPSlogs.443Used by the Browser SDK to send logs in JSON format over HTTPS.

SiteTypeEndpointPortDescription
US5HTTPShttp-intake.logs.us5.datadoghq.com443Used by custom forwarder to send logs in JSON or plain text format over HTTPS. See the Logs HTTP API documentation.
US5HTTPSlambda-http-intake.logs.us5.datadoghq.com443Used by Lambda functions to send logs in raw, Syslog, or JSON format over HTTPS.
US5HTTPSagent-http-intake.logs.us5.datadoghq.com443Used by the Agent to send logs in JSON format over HTTPS. See the Host Agent Log collection documentation.
US5HTTPSlogs.443Used by the Browser SDK to send logs in JSON format over HTTPS.

SiteTypeEndpointPortDescription
US1-FEDHTTPShttp-intake.logs.ddog-gov.com443Used by custom forwarder to send logs in JSON or plain text format over HTTPS. See the Logs HTTP API documentation.
US1-FEDHTTPSlambda-http-intake.logs.ddog-gov.datadoghq.com443Used by Lambda functions to send logs in raw, Syslog, or JSON format over HTTPS.
US1-FEDHTTPSagent-http-intake.logs.ddog-gov.datadoghq.com443Used by the Agent to send logs in JSON format over HTTPS. See the Host Agent Log collection documentation.
US1-FEDHTTPSlogs.443Used by the Browser SDK to send logs in JSON format over HTTPS.

Custom log forwarding

Any custom process or logging library able to forward logs through TCP or HTTP can be used in conjunction with Datadog Logs.

You can send logs to Datadog platform over HTTP. Refer to the Datadog Log HTTP API documentation to get started.

Test it manually with telnet. You must prefix the log entry with your Datadog API Key and add a payload.

telnet intake.logs.datadoghq.com 10514
<DATADOG_API_KEY> Log sent directly via TCP

Your payload, or Log sent directly via TCP as written in the example, can be in raw, Syslog, or JSON format. If your payload is in JSON format, Datadog automatically parses its attributes.

telnet intake.logs.datadoghq.com 10514
<DATADOG_API_KEY> {"message":"json formatted log", "ddtags":"env:my-env,user:my-user", "ddsource":"my-integration", "hostname":"my-hostname", "service":"my-service"}

Test it manually with telnet. You must prefix the log entry with your Datadog API Key and add a payload.

telnet agent-intake.logs.datadoghq.eu 443
<DATADOG_API_KEY> Log sent directly via TCP

Your payload, or Log sent directly via TCP as written in the example, can be in raw, Syslog, or JSON format. If your payload is in JSON format, Datadog automatically parses its attributes.

telnet intake.logs.datadoghq.com 10514
<DATADOG_API_KEY> {"message":"json formatted log", "ddtags":"env:my-env,user:my-user", "ddsource":"my-integration", "hostname":"my-hostname", "service":"my-service"}

The TCP endpoint is not recommended for this site. Contact support for more information.

The TCP endpoint is not supported for this site.

Notes:

  • The HTTPS API supports logs of sizes up to 1MB. However, for optimal performance, it is recommended that an individual log be no greater than 25K bytes. If you use the Datadog Agent for logging, it is configured to split a log at 256kB (256000 bytes).
  • A log event should not have more than 100 tags, and each tag should not exceed 256 characters for a maximum of 10 million unique tags per day.
  • A log event converted to JSON format should contain less than 256 attributes. Each of those attribute’s keys should be less than 50 characters, nested in less than 10 successive levels, and their respective value should be less than 1024 characters if promoted as a facet.
  • Log events can be submitted up to 18h in the past and 2h in the future.

Log events that do not comply with these limits might be transformed or truncated by the system or not indexed if outside the provided time range. However, Datadog tries to preserve as much user data as possible.

Attributes and tags

Attributes prescribe logs facets, which are used for filtering and searching in Log Explorer. See the dedicated attributes and aliasing documentation for a list of reserved and standard attributes and to learn how to support a naming convention with logs attributes and aliasing.

Attributes for stack traces

When logging stack traces, there are specific attributes that have a dedicated UI display within your Datadog application such as the logger name, the current thread, the error type, and the stack trace itself.

Stack trace

To enable these functionalities use the following attribute names:

AttributeDescription
logger.nameName of the logger
logger.thread_nameName of the current thread
error.stackActual stack trace
error.messageError message contained in the stack trace
error.kindThe type or “kind” of an error (for example, “Exception”, or “OSError”)

Note: By default, integration Pipelines attempt to remap default logging library parameters to those specific attributes and parse stack traces or traceback to automatically extract the error.message and error.kind.

For more information, see the complete source code attributes documentation.

Next steps

Once logs are collected and ingested, they are available in Log Explorer. Log Explorer is where you can search, enrich, and view alerts on your logs. See the Log Explorer documentation to begin analyzing your log data, or see the additional log management documentation below.

Log Explorer view

Further Reading


*Logging without Limits is a trademark of Datadog, Inc.