The Service Map for APM is here!

Log Management

The Log Management solution is an all-in-one comprehensive solution that comprises collection, processing, live tailing, exploration, graphing, dashboarding, alerting and archival over all the logs generated by your application, and your infrastructure.

Log Collection

Log collection is the beginning of your journey in the wonderful world of log management. Use the Datadog Agent to collect logs directly from your hosts or your containerized environments. You can collect AWS service logs with Datadog’s AWS Lambda function.If you are already using a log-shipper daemon, refer to the dedicated documentation for Rsyslog, Syslog-ng, NXlog, FluentD, and Logstash.

Integrations and Log Collection are intimately tied together. By collecting Logs the right way, you enable all the the subsequent components such as processing, parsing, and facets in the Explorer. Discover the log integrations supported by Datadog. You can also define custom log sources if there isn’t an integration for your source yet.

The different ways and places from which to collect logs.

From your hosts

Follow the Datadog Agent installation instructions to start forwarding logs alongside your metrics and traces. The Agent can tail log files or listen for logs sent over UDP / TCP, and you can configure it to filter out logs, scrub sensitive data, or aggregate multi line logs.

From a Docker environment

The Datadog Agent can collect logs directly from container stdout/stderr without using a logging driver. When the Agent’s Docker check is enabled, container and orchestrator metadata are automatically added as tags to your logs.

It is possible to collect logs from all your containers or only a subset filtered by container image, label, or name. Autodiscovery can also be used to configure log collection directly in the container labels.

In Kubernetes environments you can also leverage the daemonset installation.

From AWS services

The Datadog Agent can be used to collect logs directly from ECS or EC2 instances and applications running on them.

However, AWS services logs are collected thanks to Datadog’s Lambda function. Triggers are then defined (manually or automatically) to forward logs from any S3 bucket, Cloudwatch Log group, or Cloudwatch events.

From a custom forwarder

Any custom process or logging library able to forward logs through TCP can be used in conjuntion with Datadog Logs. The secure TCP endpoint is intake.logs.datadoghq.com:10516 (or port 10514 for insecure connections).

You must prefix the log entry with your Datadog API Key, e.g.:

<DATADOG_API_KEY> this is my log

Test it manually with telnet:

telnet intake.logs.datadoghq.com 10514 
<DATADOG_API_KEY> Log sent directly via TCP

This will produce the following result in your live tail page:

Custom telnet

Datadog Logs Endpoints

Datadog provides logging endpoints for both SSL-encrypted connections and unencrypted connections. You should use the encrypted endpoint when possible. The Datadog Agent uses the encrypted endpoint to send logs to Datadog (more information available in the Datadog security documentation).

Endpoints that can be used to send logs to Datadog:

Endpoints for SSL encrypted connections Port Description
agent-intake.logs.datadoghq.com 10516 Used by the Agent to send logs in protobuf format over an SSL-encrypted TCP connection.
intake.logs.datadoghq.com 10516 Used by custom forwarders to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection.
lambda-intake.logs.datadoghq.com 10516 Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection.
Endpoint for unencrypted connections Port Description
intake.logs.datadoghq.com 10514 Used by custom forwarders to send logs in raw, Syslog, or JSON format over an unecrypted TCP connection.
Endpoints for SSL encrypted connections Port Description
agent-intake.logs.datadoghq.eu 443 Used by the Agent to send logs in protobuf format over an SSL-encrypted TCP connection.
tcp-intake.logs.datadoghq.eu 443 Used by custom forwarders to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection.
lambda-intake.logs.datadoghq.eu 443 Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection.
Endpoint for unencrypted connections Port Description
tcp-intake.logs.datadoghq.eu 1883 Used by custom forwarders to send logs in raw, Syslog, or JSON format format over an unecrypted TCP connection.

Reserved attributes

Here are some key attributes you should pay attention to when setting up your project:

Attribute Description
host The name of the originating host as defined in metrics. We automatically retrieve corresponding host tags from the matching host in Datadog and apply them to your logs. The Agent sets this value automatically.
source This corresponds to the integration name: the technology from which the log originated. When it matches an integration name, Datadog automatically installs the corresponding parsers and facets. For example: nginx, postgresql, etc.
service The name of the application or service generating the log events. It is used to switch from Logs to APM, so make sure you define the same value when you use both products.
message By default, Datadog ingests the value of the message attribute as the body of the log entry. That value is then highlighted and displayed in the Logstream, where it is indexed for full text search.

Your logs are collected and centralized into the Log Explorer view. You can also search, enrich, and alert on your logs.

Log Explorer view

Further Reading