Send logs to Datadog

Send logs to Datadog

Overview

Datadog Log Management is used to collect logs from your application. This page shows you how get your first logs into Datadog. Before moving forward:

  1. If you haven’t already, create a Datadog account and enable Datadog Log Management.

  2. Set up a Vagrant Ubuntu 16.04 virtual machine using the following commands. For more information about Vagrant, see their Getting Started page:

    vagrant init ubuntu/xenial64
vagrant up
vagrant ssh

Once this is done, follow the sections below to discover how to:

Sending logs manually

To send logs manually, use the telnet command with your Datadog API key within the Vagrant virtual machine.

Logs can be a full-text message:

The secure TCP endpoint is telnet intake.logs.datadoghq.com with port (or port for nonsecure connections).

telnet intake.logs.datadoghq.com 10516

<DATADOG_API_KEY> Plain text log sent through TCP

telnet intake.logs.datadoghq.com 1883

<DATADOG_API_KEY> Plain text log sent through TCP
This feature is not supported.

This produces the following result in the Log Explorer Page:

Custom telnet

or a JSON object that is automatically parsed by Datadog:

telnet intake.logs.datadoghq.com 10516

<DATADOG_API_KEY> {"message":"JSON formatted log sent through TCP", "ddtags":"env:dev", "ddsource":"terminal", "hostname":"gs-hostame", "service":"user"}

telnet tcp-intake.logs.datadoghq.eu 1883

<DATADOG_API_KEY> {"message":"JSON formatted log sent through TCP", "ddtags":"env:dev", "ddsource":"terminal", "hostname":"gs-hostame", "service":"user"}
This feature is not supported.

This produces the following result in the Log Explorer Page:

JSON logs

Send logs from a file

Install the Agent

To install the Datadog Agent within your Vagrant host, use the one line install command updated with your Datadog API key:

DD_API_KEY=<DATADOG_API_KEY>  bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

DD_API_KEY=<DATADOG_API_KEY> DD_SITE="datadoghq.eu" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

DD_API_KEY=<DATADOG_API_KEY> DD_SITE="ddog-gov.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

DD_API_KEY=<DATADOG_API_KEY> DD_SITE="us3.datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

Validation

Verify the Agent is running with the status command sudo datadog-agent status. You have not enabled log collection yet, so you should see:

==========
Logs Agent
==========

  Logs Agent is not running

Note: After a few minutes, you can verify that the Agent is connected to your account by checking the Infrastructure List in Datadog.

Enable log collection

To enable log collection with the Agent, edit the datadog.yaml configuration file located at /etc/datadog-agent/datadog.yaml and set logs_enabled:true:

## @param logs_enabled - boolean - optional - default: false
## Enable Datadog Agent log collection by setting logs_enabled to true.
#
logs_enabled: true

Monitor a custom file

Create the log file

In order to collect logs from a custom file, first create the file and add one line of logs to it:

$ touch log_file_to_monitor.log

$ echo "First line of log" >> log_file_to_monitor.log

Configure the Agent

To specify to the Agent to monitor this log file:

  1. Create a new configuration folder in the configuration directory of the Agent:

    sudo mkdir /etc/datadog-agent/conf.d/custom_log_collection.d/

  2. Create your configuration file within this new configuration folder:

    sudo touch /etc/datadog-agent/conf.d/custom_log_collection.d/conf.yaml

  3. Copy and paste the following content within this conf.yaml file:

    logs:
    - type: file
      path: /home/ubuntu/log_file_to_monitor.log
      source: custom
      service: user

  4. Restart the Agent: sudo service datadog-agent restart

Validation

If the log configuration is correct, the status command sudo datadog-agent status outputs:

==========
Logs Agent
==========
    LogsProcessed: 0
    LogsSent: 0

  custom_log_collection
  ---------------------
    Type: file
    Path: /home/ubuntu/log_file_to_monitor.log
    Status: OK
    Inputs: /home/ubuntu/log_file_to_monitor.log

Add new logs to the file

Now that everything is configured correctly, add new entries to your log file to see them within Datadog:

$ echo "New line of log in the log file" >> log_file_to_monitor.log

This produces the following result in the Log Explorer Page:

File log example

Note: When using the Datadog Agent, log events larger than 256KB are split into multiple entries.

Further reading

Additional helpful documentation, links, and articles:

Introduction to Logs in DatadogLEARNING CENTER more more Log Collection & IntegrationsDOCUMENTATION more more Learn how to configure unified service taggingDOCUMENTATION more more