Getting Started with Logs

Docs > Getting Started > Getting Started with Logs

Overview

Use Datadog Log Management, also called logs, to collect logs across multiple logging sources, such as your server, container, cloud environment, application, or existing log processors and forwarders. With conventional logging, you have to choose which logs to analyze and retain to maintain cost-efficiency. With Datadog Logging without Limits*, you can collect, process, archive, explore, and monitor your logs without logging limits.

This page shows you how to get started with Log Management in Datadog. If you haven’t already, create a Datadog account.

Configure a logging source

With Log Management, you can analyze and explore data in the Log Explorer, connect Tracing and Metrics to correlate valuable data across Datadog, and use ingested logs for Datadog Cloud SIEM. The lifecycle of a log within Datadog begins at ingestion from a logging source.

Server

There are several integrations available to forward logs from your server to Datadog. Integrations use a log configuration block in their conf.yaml file, which is available in the conf.d/ folder at the root of your Agent’s configuration directory, to forward logs to Datadog from your server.

logs:
  - type: file
    path: /path/to/your/integration/access.log
    source: integration_name
    service: integration_name
    sourcecategory: http_web_access

To begin collecting logs from a server:

If you haven’t already, install the Datadog Agent based on your platform.
Note: Log collection requires Datadog Agent v6+.

Collecting logs is not enabled by default in the Datadog Agent. To enable log collection, set logs_enabled to true in your datadog.yaml file.

datadog.yaml

## @param logs_enabled - boolean - optional - default: false
    ## @env DD_LOGS_ENABLED - boolean - optional - default: false
    ## Enable Datadog Agent log collection by setting logs_enabled to true.
    logs_enabled: false
    
    ## @param logs_config - custom object - optional
    ## Enter specific configurations for your Log collection.
    ## Uncomment this parameter and the one below to enable them.
    ## See https://docs.datadoghq.com/agent/logs/
    logs_config:
    
      @param container_collect_all - boolean - optional - default: false
      @env DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL - boolean - optional - default: false
      Enable container log collection for all the containers (see ac_exclude to filter out containers)
      container_collect_all: false
    
      @param logs_dd_url - string - optional
      @env DD_LOGS_CONFIG_LOGS_DD_URL - string - optional
      Define the endpoint and port to hit when using a proxy for logs.
      As of agent version 7.70.0, proxy paths are supported. To forward logs to a
      specific proxy path, the URL scheme must be specified: https://proxy.example.com:443/logs
      logs_dd_url: <ENDPOINT>:<PORT>
    
      @param logs_no_ssl - boolean - optional - default: false
      @env DD_LOGS_CONFIG_LOGS_NO_SSL - optional - default: false
      Disable the SSL encryption. This parameter should only be used when logs are
      forwarded locally to a proxy. It is highly recommended to then handle the SSL encryption
      on the proxy side.
      logs_no_ssl: false
    
      @param processing_rules - list of custom objects - optional
      @env DD_LOGS_CONFIG_PROCESSING_RULES - list of custom objects - optional
      Global processing rules that are applied to all logs. The available rules are
      "exclude_at_match", "include_at_match" and "mask_sequences". More information in Datadog documentation:
      https://docs.datadoghq.com/agent/logs/advanced_log_collection/#global-processing-rules
      processing_rules:
        - type: <RULE_TYPE>
          name: <RULE_NAME>
          pattern: <RULE_PATTERN>
    
      @param auto_multi_line_detection - boolean - optional - default: false
      @env DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION - boolean - optional - default: false
      Enable automatic aggregation of multi-line logs for common log patterns.
      More information can be found in Datadog documentation:
      https://docs.datadoghq.com/agent/logs/auto_multiline_detection/?tab=configurationfile
      auto_multi_line_detection: true
    
      @param force_use_http - boolean - optional - default: false
      @env DD_LOGS_CONFIG_FORCE_USE_HTTP - boolean - optional - default: false
      Set this parameter to `true` to always send logs via HTTP(S) protocol and never fall back to
      raw TCP forwarding (recommended).
      #
      By default, the Agent sends logs in HTTPS batches if HTTPS connectivity can
      be established at Agent startup, and falls back to TCP otherwise. This parameter
      can be used to override this fallback behavior. It is recommended, but not the default, to
      maintain compatibility with previous Agent versions.
      #
      Note, the logs are forwarded via HTTPS (encrypted) by default. Please use `logs_no_ssl` if you
      need unencrypted HTTP instead.
      force_use_http: true
    
      @param http_protocol - string - optional - default: auto
      @env DD_LOGS_CONFIG_HTTP_PROTOCOL - string - optional - default: auto
      The transport type to use for sending logs. Possible values are "auto" or "http1".
      http_protocol: auto
    
      @param http_timeout - integer - optional - default: 10
      @env DD_LOGS_CONFIG_HTTP_TIMEOUT - integer - optional - default: 10
      The HTTP timeout to use for sending logs, in seconds.
      http_timeout: 10
    
      @param force_use_tcp - boolean - optional - default: false
      @env DD_LOGS_CONFIG_FORCE_USE_TCP - boolean - optional - default: false
      By default, logs are sent via HTTP protocol if possible, set this parameter
      to `true` to always send logs via TCP. If `force_use_http` is set to `true`, this parameter
      is ignored.
      force_use_tcp: true
    
      @param use_compression - boolean - optional - default: true
      @env DD_LOGS_CONFIG_USE_COMPRESSION - boolean - optional - default: true
      This parameter is available when sending logs via HTTP protocol. If enabled, the Agent
      compresses logs before sending them.
      use_compression: true
    
      @param compression_level - integer - optional - default: 6
      @env DD_LOGS_CONFIG_COMPRESSION_LEVEL - boolean - optional - default: false
      The compression_level parameter accepts values from 0 (no compression)
      to 9 (maximum compression but higher resource usage). Only takes effect if
      `use_compression` is set to `true`.
      compression_level: 6
    
      @param batch_wait - integer - optional - default: 5
      @env DD_LOGS_CONFIG_BATCH_WAIT - integer - optional - default: 5
      The maximum time (in seconds) the Datadog Agent waits to fill each batch of logs before sending.
      batch_wait: 5
    
      @param close_timeout - integer - optional - default: 60
      @env DD_LOGS_CONFIG_CLOSE_TIMEOUT - integer - optional - default: 60
      The maximum number of seconds the Agent spends reading from a file after it has been rotated.
      close_timeout: 60
    
      @param open_files_limit - integer - optional - default: 500
      @env DD_LOGS_CONFIG_OPEN_FILES_LIMIT - integer - optional - default: 500
      The maximum number of files that can be tailed in parallel.
      Note: the default for Mac OS is 200. The default for
      all other systems is 500.
      open_files_limit: 500
    
      @param file_wildcard_selection_mode - string - optional - default: `by_name`
      @env DD_LOGS_CONFIG_FILE_WILDCARD_SELECTION_MODE - string - optional - default: `by_name`
      The strategy used to prioritize wildcard matches if they exceed the open file limit.
      #
      Choices are `by_name` and `by_modification_time`.
      #
      `by_name` means that each log source is considered and the matching files are ordered
      in reverse name order. While there are less than `logs_config.open_files_limit` files
      being tailed, this process repeats, collecting from each configured source.
      #
      `by_modification_time` takes all log sources and first adds any log sources that
      point to a specific file. Next, it finds matches for all wildcard sources.
      This resulting list is ordered by which files have been most recently modified
      and the top `logs_config.open_files_limit` most recently modified files are
      chosen for tailing.
      #
      WARNING: `by_modification_time` is less performant than `by_name` and will trigger
      more disk I/O at the configured wildcard log paths.
      file_wildcard_selection_mode: by_name
    
      @param max_message_size_bytes - integer - optional - default: 900000
      @env DD_LOGS_CONFIG_MAX_MESSAGE_SIZE_BYTES - integer - optional - default : 900000
      The maximum size of single log message in bytes. If maxMessageSizeBytes exceeds
      the documented API limit of 1MB - any payloads larger than 1MB will be dropped by the intake.
      https://docs.datadoghq.com/api/latest/logs/
      max_message_size_bytes: 900000
    
      @param integrations_logs_files_max_size - integer - optional - default: 10
      @env DD_LOGS_CONFIG_INTEGRATIONS_LOGS_FILES_MAX_SIZE - integer - optional - default: 10
      The max size in MB that an integration logs file is allowed to use
      integrations_logs_files_max_size: 10
    
      @param integrations_logs_total_usage - integer - optional - default: 100
      @env DD_LOGS_CONFIG_INTEGRATIONS_LOGS_TOTAL_USAGE - integer - optional - default: 100
      The total combined usage all integrations logs files can use
      integrations_logs_total_usage: 100
    
      @param kublet_api_client_read_timeout - duration - optional - default: 30s
      @env DD_LOGS_CONFIG_KUBELET_API_CLIENT_READ_TIMEOUT - duration - optional - default: 30s
      Configure the kubelet API client's timeout used while streaming logs.
      kublet_api_client_read_timeout: 30
    
      @param k8s_container_use_kubelet_api - boolean - optional - default: false
      @env DD_LOGS_CONFIG_K8S_CONTAINER_USE_KUBELET_API - boolean - optional - default: false
      Enable container log collection via the kubelet API, typically used for EKS Fargate
      k8s_container_use_kubelet_api: false
    
      @param streaming - custom object - optional
      This section allows you to configure streaming logs via remote config.
      streaming:
        @param streamlogs_log_file - string - optional
        @env DD_LOGS_CONFIG_STREAMING_STREAMLOGS_LOG_FILE - string - optional
        Path to the file containing the streamlogs log file.
        Default paths:
          * Windows: c:\\programdata\\datadog\\logs\\streamlogs_info\\streamlogs.log
          * Unix: /opt/log/datadog/streamlogs_info/streamlogs.log
          * Linux: /var/log/datadog/streamlogs_info/streamlogs.log
        streamlogs_log_file: <path_to_streamlogs_log_file>

Restart the Datadog Agent.
Follow the integration activation steps or the custom files log collection steps on the Datadog site.
Note: If you’re collecting logs from custom files and need examples for tail files, TCP/UDP, journald, or Windows Events, see Custom log collection.

Container

As of Datadog Agent v6, the Agent can collect logs from containers. Each containerization service has specific configuration instructions based where the Agent is deployed or run, or how logs are routed.

For example, Docker has two different types of Agent installation available: on your host, where the Agent is external to the Docker environment, or deploying a containerized version of the Agent in your Docker environment.

Kubernetes requires that the Datadog Agent run in your Kubernetes cluster, and log collection can be configured using a DaemonSet spec, Helm chart, or with the Datadog Operator.

To begin collecting logs from a container service, follow the in-app instructions.

Cloud

You can forward logs from multiple cloud providers, such as AWS, Azure, and Google Cloud, to Datadog. Each cloud provider has its own set of configuration instructions.

For example, AWS service logs are usually stored in S3 buckets or CloudWatch Log groups. You can subscribe to these logs and forward them to an Amazon Kinesis stream to then forward them to one or multiple destinations. Datadog is one of the default destinations for Amazon Kinesis Delivery streams.

To begin collecting logs from a cloud service, follow the in-app instructions.

Client

Datadog permits log collection from clients through SDKs or libraries. For example, use the datadog-logs SDK to send logs to Datadog from JavaScript clients.

To begin collecting logs from a client, follow the in-app instructions.

Other

If you’re using existing logging services or utilities such as rsyslog, Fluentd, or Logstash, Datadog offers plugins and log forwarding options.

If you don’t see your integration, you can type it in the other integrations box and get notifications for when the integration is available.

To begin collecting logs from a cloud service, follow the in-app instructions.

Explore your logs

Once a logging source is configured, your logs are available in the Log Explorer. This is where you can filter, aggregate, and visualize your logs.

For example, if you have logs flowing in from a service that you wish to examine further, filter by service. You can further filter by status, such as ERROR, and select Group into Patterns to see which part of your service is logging the most errors.

Filtering in the Log Explorer by error pattern

Aggregate your logs into Fields and visualize as Top List to see your top logging services. Select a source, such as info or warn, and select View Logs from the dropdown menu. The side panel populates logs based on error, so you quickly see which host and services require attention.

What’s next?

Once a logging source is configured, and your logs are available in the Log Explorer, you can begin to explore a few other areas of log management.

Log configuration

Set attributes and aliasing to unify your logs environment.
Control how your logs are processed with pipelines and processors.
As Logging without Limits* decouples log ingestion and indexing, you can configure your logs and choose which logs to index, retain, or archive.

Log correlation

Connect logs and traces to exact logs associated with a specific env, service, or version.
If you’re already using metrics in Datadog, you can correlate logs and metrics to gain context of an issue.