The Service Map for APM is here!

Logstash

Agent Check Agent Check

Supported OS: Linux Mac OS Windows

Overview

Get metrics from Logstash service in real time to:

  • Visualize and monitor Logstash states.
  • Be notified about Logstash events.

Setup

The Logstash check is NOT included in the Datadog Agent package.

Installation

To install the Logstash check on your host:

  1. Download the Datadog Agent.
  2. Download the check.py file for Logstash.
  3. Place it in the Agent’s checks.d directory.
  4. Rename it to logstash.py.

Configuration

To configure the Logstash check:

  1. Create a logstash.d/ folder in the conf.d/ folder at the root of your Agent’s directory.
  2. Create a conf.yaml file in the logstash.d/ folder previously created.
  3. Consult the sample logstash.yaml file and copy its content in the conf.yaml file.
  4. Edit the conf.yaml to start collecting your metrics or logs
  5. Restart the Agent.

Metric Collection

  • Add this configuration setup to your conf.yaml file to start gathering your Logstash metrics:
init_config:

instances:
  #   The URL where Logstash provides its monitoring API. This will be used to fetch various runtime metrics about Logstash.
  #
  - url: http://localhost:9600

Configure it to point to your server and port.

See the sample conf.yaml for all available configuration options. * Restart the Agent to begin sending Logstash metrics to Datadog.

Log Collection

Datadog has an ouput plugin for Logstash that takes care of sending your logs to your Datadog platform.

To install this plugin run the following command:

  • logstash-plugin install logstash-output-datadog_logs

Then configure the datadog_logs plugin with your Datadog API key:

output {
    datadog_logs {
        api_key => "<DATADOG_API_KEY>"
    }
}
Add metadata to your logs

In order to get the best use out of your logs in Datadog, it is important to have the proper metadata associated with your logs (including hostname and source). By default, the hostname and timestamp should be properly remapped thanks to our default remapping for reserved attributes. To make sure the service is correctly remapped, add its attribute value to the Service remapping list.

Source

Setup a Logstash filter to set the source (Datadog integration name) on your logs.

filter {
  mutate {
    add_field => {
 "ddsource" => "<MY_SOURCE_VALUE>"
       }
    }
 }

This triggers the integration automatic setup in Datadog.

Custom tags

Host tags are automatically set on your logs if there is a matching hostname in your infrastructure list. Use the ddtags attribute to add custom tags to your logs:

filter {
  mutate {
    add_field => {
        "ddtags" => "env:test,<KEY:VALUE>"
       }
    }
 }

Validation

Run the Agent’s status subcommand and look for logstash under the Checks section.

Compatibility

The Logstash check is compatible with Logstash 5.6 and possible earlier versions. Currently it does not support the new pipelines metrics in Logstash 6.0 yet.

Data Collected

Metrics

logstash.process.open_file_descriptors
(gauge)
The number of open file descriptors used by this process.
logstash.process.peak_open_file_descriptors
(gauge)
The peak number of open file descriptors used by this process.
logstash.process.max_file_descriptors
(gauge)
The maximum number of file descriptors used by this process.
logstash.process.mem.total_virtual_in_bytes
(gauge)
Total virtual memory allocated to this process.
shown as byte
logstash.process.cpu.total_in_millis
(gauge)
The CPU time in milliseconds.
shown as millisecond
logstash.process.cpu.percent
(gauge)
CPU utilization in percentage.
shown as percent
logstash.process.cpu.load_average.1m
(gauge)
The average CPU load over one minute.
logstash.process.cpu.load_average.5m
(gauge)
The average CPU load over five minutes
logstash.process.cpu.load_average.15m
(gauge)
The average CPU load over fifteen minutes.
logstash.jvm.threads.count
(gauge)
Number of threads used by the JVM.
shown as thread
logstash.jvm.threads.peak_count
(gauge)
The peak number of threads used by JVM.
shown as thread
logstash.jvm.mem.heap_used_percent
(gauge)
Total Java heap memory used.
shown as percent
logstash.jvm.mem.heap_committed_in_bytes
(gauge)
Total Java heap memory commited.
shown as byte
logstash.jvm.mem.heap_max_in_bytes
(gauge)
Maximum Java heap memory size.
shown as byte
logstash.jvm.mem.heap_used_in_bytes
(gauge)
Total Java heap memory used.
shown as byte
logstash.jvm.mem.non_heap_used_in_bytes
(gauge)
Total Java non-heap memory used.
shown as byte
logstash.jvm.mem.non_heap_committed_in_bytes
(gauge)
Total Java non-heap memory committed.
shown as byte
logstash.jvm.mem.pools.survivor.peak_used_in_bytes
(gauge)
The Java memory used in the Survivor space.
shown as byte
logstash.jvm.mem.pools.survivor.used_in_bytes
(gauge)
The peak Java memory used in the Survivor space.
shown as byte
logstash.jvm.mem.pools.survivor.peak_max_in_bytes
(gauge)
The peak maximum Java memory used in the Survivor space.
shown as byte
logstash.jvm.mem.pools.survivor.max_in_bytes
(gauge)
The maximum Java memory used in the Survivor space.
shown as byte
logstash.jvm.mem.pools.survivor.committed_in_bytes
(gauge)
The commited Java memory used in the Survivor space.
shown as byte
logstash.jvm.mem.pools.old.peak_used_in_bytes
(gauge)
The peak Java memory used in the Old generation.
shown as byte
logstash.jvm.mem.pools.old.used_in_bytes
(gauge)
The Java memory used in the Old generation.
shown as byte
logstash.jvm.mem.pools.old.peak_max_in_bytes
(gauge)
The peak maximum Java memory used in the Old generation.
shown as byte
logstash.jvm.mem.pools.old.max_in_bytes
(gauge)
The maximum Java memory used in the Old generation.
shown as byte
logstash.jvm.mem.pools.old.committed_in_bytes
(gauge)
The commited Java memory used in the Old generation.
shown as byte
logstash.jvm.mem.pools.young.peak_used_in_bytes
(gauge)
The peak Java memory used in the Young space.
shown as byte
logstash.jvm.mem.pools.young.used_in_bytes
(gauge)
The Java memory used in the Young generation.
shown as byte
logstash.jvm.mem.pools.young.peak_max_in_bytes
(gauge)
The peak maximum Java memory used in the Young generation.
shown as byte
logstash.jvm.mem.pools.young.max_in_bytes
(gauge)
The maximum Java memory used in the Young generation.
shown as byte
logstash.jvm.mem.pools.young.committed_in_bytes
(gauge)
The commited Java memory used in the Young generation.
shown as byte
logstash.jvm.gc.collectors.old.collection_time_in_millis
(gauge)
Garbage collection time spent in the Old generation.
shown as millisecond
logstash.jvm.gc.collectors.old.collection_count
(gauge)
Garbage collection count in the Old generation.
logstash.jvm.gc.collectors.young.collection_time_in_millis
(gauge)
Garbage collection time spent in the Young generation.
shown as millisecond
logstash.jvm.gc.collectors.young.collection_count
(gauge)
Garbage collection time spent in the Young generation.
logstash.reloads.successes
(gauge)
Number of successful configuration reloads.
logstash.reloads.failures
(gauge)
Number of failed configuration reloads.
logstash.pipeline.events.duration_in_millis
(gauge)
Events duration in the pipeline.
shown as millisecond
logstash.pipeline.events.in
(gauge)
Number of events into the pipeline.
logstash.pipeline.events.out
(gauge)
Number of events out from the pipeline.
logstash.pipeline.events.filtered
(gauge)
Number of events filtered.
logstash.pipeline.reloads.successes
(gauge)
Number of successful pipeline reloads.
logstash.pipeline.reloads.failures
(gauge)
Number of failed pipeline reloads.
logstash.pipeline.plugins.inputs.events.out
(gauge)
Number of events out from the input plugin.
logstash.pipeline.plugins.inputs.events.queue_push_duration_in_millis
(gauge)
Duration of queue push in the input plugin.
shown as millisecond
logstash.pipeline.plugins.outputs.events.in
(gauge)
Number of events into the output plugin.
logstash.pipeline.plugins.outputs.events.out
(gauge)
Number of events out from the output plugin.
logstash.pipeline.plugins.outputs.events.duration_in_millis
(gauge)
Duration of events in the output plugin.
shown as millisecond
logstash.pipeline.plugins.filters.events.in
(gauge)
Number of events into the filter plugin.
logstash.pipeline.plugins.filters.events.out
(gauge)
Number of events out from the filter plugin.
logstash.pipeline.plugins.filters.events.duration_in_millis
(gauge)
Duration of events in the filter plugin.
shown as millisecond

Events

The Logstash check does not include any events at this time.

Service checks

logstash.can_connect:

Returns Critical if the Agent cannot connect to Logstash to collect metrics, returns OK otherwise.

Troubleshooting

Agent cannot connect

    logstash
    -------
      - instance #0 [ERROR]: "('Connection aborted.', error(111, 'Connection refused'))"
      - Collected 0 metrics, 0 events & 1 service check

Check that the url in conf.yaml is correct.

If you need further help, contact Datadog Support.


Mistake in the docs? Feel free to contribute!