Get metrics from Logstash service in real time to:
The Logstash check is NOT included in the Datadog Agent package.
If you are using Agent v6.8+ follow the instructions below to install the Logstash check on your host. See the dedicated Agent guide for installing community integrations to install checks with the Agent prior v6.8 or the Docker Agent:
Run the following command to install the integrations wheel with the Agent:
datadog-agent integration install -t datadog-logstash==<INTEGRATION_VERSION>
Configure your integration like any other packaged integration.
Edit the logstash.d/conf.yaml
file in the conf.d/
folder at the root of your Agent’s configuration directory to start collecting your Logstash metrics and logs. See the sample logstash.d/conf.yaml for all available configuration options.
Add this configuration setup to your conf.yaml
file to start gathering your Logstash metrics:
init_config:
instances:
# The URL where Logstash provides its monitoring API.
# This will be used to fetch various runtime metrics about Logstash.
#
- url: http://localhost:9600
Configure it to point to your server and port.
See the sample conf.yaml for all available configuration options.
Finally, restart the Agent to begin sending Logstash metrics to Datadog.
Datadog has an output plugin for Logstash that takes care of sending your logs to your Datadog platform.
To install this plugin run the following command:
logstash-plugin install logstash-output-datadog_logs
Then configure the datadog_logs
plugin with your Datadog API key:
output {
datadog_logs {
api_key => "<DATADOG_API_KEY>"
}
}
By default, the plugin is configured to send logs through HTTPS (port 443) using gzip compression. You can change this behavior by using the following parameters:
use_http
: Set this to false
if you want to use TCP forwarding and update the host
and port
accordingly (default is true
).use_compression
: Compression is only available for HTTP. Disable it by setting this to false
(default is true
).compression_level
: Set the compression level from HTTP. The range is from 1 to 9, 9 being the best ratio (default is 6
).Additional parameters can be used to change the endpoint used in order to go through a proxy:
host
: The proxy endpoint for logs not directly forwarded to Datadog (default value: http-intake.logs.datadoghq.com
).port
: The proxy port for logs not directly forwarded to Datadog (default value: 80
).ssl_port
: The port used for logs forwarded with a secure TCP/SSL connection to Datadog (default value: 443
).use_ssl
: Instructs the Agent to initialize a secure TCP/SSL connection to Datadog (default value: true
).no_ssl_validation
: Disables SSL hostname validation (default value: false
).This also can be used to send logs to Datadog EU by setting:
output {
datadog_logs {
api_key => "<DATADOG_API_KEY>"
host => "http-intake.logs.datadoghq.eu"
}
}
In order to get the best use out of your logs in Datadog, it is important to have the proper metadata associated with your logs, including hostname and source. By default, the hostname and timestamp should be properly remapped thanks to Datadog’s default remapping for reserved attributes. To make sure the service is correctly remapped, add its attribute value to the service remapping list.
Set up a Logstash filter to set the source (Datadog integration name) on your logs.
filter {
mutate {
add_field => {
"ddsource" => "<MY_SOURCE_VALUE>"
}
}
}
This triggers the integration automatic setup in Datadog.
Host tags are automatically set on your logs if there is a matching hostname in your infrastructure list. Use the ddtags
attribute to add custom tags to your logs:
filter {
mutate {
add_field => {
"ddtags" => "env:test,<KEY:VALUE>"
}
}
}
Run the Agent’s status
subcommand and look for logstash
under the Checks section.
The Logstash check is compatible with Logstash 5.x, 6.x and 7.x versions. It also supports the new multi-pipelines metrics introduced in Logstash 6.0. Tested with Logstash versions 5.6.15, 6.3.0 and 7.0.0.
logstash.process.open_file_descriptors (gauge) | The number of open file descriptors used by this process. |
logstash.process.peak_open_file_descriptors (gauge) | The peak number of open file descriptors used by this process. |
logstash.process.max_file_descriptors (gauge) | The maximum number of file descriptors used by this process. |
logstash.process.mem.total_virtual_in_bytes (gauge) | Total virtual memory allocated to this process. Shown as byte |
logstash.process.cpu.total_in_millis (gauge) | The CPU time in milliseconds. Shown as millisecond |
logstash.process.cpu.percent (gauge) | CPU utilization in percentage. Shown as percent |
logstash.process.cpu.load_average.1m (gauge) | The average CPU load over one minute. |
logstash.process.cpu.load_average.5m (gauge) | The average CPU load over five minutes |
logstash.process.cpu.load_average.15m (gauge) | The average CPU load over fifteen minutes. |
logstash.jvm.threads.count (gauge) | Number of threads used by the JVM. Shown as thread |
logstash.jvm.threads.peak_count (gauge) | The peak number of threads used by JVM. Shown as thread |
logstash.jvm.mem.heap_used_percent (gauge) | Total Java heap memory used. Shown as percent |
logstash.jvm.mem.heap_committed_in_bytes (gauge) | Total Java heap memory commited. Shown as byte |
logstash.jvm.mem.heap_max_in_bytes (gauge) | Maximum Java heap memory size. Shown as byte |
logstash.jvm.mem.heap_used_in_bytes (gauge) | Total Java heap memory used. Shown as byte |
logstash.jvm.mem.non_heap_used_in_bytes (gauge) | Total Java non-heap memory used. Shown as byte |
logstash.jvm.mem.non_heap_committed_in_bytes (gauge) | Total Java non-heap memory committed. Shown as byte |
logstash.jvm.mem.pools.survivor.peak_used_in_bytes (gauge) | The Java memory used in the Survivor space. Shown as byte |
logstash.jvm.mem.pools.survivor.used_in_bytes (gauge) | The peak Java memory used in the Survivor space. Shown as byte |
logstash.jvm.mem.pools.survivor.peak_max_in_bytes (gauge) | The peak maximum Java memory used in the Survivor space. Shown as byte |
logstash.jvm.mem.pools.survivor.max_in_bytes (gauge) | The maximum Java memory used in the Survivor space. Shown as byte |
logstash.jvm.mem.pools.survivor.committed_in_bytes (gauge) | The commited Java memory used in the Survivor space. Shown as byte |
logstash.jvm.mem.pools.old.peak_used_in_bytes (gauge) | The peak Java memory used in the Old generation. Shown as byte |
logstash.jvm.mem.pools.old.used_in_bytes (gauge) | The Java memory used in the Old generation. Shown as byte |
logstash.jvm.mem.pools.old.peak_max_in_bytes (gauge) | The peak maximum Java memory used in the Old generation. Shown as byte |
logstash.jvm.mem.pools.old.max_in_bytes (gauge) | The maximum Java memory used in the Old generation. Shown as byte |
logstash.jvm.mem.pools.old.committed_in_bytes (gauge) | The commited Java memory used in the Old generation. Shown as byte |
logstash.jvm.mem.pools.young.peak_used_in_bytes (gauge) | The peak Java memory used in the Young space. Shown as byte |
logstash.jvm.mem.pools.young.used_in_bytes (gauge) | The Java memory used in the Young generation. Shown as byte |
logstash.jvm.mem.pools.young.peak_max_in_bytes (gauge) | The peak maximum Java memory used in the Young generation. Shown as byte |
logstash.jvm.mem.pools.young.max_in_bytes (gauge) | The maximum Java memory used in the Young generation. Shown as byte |
logstash.jvm.mem.pools.young.committed_in_bytes (gauge) | The commited Java memory used in the Young generation. Shown as byte |
logstash.jvm.gc.collectors.old.collection_time_in_millis (gauge) | Garbage collection time spent in the Old generation. Shown as millisecond |
logstash.jvm.gc.collectors.old.collection_count (gauge) | Garbage collection count in the Old generation. |
logstash.jvm.gc.collectors.young.collection_time_in_millis (gauge) | Garbage collection time spent in the Young generation. Shown as millisecond |
logstash.jvm.gc.collectors.young.collection_count (gauge) | Garbage collection time spent in the Young generation. |
logstash.reloads.successes (gauge) | Number of successful configuration reloads. |
logstash.reloads.failures (gauge) | Number of failed configuration reloads. |
logstash.pipeline.events.duration_in_millis (gauge) | Events duration in the pipeline. Shown as millisecond |
logstash.pipeline.events.in (gauge) | Number of events into the pipeline. |
logstash.pipeline.events.out (gauge) | Number of events out from the pipeline. |
logstash.pipeline.events.filtered (gauge) | Number of events filtered. |
logstash.pipeline.reloads.successes (gauge) | Number of successful pipeline reloads. |
logstash.pipeline.reloads.failures (gauge) | Number of failed pipeline reloads. |
logstash.pipeline.plugins.inputs.events.out (gauge) | Number of events out from the input plugin. |
logstash.pipeline.plugins.inputs.events.queue_push_duration_in_millis (gauge) | Duration of queue push in the input plugin. Shown as millisecond |
logstash.pipeline.plugins.outputs.events.in (gauge) | Number of events into the output plugin. |
logstash.pipeline.plugins.outputs.events.out (gauge) | Number of events out from the output plugin. |
logstash.pipeline.plugins.outputs.events.duration_in_millis (gauge) | Duration of events in the output plugin. Shown as millisecond |
logstash.pipeline.plugins.filters.events.in (gauge) | Number of events into the filter plugin. |
logstash.pipeline.plugins.filters.events.out (gauge) | Number of events out from the filter plugin. |
logstash.pipeline.plugins.filters.events.duration_in_millis (gauge) | Duration of events in the filter plugin. Shown as millisecond |
The Logstash check does not include any events.
logstash.can_connect
:
Returns Critical
if the Agent cannot connect to Logstash to collect metrics; returns OK
otherwise.
logstash
-------
- instance #0 [ERROR]: "('Connection aborted.', error(111, 'Connection refused'))"
- Collected 0 metrics, 0 events & 1 service check
Check that the url
in conf.yaml
is correct.
If you need further help, contact Datadog support.
On this Page