Logging is here!

AWS ES

Crawler Crawler

Overview

Amazon Elasticsearch Service is a managed service that makes it easy to deploy, operate, and scale Elasticsearch in the AWS Cloud.

Enable this integration to see custom tags and metrics for your ES clusters in Datadog.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first.

Configuration

  1. In the AWS integration tile, ensure that ES is checked under metric collection.

  2. Add those permissions to your Datadog IAM policy in order to collect Amazon ES metrics:

    • es:ListTags: Add custom ES domain tags to ES metrics
    • es:ListDomainNames: Add custom ES domain tags to ES metrics
    • es:DescribeElasticsearchDomains: Add custom ES domain tags to ES metrics

    For more information on ES policies, review the documentation on the AWS website.

  3. Install the Datadog - AWS ES integration.

Data Collected

Metrics

aws.es.automated_snapshot_failure
(gauge)
The number of failed automated snapshots for the cluster.
aws.es.cluster_statusgreen
(gauge)
Indicates whether all index shards are allocated to nodes in the cluster.
aws.es.cluster_statusred
(gauge)
Indicates whether both primary and replica shards of at least one index are not allocated to nodes in a cluster.
aws.es.cluster_statusyellow
(gauge)
Indicates whether replica shards are not allocated to nodes in a cluster.
aws.es.cluster_used_space
(gauge)
The total used space in megabytes for a cluster.
shown as mebibyte
aws.es.cpucredit_balance
(gauge)
The remaining CPU credits available for data nodes in the cluster.
aws.es.cpuutilization
(gauge)
The average percentage of CPU resources used across all nodes in the cluster.
shown as percent
aws.es.cpuutilization.minimum
(gauge)
The minimum percentage of CPU resources used by any node in the cluster.
shown as percent
aws.es.cpuutilization.maximum
(gauge)
The maximum percentage of CPU resources used by any node in the cluster.
shown as percent
aws.es.deleted_documents
(gauge)
The total number of deleted documents across all indices in the cluster.
aws.es.disk_queue_depth
(gauge)
The average number of pending input and output (I/O) requests for an EBS volume. across all nodes in the cluster
aws.es.disk_queue_depth.minimum
(gauge)
The minimum number for any node in the cluster of pending input and output (I/O) requests for an EBS volume.
aws.es.disk_queue_depth.maximum
(gauge)
The maximum number for any node in the cluster of pending input and output (I/O) requests for an EBS volume.
aws.es.free_storage_space
(gauge)
The average free space, in megabytes, across all the data nodes in a cluster.
shown as mebibyte
aws.es.free_storage_space.minimum
(gauge)
The free space, in megabytes, for the single data node with the least available free space in a cluster.
shown as mebibyte
aws.es.free_storage_space.maximum
(gauge)
The free space, in megabytes, for the single data node with the most available free space in a cluster.
shown as mebibyte
aws.es.free_storage_space.sum
(gauge)
The free space, in megabytes, for all data nodes in the cluster.
shown as mebibyte
aws.es.jvmmemory_pressure
(gauge)
The average percentage of the Java heap used for all data nodes in the cluster.
shown as percent
aws.es.jvmmemory_pressure.minimum
(gauge)
The minimum percentage of the Java heap used by any data node in the cluster.
shown as percent
aws.es.jvmmemory_pressure.maximum
(gauge)
The maximum percentage of the Java heap used by any data node in the cluster.
shown as percent
aws.es.master_cpuutilization
(gauge)
The maximum percentage of CPU resources used by the dedicated master nodes.
shown as percent
aws.es.master_free_storage_space
(gauge)
This metric is not relevant and can be ignored. The service does not use master nodes as data nodes.
shown as mebibyte
aws.es.master_jvmmemory_pressure
(gauge)
The maximum percentage of the Java heap used for all dedicated master nodes in the cluster.
shown as percent
aws.es.master_cpucredit_balance
(gauge)
The remaining CPU credits available for dedicated master nodes in the cluster.
aws.es.nodes
(gauge)
The number of nodes in the Amazon ES cluster.
aws.es.read_iops
(gauge)
The number of input and output (I/O) operations per second for read operations on EBS volumes.
shown as operation
aws.es.read_iops.minimum
(gauge)
The minimum number for any node of input and output (I/O) operations per second for read operations on EBS volumes.
shown as operation
aws.es.read_iops.maximum
(gauge)
The maximum number for any node of input and output (I/O) operations per second for read operations on EBS volumes.
shown as operation
aws.es.read_latency
(gauge)
The latency, in seconds, for read operations on EBS volumes.
shown as second
aws.es.read_latency.minimum
(gauge)
The minimum latency for any node, in seconds, for read operations on EBS volumes.
shown as second
aws.es.read_latency.maximum
(gauge)
The maximum latency for any node, in seconds, for read operations on EBS volumes.
shown as second
aws.es.read_throughput
(gauge)
The throughput, in bytes per second, for read operations on EBS volumes.
shown as byte
aws.es.read_throughput.minimum
(gauge)
The minimum throughput for any node, in bytes per second, for read operations on EBS volumes.
shown as byte
aws.es.read_throughput.maximum
(gauge)
The maximum throughput for any node, in bytes per second, for read operations on EBS volumes.
shown as byte
aws.es.searchable_documents
(gauge)
The total number of searchable documents across all indices in the cluster.
aws.es.write_iops
(gauge)
The number of input and output (I/O) operations per second for write operations on EBS volumes.
shown as operation
aws.es.write_iops.minimum
(gauge)
The minimum number for any node of input and output (I/O) operations per second for write operations on EBS volumes.
shown as operation
aws.es.write_iops.maximum
(gauge)
The maximum number for any node of input and output (I/O) operations per second for write operations on EBS volumes.
shown as operation
aws.es.write_latency
(gauge)
The latency, in seconds, for write operations on EBS volumes.
shown as second
aws.es.write_latency.minimum
(gauge)
The minimum latency for any node, in seconds, for write operations on EBS volumes.
shown as second
aws.es.write_latency.maximum
(gauge)
The maximum latency for any node, in seconds, for write operations on EBS volumes.
shown as second
aws.es.write_throughput
(gauge)
The throughput, in bytes per second, for write operations on EBS volumes.
shown as byte
aws.es.write_throughput.minimum
(gauge)
The minimum throughput for any node, in bytes per second, for write operations on EBS volumes.
shown as byte
aws.es.write_throughput.maximum
(gauge)
The maximum throughput for any node, in bytes per second, for write operations on EBS volumes.
shown as byte

Each of the metrics retrieved from AWS will be assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.

Events

The AWS ES integration does not include any event at this time.

Service Checks

The AWS ES integration does not include any service check at this time.

Troubleshooting

Need help? Contact Datadog Support.