Amazon EMR

개요

Amazon EMR는 대량의 데이터를 쉽고 빠르며 비용 효율적으로 처리할 수 있게 도와드리는 웹 서비스입니다.

본 통합을 활성화하여 Datadog에서 EMR 메트릭을 확인하세요.

설정

설치

아직 설정하지 않은 경우 먼저 Amazon Web Services 통합을 설정하세요.

메트릭 수집

  1. AWS 통합 페이지에서 Metric Collection 탭의 EMR이 활성화되어 있는지 확인하세요.

  2. Amazon EMR 메트릭을 수집하려면 다음 권한을 Datadog IAM 정책에 추가합니다. 자세한 정보는 AWS 웹사이트의 EMR 정책을 참조하세요.

    AWS 권한설명
    elasticmapreduce:ListClusters이용 가능한 클러스터 목록.
    elasticmapreduce:DescribeCluster클라우드와치(CloudWatch) EMR 메트릭에 태그 추가.

  3. Datadog - Amazon EMR 통합을 설치하세요.

로그 수집

로깅 활성화

Amazon EMR를 설정하여 S3 버킷 또는 클라우드와치(CloudWatch) 중 하나로 로그를 전송합니다.

참고: S3 버킷에 로깅하는 경우 amazon_emr이 대상 접두어( Target prefix)로 설정되어 있는지 확인하세요.

Datadog에 로그 전송

  1. 이미 하지 않은 경우 Datadog 포워더 람다 함수를 설정하세요.

  2. 람다 함수가 설치되면 AWS 콘솔에서 Amazon EMR 로그를 포함하는 S3 버킷 또는 클라우드와치(CloudWatch) 로그 그룹에 수동 트리거를 추가합니다.

수집한 데이터

메트릭

aws.elasticmapreduce.apps_completed
(gauge)		The average number of applications submitted to YARN that have completed. (Hadoop v2 only)
aws.elasticmapreduce.apps_completed.sum
(gauge)		The sum of the number of applications submitted to YARN that have completed. (Hadoop v2 only)
aws.elasticmapreduce.apps_failed
(gauge)		The average number of applications submitted to YARN that have failed to complete. (Hadoop v2 only)
aws.elasticmapreduce.apps_failed.sum
(gauge)		The sum of the number of applications submitted to YARN that have failed to complete. (Hadoop v2 only)
aws.elasticmapreduce.apps_killed
(gauge)		The average number of applications submitted to YARN that have been killed. (Hadoop v2 only)
aws.elasticmapreduce.apps_killed.sum
(gauge)		The sum of the number of applications submitted to YARN that have been killed. (Hadoop v2 only)
aws.elasticmapreduce.apps_pending
(gauge)		The average number of applications submitted to YARN that are in a pending state. (Hadoop v2 only)
aws.elasticmapreduce.apps_pending.sum
(gauge)		The sum of the number of applications submitted to YARN that are in a pending state. (Hadoop v2 only)
aws.elasticmapreduce.apps_running
(gauge)		The average number of applications submitted to YARN that are running. (Hadoop v2 only)
aws.elasticmapreduce.apps_running.sum
(gauge)		The sum of the number of applications submitted to YARN that are running. (Hadoop v2 only)
aws.elasticmapreduce.apps_submitted
(gauge)		The average number of applications submitted to YARN. (Hadoop v2 only)
aws.elasticmapreduce.apps_submitted.sum
(gauge)		The sum of the number of applications submitted to YARN. (Hadoop v2 only)
aws.elasticmapreduce.backup_failed
(count)		Whether the last backup failed. (Hadoop v1 only)
aws.elasticmapreduce.capacity_remaining_gb
(gauge)		The average amount of remaining HDFS disk capacity. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.capacity_remaining_gb.sum
(gauge)		The sum of the amount of remaining HDFS disk capacity. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.container_allocated
(gauge)		The average number of resource containers allocated by the ResourceManager. (Hadoop v2 only)
aws.elasticmapreduce.container_allocated.sum
(gauge)		The sum of the number of resource containers allocated by the ResourceManager. (Hadoop v2 only)
aws.elasticmapreduce.container_pending
(gauge)		The average number of containers in the queue that have not yet been allocated. (Hadoop v2 only)
aws.elasticmapreduce.container_pending.sum
(gauge)		The sum of the number of containers in the queue that have not yet been allocated. (Hadoop v2 only)
aws.elasticmapreduce.container_pending_ratio
(gauge)		The average percentage of containers in the queue that have not yet been allocated. (Hadoop v2 only)
Shown as percent
aws.elasticmapreduce.container_reserved
(gauge)		The average number of containers reserved. (Hadoop v2 only)
aws.elasticmapreduce.container_reserved.sum
(gauge)		The sum of the number of containers reserved. (Hadoop v2 only)
aws.elasticmapreduce.core_nodes_pending
(gauge)		The average number of core nodes waiting to be assigned. All of the core nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_nodes_pending.sum
(gauge)		The sum of the number of core nodes waiting to be assigned. All of the core nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_nodes_requested
(gauge)		The average number of core nodes in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.core_nodes_requested.sum
(gauge)		The sum of the number of core nodes in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.core_nodes_running
(gauge)		The average number of core nodes working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_nodes_running.sum
(gauge)		The sum of the number of core nodes working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_units_requested
(gauge)		The average number of core units in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.core_units_requested.sum
(gauge)		The sum of the number of core units in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.core_units_running
(gauge)		The target number of core units working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_units_running.sum
(gauge)		The sum of the number of core units working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_vcpu_requested
(gauge)		The average number of core vCPUs in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.core_vcpu_requested.sum
(gauge)		The sum of the number of core vCPUs in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.core_vcpu_running
(gauge)		The average number of core vCPUs working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_vcpu_running.sum
(gauge)		The sum of the number of core vCPUs working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.corrupt_blocks
(gauge)		The average number of blocks that HDFS reports as corrupted. (Hadoop v2 only)
Shown as block
aws.elasticmapreduce.corrupt_blocks.sum
(gauge)		The sum of the number of blocks that HDFS reports as corrupted. (Hadoop v2 only)
Shown as block
aws.elasticmapreduce.dfs_fsnamesystem_pending_replication_blocks
(gauge)		The status of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requests. (Hadoop v2 only)
aws.elasticmapreduce.hbase_backup_failed
(gauge)		Whether the last backup failed. This is set to 0 by default and updated to 1 if the previous backup attempt failed. This metric is only reported for HBase clusters. (Hadoop v2 only)
aws.elasticmapreduce.hbase_most_recent_backup_duration
(gauge)		The amount of time it took the previous backup to complete. This metric is set regardless of whether the last completed backup succeeded or failed. While the backup is ongoing, this metric returns the number of minutes after the backup started. This metric is only reported for HBase clusters.
Shown as minute
aws.elasticmapreduce.hbase_time_since_last_successful_backup
(gauge)		The number of elapsed minutes after the last successful HBase backup started on your cluster. This metric is only reported for HBase clusters.
Shown as minute
aws.elasticmapreduce.hdfsbytes_read
(gauge)		The average number of bytes read from HDFS.
Shown as byte
aws.elasticmapreduce.hdfsbytes_read.sum
(gauge)		The sum of the number of bytes read from HDFS.
Shown as byte
aws.elasticmapreduce.hdfsbytes_written
(gauge)		The average number of bytes written to HDFS.
Shown as byte
aws.elasticmapreduce.hdfsbytes_written.sum
(gauge)		The sum of the number of bytes written to HDFS.
Shown as byte
aws.elasticmapreduce.hdfsutilization
(gauge)		The percentage of HDFS storage currently used.
Shown as percent
aws.elasticmapreduce.is_idle
(gauge)		Indicates that a cluster is no longer performing work, but is still alive and accruing charges. It is set to 1 if no tasks are running and no jobs are running, and set to 0 otherwise. This value is checked at five-minute intervals and a value of 1 indicates only that the cluster was idle when checked, not that it was idle for the entire five minutes.
aws.elasticmapreduce.jobs_failed
(gauge)		The average number of jobs in the cluster that have failed. (Hadoop v1 only)
aws.elasticmapreduce.jobs_failed.sum
(gauge)		The sum of the number of jobs in the cluster that have failed. (Hadoop v1 only)
aws.elasticmapreduce.jobs_running
(gauge)		The average number of jobs in the cluster that are currently running. (Hadoop v1 only)
aws.elasticmapreduce.jobs_running.sum
(gauge)		The sum of the number of jobs in the cluster that are currently running. (Hadoop v1 only)
aws.elasticmapreduce.live_data_nodes
(gauge)		The percentage of data nodes that are receiving work from Hadoop.
Shown as percent
aws.elasticmapreduce.live_task_trackers
(gauge)		The percentage of task trackers that are functional. (Hadoop v1 only)
Shown as percent
aws.elasticmapreduce.map_slots_open
(gauge)		The average unused map task capacity. This is calculated as the maximum number of map tasks for a given cluster, less the total number of map tasks currently running in that cluster. (Hadoop v1 only)
aws.elasticmapreduce.map_slots_open.sum
(gauge)		The sum of the unused map task capacity. This is calculated as the maximum number of map tasks for a given cluster, less the total number of map tasks currently running in that cluster. (Hadoop v1 only)
aws.elasticmapreduce.memory_allocated_mb
(gauge)		The average amount of memory allocated to the cluster. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.memory_allocated_mb.sum
(gauge)		The sum of the amount of memory allocated to the cluster. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.memory_available_mb
(gauge)		The average amount of memory available to be allocated. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.memory_available_mb.sum
(gauge)		The sum of the amount of memory available to be allocated. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.memory_reserved_mb
(gauge)		The average amount of memory reserved. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.memory_reserved_mb.sum
(gauge)		The sum of the amount of memory reserved. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.memory_total_mb
(gauge)		The average total amount of memory in the cluster. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.memory_total_mb.sum
(gauge)		The sum of the total amount of memory in the cluster. (Hadoop v2 only)
Shown as byte
aws.elasticmapreduce.missing_blocks
(gauge)		The average number of blocks in which HDFS has no replicas. These might be corrupt blocks.
Shown as block
aws.elasticmapreduce.missing_blocks.sum
(gauge)		The sum of the number of blocks in which HDFS has no replicas. These might be corrupt blocks.
Shown as block
aws.elasticmapreduce.mractive_nodes
(gauge)		The average number of nodes presently running MapReduce tasks or jobs. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mractive_nodes.sum
(gauge)		The sum of the number of nodes presently running MapReduce tasks or jobs. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrdecommissioned_nodes
(gauge)		The average number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrdecommissioned_nodes.sum
(gauge)		The sum of the number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrlost_nodes
(gauge)		The average number of nodes allocated to MapReduce that have been marked in a LOST state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrlost_nodes.sum
(gauge)		The sum of the number of nodes allocated to MapReduce that have been marked in a LOST state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrrebooted_nodes
(gauge)		The average number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrrebooted_nodes.sum
(gauge)		The sum of the number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrtotal_nodes
(gauge)		The average number of nodes presently available to MapReduce jobs. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrtotal_nodes.sum
(gauge)		The sum of the number of nodes presently available to MapReduce jobs. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrunhealthy_nodes
(gauge)		The average number of nodes available to MapReduce jobs marked in an UNHEALTHY state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.mrunhealthy_nodes.sum
(gauge)		The sum of the number of nodes available to MapReduce jobs marked in an UNHEALTHY state. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.multi_master_instance_group_nodes_requested
(count)		The number of requested master nodes. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.multi_master_instance_group_nodes_running
(count)		The number of running master nodes. (Hadoop v2 only)
Shown as node
aws.elasticmapreduce.multi_master_instance_group_nodes_running_percentage
(gauge)		The percentage of master nodes that are running over the requested master node instance count. (Hadoop v2 only)
Shown as percent
aws.elasticmapreduce.no_of_black_listed_task_trackers
(gauge)		The average number of blacklisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.no_of_black_listed_task_trackers.sum
(gauge)		The sum of the number of blacklisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.no_of_gray_listed_task_trackers
(gauge)		The average number of graylisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.no_of_gray_listed_task_trackers.sum
(gauge)		The sum of the number of graylisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.pending_deletion_blocks
(gauge)		The average number of blocks marked for deletion. (Hadoop v2 only)
Shown as block
aws.elasticmapreduce.pending_deletion_blocks.sum
(gauge)		The sum of the number of blocks marked for deletion. (Hadoop v2 only)
Shown as block
aws.elasticmapreduce.reduce_slots_open
(gauge)		Average unused reduce task capacity. This is calculated as the maximum reduce task capacity for a given cluster, less the number of reduce tasks currently running in that cluster. (Hadoop v1 only)
aws.elasticmapreduce.reduce_slots_open.sum
(gauge)		The sum of unused reduce task capacity. This is calculated as the maximum reduce task capacity for a given cluster, less the number of reduce tasks currently running in that cluster. (Hadoop v1 only)
aws.elasticmapreduce.remaining_map_tasks
(gauge)		The average number of remaining map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. A remaining map task is one that is not in any of the following states: Running, Killed, or Completed. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.remaining_map_tasks.sum
(gauge)		The sum of the number of remaining map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. A remaining map task is one that is not in any of the following states: Running, Killed, or Completed. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.remaining_map_tasks_per_slot
(gauge)		The ratio of the total map tasks remaining to the total map slots available in the cluster. (Hadoop v1 only)
aws.elasticmapreduce.remaining_reduce_tasks
(gauge)		The average number of remaining reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.remaining_reduce_tasks.sum
(gauge)		The sum of the number of remaining reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.running_map_tasks
(gauge)		The average number of running map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.running_map_tasks.sum
(gauge)		The sum of the number of running map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.running_reduce_tasks
(gauge)		The average number of running reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.running_reduce_tasks.sum
(gauge)		The sum of the number of running reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. (Hadoop v1 only)
Shown as task
aws.elasticmapreduce.s_3bytes_read
(gauge)		The average number of bytes read from Amazon S3.
Shown as byte
aws.elasticmapreduce.s_3bytes_read.sum
(gauge)		The sum of the number of bytes read from Amazon S3.
Shown as byte
aws.elasticmapreduce.s_3bytes_written
(gauge)		The average number of bytes written to Amazon S3.
Shown as byte
aws.elasticmapreduce.s_3bytes_written.sum
(gauge)		The sum of the number of bytes written to Amazon S3.
Shown as byte
aws.elasticmapreduce.task_nodes_pending
(gauge)		The average number of task nodes waiting to be assigned. All of the task nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.task_nodes_pending.sum
(gauge)		The sum of the number of task nodes waiting to be assigned. All of the task nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.task_nodes_requested
(gauge)		The average number of task nodes in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.task_nodes_requested.sum
(gauge)		The sum of the number of task nodes in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.task_nodes_running
(gauge)		The average number of task nodes working. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.task_nodes_running.sum
(gauge)		The sum of the number of task nodes working. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.task_units_requested
(gauge)		The average number of task units in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.task_units_requested.sum
(gauge)		The sum of the number of task units in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.task_units_running
(gauge)		The average number of task units working. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.task_units_running.sum
(gauge)		The sum of the number of task units working. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.task_vcpu_requested
(gauge)		The average number of task vCPUs in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.task_vcpu_requested.sum
(gauge)		The sum of the number of task vCPUs in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.task_vcpu_running
(gauge)		The average number of task vCPUs working. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.task_vcpu_running.sum
(gauge)		The sum of the number of task vCPUs working. Data points for this metric are reported only when a corresponding instance group exists. (Hadoop v1 only)
Shown as node
aws.elasticmapreduce.total_load
(gauge)		The average total number of concurrent data transfers.
aws.elasticmapreduce.total_load.sum
(gauge)		The sum of the total number of concurrent data transfers.
aws.elasticmapreduce.total_map_tasks
(gauge)		The average total number of map tasks.
Shown as task
aws.elasticmapreduce.total_map_tasks.sum
(gauge)		The sum of the total number of map tasks.
Shown as task
aws.elasticmapreduce.total_nodes_requested
(gauge)		The sum total number of nodes in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.total_nodes_requested.average
(gauge)		The average of total number of nodes in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.total_nodes_running
(gauge)		The current average number of nodes available in a running cluster.
Shown as node
aws.elasticmapreduce.total_nodes_running.sum
(gauge)		The current sum of nodes available in a running cluster.
Shown as node
aws.elasticmapreduce.total_reduce_tasks
(gauge)		The average total number of reduce tasks.
Shown as task
aws.elasticmapreduce.total_reduce_tasks.sum
(gauge)		The sum of the total number of reduce tasks.
Shown as task
aws.elasticmapreduce.total_units_requested
(gauge)		The average total number of units in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.total_units_requested.sum
(gauge)		The sum of total number of units in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.total_units_running
(gauge)		The current average number of units available in a running cluster.
Shown as node
aws.elasticmapreduce.total_units_running.sum
(gauge)		The current sum of units available in a running cluster.
Shown as node
aws.elasticmapreduce.total_vcpu_requested
(gauge)		The average total number of vCPUs in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.total_vcpu_requested.sum
(gauge)		The sum of total number of vCPUs in a cluster as determined by managed scaling.
Shown as node
aws.elasticmapreduce.total_vcpu_running
(gauge)		The current average number of vCPUs available in a running cluster.
Shown as node
aws.elasticmapreduce.total_vcpu_running.sum
(gauge)		The current sum of vCPUs available in a running cluster.
Shown as node
aws.elasticmapreduce.under_replicated_blocks
(gauge)		The average number of blocks that need to be replicated one or more times. (Hadoop v2 only)
Shown as block
aws.elasticmapreduce.under_replicated_blocks.sum
(gauge)		The sum of the number of blocks that need to be replicated one or more times. (Hadoop v2 only)
Shown as block
aws.elasticmapreduce.yarnmemory_available_percentage
(gauge)		The percentage of remaining memory available to YARN. (Hadoop v2 only)
Shown as percent

AWS에서 검색된 각 메트릭에는 AWS 콘솔에 나타나는 것과 동일한 태그가 할당됩니다, 호스트 이름, 보안 그룹 등을 포함하되 이에 국한되지 않습니다.

이벤트

Amazon EMR 통합은 이벤트를 포함하지 않습니다.

서비스 점검

Amazon EMR 통합은 서비스 점검을 포함하지 않습니다.

트러블슈팅

도움이 필요하신가요? Datadog 고객 지원팀에 문의해주세요.