Amazon Elastic Map Reduce
Nouvelles annonces sur les technologies sans serveur et réseau ainsi que sur le RUM (Real-User Monitoring) dévoilées à la conférence Dash ! Nouvelles annonces dévoilées à la conférence Dash !

Amazon Elastic Map Reduce

Crawler Crawler

Présentation

Amazon Elastic Map Reduce (Amazon EMR) est un service Web qui facilite le traitement rapide et rentable de grandes quantités de données.

Activez cette intégration pour visualiser dans Datadog vos métriques d’EMR.

Implémentation

Installation

Si vous ne l’avez pas déjà fait, configurez d’abord l’intégration Amazon Web Services.

Collecte de métriques

  1. Dans le carré d’intégration AWS, assurez-vous que l’option EMR est cochée dans la section concernant la collecte des métriques.

  2. Ajoutez les autorisations suivantes à votre stratégie IAM Datadog afin de recueillir des métriques Amazon EMR. Pour en savoir plus sur les stratégies EMR, consultez la documentation du site Web d’AWS.

    Autorisation AWSDescription
    elasticmapreduce:ListClustersÉnumère les clusters disponibles.
    elasticmapreduce:DescribeClusterAjoute des tags aux métriques EMR de CloudWatch.
  3. Installez l’intégration Datadog/AWS EMR.

Collecte de logs

Activer le logging

Configurez Amazon EMR de façon à ce que ses logs soient envoyés vers un compartiment S3 ou vers Cloudwatch.

Remarque : si vous envoyez vos logs vers un compartiment S3, assurez-vous que amazon_emr est défini en tant que Target prefix.

Envoyer des logs à Datadog

  1. Si vous ne l’avez pas déjà fait, configurez la fonction Lambda de collecte de logs AWS avec Datadog.
  2. Une fois la fonction Lambda installée, ajoutez manuellement un déclencheur sur le compartiment S3 ou sur le groupe de logs Cloudwatch qui contient vos logs Amazon EMR dans la console AWS :

Données collectées

Métriques

aws.elasticmapreduce.apps_completed
(gauge)
The average number of applications submitted to YARN that have completed.
aws.elasticmapreduce.apps_completed.sum
(gauge)
The sum of the number of applications submitted to YARN that have completed.
aws.elasticmapreduce.apps_failed
(gauge)
The average number of applications submitted to YARN that have failed to complete.
aws.elasticmapreduce.apps_failed.sum
(gauge)
The sum of the number of applications submitted to YARN that have failed to complete.
aws.elasticmapreduce.apps_killed
(gauge)
The average number of applications submitted to YARN that have been killed.
aws.elasticmapreduce.apps_killed.sum
(gauge)
The sum of the number of applications submitted to YARN that have been killed.
aws.elasticmapreduce.apps_pending
(gauge)
The average number of applications submitted to YARN that are in a pending state.
aws.elasticmapreduce.apps_pending.sum
(gauge)
The sum of the number of applications submitted to YARN that are in a pending state.
aws.elasticmapreduce.apps_running
(gauge)
The average number of applications submitted to YARN that are running.
aws.elasticmapreduce.apps_running.sum
(gauge)
The sum of the number of applications submitted to YARN that are running.
aws.elasticmapreduce.apps_submitted
(gauge)
The average number of applications submitted to YARN.
aws.elasticmapreduce.apps_submitted.sum
(gauge)
The sum of the number of applications submitted to YARN.
aws.elasticmapreduce.capacity_remaining_gb
(gauge)
The average amount of remaining HDFS disk capacity.
Shown as byte
aws.elasticmapreduce.capacity_remaining_gb.sum
(gauge)
The sum of the amount of remaining HDFS disk capacity.
Shown as byte
aws.elasticmapreduce.container_allocated
(gauge)
The average number of resource containers allocated by the ResourceManager.
aws.elasticmapreduce.container_allocated.sum
(gauge)
The sume of the number of resource containers allocated by the ResourceManager.
aws.elasticmapreduce.container_pending
(gauge)
The average number of containers in the queue that have not yet been allocated.
aws.elasticmapreduce.container_pending.sum
(gauge)
The sum of the number of containers in the queue that have not yet been allocated.
aws.elasticmapreduce.container_pending_ratio
(gauge)
The average percentage of containers in the queue that have not yet been allocated.
Shown as percent
aws.elasticmapreduce.container_reserved
(gauge)
The average number of containers reserved.
aws.elasticmapreduce.container_reserved.sum
(gauge)
The sum of the number of containers reserved.
aws.elasticmapreduce.core_nodes_pending
(gauge)
The average number of core nodes waiting to be assigned. All of the core nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_nodes_pending.sum
(gauge)
The sum of the number of core nodes waiting to be assigned. All of the core nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_nodes_running
(gauge)
The average number of core nodes working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.core_nodes_running.sum
(gauge)
The sume of the number of core nodes working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.corrupt_blocks
(gauge)
The average number of blocks that HDFS reports as corrupted.
Shown as block
aws.elasticmapreduce.corrupt_blocks.sum
(gauge)
The sum of the number of blocks that HDFS reports as corrupted.
Shown as block
aws.elasticmapreduce.dfs_fsnamesystem_pending_replication_blocks
(gauge)
The status of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requests.
aws.elasticmapreduce.hbase_backup_failed
(gauge)
Whether the last backup failed. This is set to 0 by default and updated to 1 if the previous backup attempt failed. This metric is only reported for HBase clusters.
aws.elasticmapreduce.hbase_most_recent_backup_duration
(gauge)
The amount of time it took the previous backup to complete. This metric is set regardless of whether the last completed backup succeeded or failed. While the backup is ongoing, this metric returns the number of minutes after the backup started. This metric is only reported for HBase clusters.
Shown as minute
aws.elasticmapreduce.hbase_time_since_last_successful_backup
(gauge)
The number of elapsed minutes after the last successful HBase backup started on your cluster. This metric is only reported for HBase clusters.
Shown as minute
aws.elasticmapreduce.hdfsbytes_read
(gauge)
The average number of bytes read from HDFS.
Shown as byte
aws.elasticmapreduce.hdfsbytes_read.sum
(gauge)
The sum of the number of bytes read from HDFS.
Shown as byte
aws.elasticmapreduce.hdfsbytes_written
(gauge)
The average number of bytes written to HDFS.
Shown as byte
aws.elasticmapreduce.hdfsbytes_written.sum
(gauge)
The sum of the number of bytes written to HDFS.
Shown as byte
aws.elasticmapreduce.hdfsutilization
(gauge)
The percentage of HDFS storage currently used.
Shown as percent
aws.elasticmapreduce.is_idle
(gauge)
Indicates that a cluster is no longer performing work, but is still alive and accruing charges. It is set to 1 if no tasks are running and no jobs are running, and set to 0 otherwise. This value is checked at five-minute intervals and a value of 1 indicates only that the cluster was idle when checked, not that it was idle for the entire five minutes.
aws.elasticmapreduce.jobs_failed
(gauge)
The average number of jobs in the cluster that have failed.
aws.elasticmapreduce.jobs_failed.sum
(gauge)
The sum of the number of jobs in the cluster that have failed.
aws.elasticmapreduce.jobs_running
(gauge)
The average number of jobs in the cluster that are currently running.
aws.elasticmapreduce.jobs_running.sum
(gauge)
The sum of the number of jobs in the cluster that are currently running.
aws.elasticmapreduce.live_data_nodes
(gauge)
The percentage of data nodes that are receiving work from Hadoop.
Shown as percent
aws.elasticmapreduce.live_task_trackers
(gauge)
The percentage of task trackers that are functional.
Shown as percent
aws.elasticmapreduce.map_slots_open
(gauge)
The average unused map task capacity. This is calculated as the maximum number of map tasks for a given cluster, less the total number of map tasks currently running in that cluster.
aws.elasticmapreduce.map_slots_open.sum
(gauge)
The sum of the unused map task capacity. This is calculated as the maximum number of map tasks for a given cluster, less the total number of map tasks currently running in that cluster.
aws.elasticmapreduce.memory_allocated_mb
(gauge)
The average amount of memory allocated to the cluster.
Shown as byte
aws.elasticmapreduce.memory_allocated_mb.sum
(gauge)
The sum of the amount of memory allocated to the cluster.
Shown as byte
aws.elasticmapreduce.memory_available_mb
(gauge)
The average amount of memory available to be allocated.
Shown as byte
aws.elasticmapreduce.memory_available_mb.sum
(gauge)
The sum of the amount of memory available to be allocated.
Shown as byte
aws.elasticmapreduce.memory_reserved_mb
(gauge)
The average amount of memory reserved.
Shown as byte
aws.elasticmapreduce.memory_reserved_mb.sum
(gauge)
The sum of the amount of memory reserved.
Shown as byte
aws.elasticmapreduce.memory_total_mb
(gauge)
The average total amount of memory in the cluster.
Shown as byte
aws.elasticmapreduce.memory_total_mb.sum
(gauge)
The sum of the total amount of memory in the cluster.
Shown as byte
aws.elasticmapreduce.missing_blocks
(gauge)
The average number of blocks in which HDFS has no replicas. These might be corrupt blocks.
Shown as block
aws.elasticmapreduce.missing_blocks.sum
(gauge)
The sum of the number of blocks in which HDFS has no replicas. These might be corrupt blocks.
Shown as block
aws.elasticmapreduce.mractive_nodes
(gauge)
The average number of nodes presently running MapReduce tasks or jobs.
Shown as node
aws.elasticmapreduce.mractive_nodes.sum
(gauge)
The sum of the number of nodes presently running MapReduce tasks or jobs.
Shown as node
aws.elasticmapreduce.mrdecommissioned_nodes
(gauge)
The average number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state.
aws.elasticmapreduce.mrdecommissioned_nodes.sum
(gauge)
The sum of the number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state.
aws.elasticmapreduce.mrlost_nodes
(gauge)
The average number of nodes allocated to MapReduce that have been marked in a LOST state.
Shown as node
aws.elasticmapreduce.mrlost_nodes.sum
(gauge)
The sum of the number of nodes allocated to MapReduce that have been marked in a LOST state.
Shown as node
aws.elasticmapreduce.mrrebooted_nodes
(gauge)
The average number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state.
Shown as node
aws.elasticmapreduce.mrrebooted_nodes.sum
(gauge)
The sum of the number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state.
Shown as node
aws.elasticmapreduce.mrtotal_nodes
(gauge)
The average number of nodes presently available to MapReduce jobs.
Shown as node
aws.elasticmapreduce.mrtotal_nodes.sum
(gauge)
The sum of the number of nodes presently available to MapReduce jobs.
Shown as node
aws.elasticmapreduce.mrunhealthy_nodes
(gauge)
The average number of nodes available to MapReduce jobs marked in an UNHEALTHY state.
Shown as node
aws.elasticmapreduce.mrunhealthy_nodes.sum
(gauge)
The sum of the number of nodes available to MapReduce jobs marked in an UNHEALTHY state.
Shown as node
aws.elasticmapreduce.no_of_black_listed_task_trackers
(gauge)
The average number of blackisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.no_of_black_listed_task_trackers.sum
(gauge)
The sum of the number of blackisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.no_of_gray_listed_task_trackers
(gauge)
The average number of graylisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.no_of_gray_listed_task_trackers.sum
(gauge)
The sum of the number of graylisted TaskTracker nodes.
Shown as node
aws.elasticmapreduce.pending_deletion_blocks
(gauge)
The average number of blocks marked for deletion.
Shown as block
aws.elasticmapreduce.pending_deletion_blocks.sum
(gauge)
The sum of the number of blocks marked for deletion.
Shown as block
aws.elasticmapreduce.reduce_slots_open
(gauge)
Average unused reduce task capacity. This is calculated as the maximum reduce task capacity for a given cluster, less the number of reduce tasks currently running in that cluster.
aws.elasticmapreduce.reduce_slots_open.sum
(gauge)
The sum of unused reduce task capacity. This is calculated as the maximum reduce task capacity for a given cluster, less the number of reduce tasks currently running in that cluster.
aws.elasticmapreduce.remaining_map_tasks
(gauge)
The average number of remaining map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. A remaining map task is one that is not in any of the following states: Running, Killed, or Completed.
Shown as task
aws.elasticmapreduce.remaining_map_tasks.sum
(gauge)
The sum of the number of remaining map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. A remaining map task is one that is not in any of the following states: Running, Killed, or Completed.
Shown as task
aws.elasticmapreduce.remaining_map_tasks_per_slot
(gauge)
The ratio of the total map tasks remaining to the total map slots available in the cluster.
aws.elasticmapreduce.remaining_reduce_tasks
(gauge)
The average number of remaining reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.
Shown as task
aws.elasticmapreduce.remaining_reduce_tasks.sum
(gauge)
The sum of the number of remaining reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.
Shown as task
aws.elasticmapreduce.running_map_tasks
(gauge)
The average number of running map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.
Shown as task
aws.elasticmapreduce.running_map_tasks.sum
(gauge)
The sum of the number of running map tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.
Shown as task
aws.elasticmapreduce.running_reduce_tasks
(gauge)
The average number of running reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.
Shown as task
aws.elasticmapreduce.running_reduce_tasks.sum
(gauge)
The sum of the number of running reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.
Shown as task
aws.elasticmapreduce.s_3bytes_read
(gauge)
The average number of bytes read from Amazon S3.
Shown as byte
aws.elasticmapreduce.s_3bytes_read.sum
(gauge)
The sum of the number of bytes read from Amazon S3.
Shown as byte
aws.elasticmapreduce.s_3bytes_written
(gauge)
The average number of bytes written to Amazon S3.
Shown as byte
aws.elasticmapreduce.s_3bytes_written.sum
(gauge)
The sum of the number of bytes written to Amazon S3.
Shown as byte
aws.elasticmapreduce.task_nodes_pending
(gauge)
The average number of task nodes waiting to be assigned. All of the task nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.task_nodes_pending.sum
(gauge)
The sum of the number of task nodes waiting to be assigned. All of the task nodes requested may not be immediately available; this metric reports the pending requests. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.task_nodes_running
(gauge)
The average number of task nodes working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.task_nodes_running.sum
(gauge)
The sum of the number of task nodes working. Data points for this metric are reported only when a corresponding instance group exists.
Shown as node
aws.elasticmapreduce.total_load
(gauge)
The average total number of concurrent data transfers.
aws.elasticmapreduce.total_load.sum
(gauge)
The sum of the total number of concurrent data transfers.
aws.elasticmapreduce.total_map_tasks
(gauge)
The average total number of map tasks.
Shown as task
aws.elasticmapreduce.total_map_tasks.sum
(gauge)
The sum of the total number of map tasks.
Shown as task
aws.elasticmapreduce.total_reduce_tasks
(gauge)
The average total number of reduce tasks.
Shown as task
aws.elasticmapreduce.total_reduce_tasks.sum
(gauge)
The sum of the total number of reduce tasks.
Shown as task
aws.elasticmapreduce.under_replicated_blocks
(gauge)
The average number of blocks that need to be replicated one or more times.
Shown as block
aws.elasticmapreduce.under_replicated_blocks.sum
(gauge)
The sum of the number of blocks that need to be replicated one or more times.
Shown as block

Chacune des métriques récupérées à partir d’AWS se voit assigner les mêmes tags que ceux qui apparaissent dans la console AWS, y compris, mais sans s’y limiter, le hostname et les groupes de sécurité.

Événements

L’intégration AWS Elastic MapReduce n’inclut aucun événement.

Checks de service

L’intégration AWS Elastic MapReduce n’inclut aucun check de service.

Dépannage

Besoin d’aide ? Contactez l’assistance Datadog.