Información general

AWS Glue es un servicio de ETL (extracción, transformación y carga) totalmente gestionado que hace más simple y rentable la categorización, limpieza, mejora y traslado fiable de datos entre varios almacenes de datos.

Habilita esta integración para ver todas tus métricas de Glue en Datadog.

Configuración

Instalación

Si aún no lo has hecho, configura primero la integración de Amazon Web Services.

Recopilación de métricas

  1. En la página de la integración de AWS, asegúrate de que Glue está activado en la pestaña Metric Collection.
  2. Instala la integración de Datadog y AWS Glue.

APM

Activar logging

Configura AWS Glue para enviar logs a un bucket de S3 o a CloudWatch.

Nota: Si vas a loguear en un bucket de S3, asegúrate de que amazon_glue está configurado como Target prefix (Prefijo de destino).

Enviar logs a Datadog

  1. Si aún no lo has hecho, configura la función de Lambda de Datadog Forwarder.

  2. Una vez instalada la función de Lambda, añade manualmente un activador en el bucket de S3 o grupo de logs de CloudWatch que contenga tus logs de AWS Glue en la consola de AWS:

Datos recopilados

Métricas

aws.glue.driver.executor_allocation_manager.executors.number_all_executors
(gauge)
The number of actively running job executors.
aws.glue.driver.executor_allocation_manager.executors.number_max_needed_executors
(gauge)
The number of maximum (actively running and pending) job executors needed to satisfy the current load.
aws.glue.glue_alljvm_heap_usage
(gauge)
The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for all executors.
Shown as percent
aws.glue.glue_alljvm_heap_used
(gauge)
The number of memory bytes used by the JVM heap for all executors.
Shown as byte
aws.glue.glue_alls_3filesystem_readbytes
(gauge)
The average number of bytes read from Amazon S3 all executors since the previous report.
aws.glue.glue_allsystem_cpu_system_load
(gauge)
The average fraction of CPU system load used (scale: 0-1) by all executors.
Shown as percent
aws.glue.glue_driver_aggregate_bytes_read
(count)
The number of bytes read from all data sources by all completed Spark tasks running in all executors.
Shown as byte
aws.glue.glue_driver_aggregate_elapsed_time
(count)
The ETL elapsed time in milliseconds (does not include the job bootstrap times).
Shown as millisecond
aws.glue.glue_driver_aggregate_num_completed_stages
(count)
The number of completed stages in the job.
aws.glue.glue_driver_aggregate_num_completed_tasks
(count)
The number of completed tasks in the job.
aws.glue.glue_driver_aggregate_num_failed_tasks
(count)
The number of failed tasks.
aws.glue.glue_driver_aggregate_num_killed_tasks
(count)
The number of tasks killed.
aws.glue.glue_driver_aggregate_records_read
(count)
The number of records read from all data sources by all completed Spark tasks running in all executors.
aws.glue.glue_driver_aggregate_shuffle_bytes_written
(count)
The number of bytes written by all executors to shuffle data between them since the previous report.
aws.glue.glue_driver_aggregate_shuffle_local_bytes_read
(count)
The number of bytes read by all executors to shuffle data between them since the previous report.
aws.glue.glue_driver_block_manager_disk_disk_space_used_mb
(gauge)
The average number of megabytes of disk spaced used across all executors.
aws.glue.glue_driver_jvm_heap_usage
(gauge)
The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for driver.
Shown as percent
aws.glue.glue_driver_jvm_heap_used
(gauge)
The number of memory bytes used by the JVM heap for the driver.
Shown as byte
aws.glue.glue_driver_s3_filesystem_readbytes
(gauge)
The average number of bytes read from Amazon S3 by the driver since the previous report.
aws.glue.glue_driver_s3_filesystem_writebytes
(gauge)
The average number of bytes written to Amazon S3 by the driver since the previous report.
aws.glue.glue_driver_system_cpu_system_load
(gauge)
The average fraction of CPU system load used (scale: 0-1) by the driver.
Shown as percent
aws.glue.glue_executor_id_jvm_heap_usage
(gauge)
The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for executor identified.
Shown as percent
aws.glue.glue_executor_id_jvm_heap_used
(gauge)
The number of memory bytes used by the JVM heap for the executor identified.
Shown as byte
aws.glue.glue_executor_id_system_cpu_system_load
(gauge)
The average fraction of CPU system load used (scale: 0-1) by the executor identified.
Shown as percent
aws.glue.glue_executor_ids_3_filesystem_readbytes
(gauge)
The average number of bytes read from Amazon S3 by the executor identified since the previous report.
aws.glue.glue_executor_ids_3_filesystem_writebytes
(gauge)
The average number of bytes written to Amazon S3 by the executor identified since the previous report.

Eventos

La integración de AWS Glue no incluye ningún evento.

Checks de servicio

La integración de AWS Glue no incluye ningún check de servicio.

Resolución de problemas

¿Necesitas ayuda? Ponte en contacto con el servicio de asistencia de Datadog.