Amazon Glue
セキュリティモニタリングが使用可能です セキュリティモニタリングが使用可能です

Amazon Glue

Crawler Crawler

概要

Amazon Glue は、シンプルかつコスト効率よくデータを分類、クリーニング、補完したり、さまざまなデータストア間のデータ移動を高い信頼性で行うことができるフルマネージド型 ETL (抽出、変換、ロード) サービスです。

このインテグレーションを有効にすると、Datadog にすべての Glue メトリクスを表示できます。

セットアップ

インストール

Amazon Web Services インテグレーションをまだセットアップしていない場合は、最初にセットアップします。

メトリクスの収集

  1. AWS インテグレーションタイルのメトリクス収集で、Glue をオンにします。
  2. Datadog - Amazon Glue インテグレーションをインストールします。

ログの収集

ログの有効化

Amazon Glue から S3 バケットまたは CloudWatch のいずれかにログを送信するよう構成します。

: S3 バケットにログを送る場合は、Target prefixamazon_glue に設定されているかを確認してください。

Datadog へのログの送信

  1. Datadog ログコレクション AWS Lambda 関数 をまだ設定していない場合は、設定を行ってください。
  2. lambda 関数がインストールされたら、AWS コンソールから、Amazon Glue ログを含む S3 バケットまたは CloudWatch のロググループに手動でトリガーを追加します。

収集データ

メトリクス

aws.glue.driver.aggregate.bytesRead
(count)
The number of bytes read from all data sources by all completed Spark tasks running in all executors.
Shown as byte
aws.glue.driver.aggregate.elapsedTime
(count)
The ETL elapsed time in milliseconds (does not include the job bootstrap times).
Shown as millisecond
aws.glue.driver.aggregate.numCompletedStages
(count)
The number of completed stages in the job.
aws.glue.driver.aggregate.numCompletedTasks
(count)
The number of completed tasks in the job.
aws.glue.driver.aggregate.numFailedTasks
(count)
The number of failed tasks.
aws.glue.driver.aggregate.numKilledTasks
(count)
The number of tasks killed.
aws.glue.driver.aggregate.recordsRead
(count)
The number of records read from all data sources by all completed Spark tasks running in all executors.
aws.glue.driver.aggregate.shuffleBytesWritten
(count)
The number of bytes written by all executors to shuffle data between them since the previous report.
aws.glue.driver.aggregate.shuffleLocalBytesRead
(count)
The number of bytes read by all executors to shuffle data between them since the previous report.
aws.glue.driver.block_manager.disk.disk_space_used_mb
(count)
The average number of megabytes of disk spaced used across all executors.
aws.glue.driver.executor_allocation_manager.executors.number_all_executors
(count)
The number of actively running job executors.
aws.glue.driver.executor_allocation_manager.executors.number_max_needed_executors
(count)
The number of maximum (actively running and pending) job executors needed to satisfy the current load.
aws.glue.driver.jvm.heap.usage
(count)
The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for driver.
Shown as percent
aws.glue.executor_id.jvm.heap.usage
(count)
The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for executor identified.
Shown as percent
aws.glue.ALL.jvm.heap.usage
(count)
The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for all executors.
Shown as percent
aws.glue.driver.jvm.heap.used
(count)
The number of memory bytes used by the JVM heap for the driver.
Shown as byte
aws.glue.executor_id.jvm.heap.used
(count)
The number of memory bytes used by the JVM heap for the executor identified.
Shown as byte
aws.glue.ALL.jvm.heap.used
(count)
The number of memory bytes used by the JVM heap for all executors.
Shown as byte
aws.glue.driver.s3.file_system.read_bytes
(count)
The average number of bytes read from Amazon S3 by the driver since the previous report.
aws.glue.executor_id.s3.file_system.read_bytes
(count)
The average number of bytes read from Amazon S3 by the executor identified since the previous report.
aws.glue.ALL.s3.file_system.read_bytes
(count)
The average number of bytes read from Amazon S3 all executors since the previous report.
aws.glue.driver.s3.file_system.write_bytes
(count)
The average number of bytes written to Amazon S3 by the driver since the previous report.
aws.glue.executor_id.s3.file_system.write_bytes
(count)
The average number of bytes written to Amazon S3 by the executor identified since the previous report.
aws.glue.ALL.s3.file_system.write_bytes
(count)
The average number of bytes written to Amazon S3 by the all executors since the previous report.
aws.glue.driver.system.cpu_system_load
(count)
The average fraction of CPU system load used (scale: 0-1) by the driver.
aws.glue.executor_id.system.cpu_system_load
(count)
The average fraction of CPU system load used (scale: 0-1) by the executor identified.
aws.glue.ALL.system.cpu_system_load
(count)
The average fraction of CPU system load used (scale: 0-1) by all executors.

イベント

Amazon Glue インテグレーションには、イベントは含まれません。

サービスのチェック

Amazon Glue インテグレーションには、サービスのチェック機能は含まれません。

トラブルシューティング

ご不明な点は、Datadog のサポートチームまでお問合せください。