Spark

Spark

Agent Check Agentチェック

Supported OS Linux Mac OS Windows

Spark のグラフ

概要

このチェックは、Datadog Agent を通じて Spark を監視します。以下の Spark メトリクスを収集します。

  • ドライバーとエグゼキューター: RDD ブロック、使用メモリ量、使用ディスク容量、処理時間など
  • RDD: パーティション数、使用メモリ量、使用ディスク容量。
  • タスク: アクティブなタスク数、スキップされたタスク数、失敗したタスク数、合計タスク数。
  • ジョブの状態: アクティブなジョブ数、完了したジョブ数、スキップされたジョブ数、失敗したジョブ数。

セットアップ

インストール

Spark チェックは Datadog Agent パッケージに含まれています。Mesos マスター(Mesos の Spark)、YARN ResourceManager(YARN の Spark)、Spark マスター(Spark Standalone)に追加でインストールする必要はありません。

コンフィギュレーション

ホスト

ホストで実行中の Agent に対してこのチェックを構成するには:

  1. Agent のコンフィギュレーションディレクトリのルートにある conf.d/ フォルダーの spark.d/conf.yaml ファイルを編集します。以下のパラメーターは、更新が必要な場合があります。使用可能なすべてのコンフィギュレーションオプションの詳細については、サンプル spark.d/conf.yaml を参照してください。

    init_config:
    
    instances:
      - spark_url: http://localhost:8080 # Spark master web UI
        #   spark_url: http://<Mesos_master>:5050 # Mesos master web UI
        #   spark_url: http://<YARN_ResourceManager_address>:8088 # YARN ResourceManager address
    
        spark_cluster_mode: spark_yarn_mode # default
        #   spark_cluster_mode: spark_mesos_mode
        #   spark_cluster_mode: spark_yarn_mode
        #   spark_cluster_mode: spark_driver_mode
    
        # required; adds a tag 'cluster_name:<CLUSTER_NAME>' to all metrics
        cluster_name: "<CLUSTER_NAME>"
        # spark_pre_20_mode: true   # if you use Standalone Spark < v2.0
        # spark_proxy_enabled: true # if you have enabled the spark UI proxy
    
  2. Agent を再起動します

コンテナ化

コンテナ環境の場合は、オートディスカバリーのインテグレーションテンプレートのガイドを参照して、次のパラメーターを適用してください。

パラメーター
<インテグレーション名> spark
<初期コンフィギュレーション> 空白または {}
<インスタンスコンフィギュレーション> {"spark_url": "%%host%%:8080", "cluster_name":"<CLUSTER_NAME>"}

ログの収集

  1. Datadog Agent で、ログの収集はデフォルトで無効になっています。以下のように、datadog.yaml ファイルでこれを有効にします。

     logs_enabled: true
    
  2. spark.d/conf.yaml ファイルのコメントを解除して、ログコンフィギュレーションブロックを編集します。環境に基づいて、 typepathservice パラメーターの値を変更してください。使用可能なすべての構成オプションの詳細については、サンプル spark.d/conf.yaml を参照してください。

     logs:
       - type: file
         path: <LOG_FILE_PATH>
         source: spark
         service: <SERVICE_NAME>
         # To handle multi line that starts with yyyy-mm-dd use the following pattern
         # log_processing_rules:
         #   - type: multi_line
         #     pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])
         #     name: new_log_start_with_date
    
  3. Agent を再起動します

Docker 環境でログを収集する Agent を構成する追加の情報に関しては、Datadog ドキュメントを参照してください。

検証

Agent の status サブコマンドを実行し、Checks セクションで spark を探します。

収集データ

メトリクス

spark.job.count
(count)
Number of jobs
Shown as task
spark.job.num_tasks
(count)
Number of tasks in the application
Shown as task
spark.job.num_active_tasks
(count)
Number of active tasks in the application
Shown as task
spark.job.num_skipped_tasks
(count)
Number of skipped tasks in the application
Shown as task
spark.job.num_failed_tasks
(count)
Number of failed tasks in the application
Shown as task
spark.job.num_completed_tasks
(count)
Number of completed tasks in the application
Shown as task
spark.job.num_active_stages
(count)
Number of active stages in the application
Shown as stage
spark.job.num_completed_stages
(count)
Number of completed stages in the application
Shown as stage
spark.job.num_skipped_stages
(count)
Number of skipped stages in the application
Shown as stage
spark.job.num_failed_stages
(count)
Number of failed stages in the application
Shown as stage
spark.stage.count
(count)
Number of stages
Shown as task
spark.stage.num_active_tasks
(count)
Number of active tasks in the application's stages
Shown as task
spark.stage.num_complete_tasks
(count)
Number of complete tasks in the application's stages
Shown as task
spark.stage.num_failed_tasks
(count)
Number of failed tasks in the application's stages
Shown as task
spark.stage.executor_run_time
(count)
Time spent by the executor in the application's stages
Shown as millisecond
spark.stage.input_bytes
(count)
Input bytes in the application's stages
Shown as byte
spark.stage.input_records
(count)
Input records in the application's stages
Shown as record
spark.stage.output_bytes
(count)
Output bytes in the application's stages
Shown as byte
spark.stage.output_records
(count)
Output records in the application's stages
Shown as record
spark.stage.shuffle_read_bytes
(count)
Number of bytes read during a shuffle in the application's stages
Shown as byte
spark.stage.shuffle_read_records
(count)
Number of records read during a shuffle in the application's stages
Shown as record
spark.stage.shuffle_write_bytes
(count)
Number of shuffled bytes in the application's stages
Shown as byte
spark.stage.shuffle_write_records
(count)
Number of shuffled records in the application's stages
Shown as record
spark.stage.memory_bytes_spilled
(count)
Number of bytes spilled to disk in the application's stages
Shown as byte
spark.stage.disk_bytes_spilled
(count)
Max size on disk of the spilled bytes in the application's stages
Shown as byte
spark.driver.rdd_blocks
(count)
Number of RDD blocks in the driver
Shown as block
spark.driver.memory_used
(count)
Amount of memory used in the driver
Shown as byte
spark.driver.disk_used
(count)
Amount of disk used in the driver
Shown as byte
spark.driver.active_tasks
(count)
Number of active tasks in the driver
Shown as task
spark.driver.failed_tasks
(count)
Number of failed tasks in the driver
Shown as task
spark.driver.completed_tasks
(count)
Number of completed tasks in the driver
Shown as task
spark.driver.total_tasks
(count)
Number of total tasks in the driver
Shown as task
spark.driver.total_duration
(count)
Time spent in the driver
Shown as millisecond
spark.driver.total_input_bytes
(count)
Number of input bytes in the driver
Shown as byte
spark.driver.total_shuffle_read
(count)
Number of bytes read during a shuffle in the driver
Shown as byte
spark.driver.total_shuffle_write
(count)
Number of shuffled bytes in the driver
Shown as byte
spark.driver.max_memory
(count)
Maximum memory used in the driver
Shown as byte
spark.executor.count
(count)
Number of executors
Shown as task
spark.executor.rdd_blocks
(count)
Number of persisted RDD blocks in the application's executors
Shown as block
spark.executor.memory_used
(count)
Amount of memory used for cached RDDs in the application's executors
Shown as byte
spark.executor.max_memory
(count)
Max memory across all executors working for a particular application
Shown as byte
spark.executor.disk_used
(count)
Amount of disk space used by persisted RDDs in the application's executors
Shown as byte
spark.executor.active_tasks
(count)
Number of active tasks in the application's executors
Shown as task
spark.executor.failed_tasks
(count)
Number of failed tasks in the application's executors
Shown as task
spark.executor.completed_tasks
(count)
Number of completed tasks in the application's executors
Shown as task
spark.executor.total_tasks
(count)
Total number of tasks in the application's executors
Shown as task
spark.executor.total_duration
(count)
Time spent by the application's executors executing tasks
Shown as millisecond
spark.executor.total_input_bytes
(count)
Total number of input bytes in the application's executors
Shown as byte
spark.executor.total_shuffle_read
(count)
Total number of bytes read during a shuffle in the application's executors
Shown as byte
spark.executor.total_shuffle_write
(count)
Total number of shuffled bytes in the application's executors
Shown as byte
spark.executor_memory
(count)
Maximum memory available for caching RDD blocks in the application's executors
Shown as byte
spark.executor.id.rdd_blocks
(count)
Number of persisted RDD blocks in this executor
Shown as block
spark.executor.id.memory_used
(count)
Amount of memory used for cached RDDs in this executor.
Shown as byte
spark.executor.id.max_memory
(count)
Total amount of memory available for storage for this executor
Shown as byte
spark.executor.id.disk_used
(count)
Amount of disk space used by persisted RDDs in this executor
Shown as byte
spark.executor.id.active_tasks
(count)
Number of active tasks in this executor
Shown as task
spark.executor.id.failed_tasks
(count)
Number of failed tasks in this executor
Shown as task
spark.executor.id.completed_tasks
(count)
Number of completed tasks in this executor
Shown as task
spark.executor.id.total_tasks
(count)
Total number of tasks in this executor
Shown as task
spark.executor.id.total_duration
(count)
Time spent by the executor executing tasks
Shown as millisecond
spark.executor.id.total_input_bytes
(count)
Total number of input bytes in the executor
Shown as byte
spark.executor.id.total_shuffle_read
(count)
Total number of bytes read during a shuffle in the executor
Shown as byte
spark.executor.id.total_shuffle_write
(count)
Total number of shuffled bytes in the executor
Shown as byte
spark.rdd.count
(count)
Number of RDDs
spark.rdd.num_partitions
(count)
Number of persisted RDD partitions in the application
spark.rdd.num_cached_partitions
(count)
Number of in-memory cached RDD partitions in the application
spark.rdd.memory_used
(count)
Amount of memory used in the application's persisted RDDs
Shown as byte
spark.rdd.disk_used
(count)
Amount of disk space used by persisted RDDs in the application
Shown as byte
spark.streaming.statistics.avg_input_rate
(gauge)
Average streaming input data rate
Shown as byte
spark.streaming.statistics.avg_processing_time
(gauge)
Average application's streaming batch processing time
Shown as millisecond
spark.streaming.statistics.avg_scheduling_delay
(gauge)
Average application's streaming batch scheduling delay
Shown as millisecond
spark.streaming.statistics.avg_total_delay
(gauge)
Average application's streaming batch total delay
Shown as millisecond
spark.streaming.statistics.batch_duration
(gauge)
Application's streaming batch duration
Shown as millisecond
spark.streaming.statistics.num_active_batches
(gauge)
Number of active streaming batches
Shown as job
spark.streaming.statistics.num_active_receivers
(gauge)
Number of active streaming receivers
Shown as object
spark.streaming.statistics.num_inactive_receivers
(gauge)
Number of inactive streaming receivers
Shown as object
spark.streaming.statistics.num_processed_records
(count)
Number of processed streaming records
Shown as record
spark.streaming.statistics.num_received_records
(count)
Number of received streaming records
Shown as record
spark.streaming.statistics.num_receivers
(gauge)
Number of streaming application's receivers
Shown as object
spark.streaming.statistics.num_retained_completed_batches
(count)
Number of retained completed application's streaming batches
Shown as job
spark.streaming.statistics.num_total_completed_batches
(count)
Total number of completed application's streaming batches
Shown as job
spark.structured_streaming.input_rate
(gauge)
Average streaming input data rate
Shown as record
spark.structured_streaming.latency
(gauge)
Average latency for the structured streaming application.
Shown as millisecond
spark.structured_streaming.processing_rate
(gauge)
Number of received streaming records per second
Shown as row
spark.structured_streaming.rows_count
(gauge)
Count of rows.
Shown as row
spark.structured_streaming.used_bytes
(gauge)
Number of bytes used in memory.
Shown as byte

イベント

Spark チェックには、イベントは含まれません。

サービスのチェック

spark.resource_manager.can_connect
Agent が Spark インスタンスの ResourceManager に接続できない場合は、CRITICAL を返します。それ以外の場合は、OK を返します。
Statuses: ok, クリティカル

spark.application_master.can_connect
Agent が Spark インスタンスの ApplicationMaster に接続できない場合は、CRITICAL を返します。それ以外の場合は、OK を返します。
Statuses: ok, クリティカル

トラブルシューティング

AWS EMR 上の Spark

AWS EMR 上の Spark のメトリクスを受信するには、ブートストラップアクションを使用して Datadog Agent をインストールします。

Agent v5 の場合は、各 EMR ノードに正しい値が指定された /etc/dd-agent/conf.d/spark.yaml 構成ファイルを作成します。

Agent v6/7 の場合は、各 EMR ノードに正しい値が指定された /etc/datadog-agent/conf.d/spark.d/conf.yaml 構成ファイルを作成します。

その他の参考資料

お役に立つドキュメント、リンクや記事: