概要
Apache Spark レシーバーは、Apache Spark メトリクスの収集と Spark Overview ダッシュボードへのアクセスを可能にします。apachesparkreceiver の最新バージョンの仕様に従ってレシーバーを構成してください。
詳しくは、OpenTelemetry プロジェクトドキュメントの Apache Spark レシーバーを参照してください。
セットアップ
Datadog で使用するために OpenTelemetry で Apache Spark メトリクスを収集するには:
- OpenTelemetry Collector の構成で Apache Spark レシーバーを構成します。
- OpenTelemetry Collector が Datadog にエクスポートするように構成されていることを確認します。
詳細な構成オプションと要件については、Apache Spark レシーバーのドキュメントを参照してください。
収集されたデータ
| OTEL | DESCRIPTION | FILTER |
|---|
| spark.driver.block_manager.disk.usage | Disk space used by the BlockManager. | |
| spark.driver.block_manager.memory.usage | Memory usage for the driver’s BlockManager. | |
| spark.driver.dag_scheduler.stage.count | Number of stages the DAGScheduler is either running or needs to run. | |
| spark.executor.disk.usage | Disk space used by this executor for RDD storage. | |
| spark.executor.memory.usage | Storage memory used by this executor. | |
| spark.job.stage.active | Number of active stages in this job. | |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: failed |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: skipped |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: completed |
| spark.job.task.active | Number of active tasks in this job. | |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: completed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: failed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: skipped |
| spark.stage.io.records | Number of records written and read in this stage. | direction: in |
| spark.stage.io.records | Number of records written and read in this stage. | direction: out |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: in |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: out |
| spark.stage.shuffle.io.read.size | Amount of data read in shuffle operations in this stage. | |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: in |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: out |
詳細は OpenTelemetry メトリクス マッピング を参照してください。
参考資料
