Apache Spark Metrics

이 페이지는 아직 영어로 제공되지 않습니다. 번역 작업 중입니다.
현재 번역 프로젝트에 대한 질문이나 피드백이 있으신 경우 언제든지 연락주시기 바랍니다.

Overview

OpenTelemetry Apache Spark metrics in a Spark dashboard

The Apache Spark receiver allows for collection of Apache Spark metrics and access to the Spark Overview dashboard. Configure the receiver according to the specifications of the latest version of the apachesparkreceiver.

For more information, see the OpenTelemetry project documentation for the Apache Spark receiver.

Setup

To collect Apache Spark metrics with OpenTelemetry for use with Datadog:

  1. Configure the Apache Spark receiver in your OpenTelemetry Collector configuration.
  2. Ensure the OpenTelemetry Collector is configured to export to Datadog.

See the Apache Spark receiver documentation for detailed configuration options and requirements.

Data collected

OTELDATADOGDESCRIPTIONFILTERTRANSFORM
spark.driver.block_manager.disk.usagespark.driver.disk_usedDisk space used by the BlockManager.× 1048576
spark.driver.block_manager.memory.usagespark.driver.memory_usedMemory usage for the driver’s BlockManager.× 1048576
spark.driver.dag_scheduler.stage.countspark.stage.countNumber of stages the DAGScheduler is either running or needs to run.
spark.executor.disk.usagespark.executor.disk_usedDisk space used by this executor for RDD storage.
spark.executor.disk.usagespark.rdd.disk_usedDisk space used by this executor for RDD storage.
spark.executor.memory.usagespark.executor.memory_usedStorage memory used by this executor.
spark.executor.memory.usagespark.rdd.memory_usedStorage memory used by this executor.
spark.job.stage.activespark.job.num_active_stagesNumber of active stages in this job.
spark.job.stage.resultspark.job.num_completed_stagesNumber of stages with a specific result in this job.job_result: completed
spark.job.stage.resultspark.job.num_failed_stagesNumber of stages with a specific result in this job.job_result: failed
spark.job.stage.resultspark.job.num_skipped_stagesNumber of stages with a specific result in this job.job_result: skipped
spark.job.task.activespark.job.num_tasks{status: running}Number of active tasks in this job.
spark.job.task.resultspark.job.num_skipped_tasksNumber of tasks with a specific result in this job.job_result: skipped
spark.job.task.resultspark.job.num_failed_tasksNumber of tasks with a specific result in this job.job_result: failed
spark.job.task.resultspark.job.num_completed_tasksNumber of tasks with a specific result in this job.job_result: completed
spark.stage.io.recordsspark.stage.input_recordsNumber of records written and read in this stage.direction: in
spark.stage.io.recordsspark.stage.output_recordsNumber of records written and read in this stage.direction: out
spark.stage.io.sizespark.stage.input_bytesAmount of data written and read at this stage.direction: in
spark.stage.io.sizespark.stage.output_bytesAmount of data written and read at this stage.direction: out
spark.stage.shuffle.io.read.sizespark.stage.shuffle_read_bytesAmount of data read in shuffle operations in this stage.
spark.stage.shuffle.io.recordsspark.stage.shuffle_read_recordsNumber of records written or read in shuffle operations in this stage.direction: in
spark.stage.shuffle.io.recordsspark.stage.shuffle_write_recordsNumber of records written or read in shuffle operations in this stage.direction: out

See OpenTelemetry Metrics Mapping for more information.

Further reading

추가 유용한 문서, 링크 및 기사:

Setting Up the OpenTelemetry CollectorDOCUMENTATION more more