이 페이지는 아직 한국어로 제공되지 않습니다. 번역 작업 중입니다.
현재 번역 프로젝트에 대한 질문이나 피드백이 있으신 경우
언제든지 연락주시기 바랍니다.Overview
The Apache Spark receiver allows for collection of Apache Spark metrics and access to the Spark Overview dashboard. Configure the receiver according to the specifications of the latest version of the apachesparkreceiver.
For more information, see the OpenTelemetry project documentation for the Apache Spark receiver.
Setup
To collect Apache Spark metrics with OpenTelemetry for use with Datadog:
- Configure the Apache Spark receiver in your OpenTelemetry Collector configuration.
- Ensure the OpenTelemetry Collector is configured to export to Datadog.
See the Apache Spark receiver documentation for detailed configuration options and requirements.
Data collected
| OTEL | DESCRIPTION | FILTER |
|---|
| spark.driver.block_manager.disk.usage | Disk space used by the BlockManager. | |
| spark.driver.block_manager.memory.usage | Memory usage for the driver’s BlockManager. | |
| spark.driver.dag_scheduler.stage.count | Number of stages the DAGScheduler is either running or needs to run. | |
| spark.executor.disk.usage | Disk space used by this executor for RDD storage. | |
| spark.executor.memory.usage | Storage memory used by this executor. | |
| spark.job.stage.active | Number of active stages in this job. | |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: failed |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: skipped |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: completed |
| spark.job.task.active | Number of active tasks in this job. | |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: completed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: failed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: skipped |
| spark.stage.io.records | Number of records written and read in this stage. | direction: in |
| spark.stage.io.records | Number of records written and read in this stage. | direction: out |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: in |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: out |
| spark.stage.shuffle.io.read.size | Amount of data read in shuffle operations in this stage. | |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: in |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: out |
For the full mapping between OpenTelemetry and Datadog metric names, see OpenTelemetry Metrics Mapping.
Further reading
