El receptor de Apache Spark permite recopilar métricas de Apache Spark y acceder al dashboard de Información general de Spark. Configura el receptor según las especificaciones de la última versión del apachesparkreceiver.
Para más información, consulta la documentación del proyecto de OpenTelemetry para el receptor de Apache Spark.
Configuración
Para recopilar métricas de Apache Spark con OpenTelemetry para su uso con Datadog:
- Configura el receptor de Apache Spark en tu configuración de OpenTelemetry Collector.
- Asegúrate de que el OpenTelemetry Collector está configurado para exportar a Datadog.
Consulta la documentación del receptor de Apache Spark para obtener información detallada sobre las opciones y requisitos de configuración.
Datos recopilados
| OTEL | DESCRIPTION | FILTER |
|---|
| spark.driver.block_manager.disk.usage | Disk space used by the BlockManager. | |
| spark.driver.block_manager.memory.usage | Memory usage for the driver’s BlockManager. | |
| spark.driver.dag_scheduler.stage.count | Number of stages the DAGScheduler is either running or needs to run. | |
| spark.executor.disk.usage | Disk space used by this executor for RDD storage. | |
| spark.executor.memory.usage | Storage memory used by this executor. | |
| spark.job.stage.active | Number of active stages in this job. | |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: failed |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: skipped |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: completed |
| spark.job.task.active | Number of active tasks in this job. | |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: completed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: failed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: skipped |
| spark.stage.io.records | Number of records written and read in this stage. | direction: in |
| spark.stage.io.records | Number of records written and read in this stage. | direction: out |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: in |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: out |
| spark.stage.shuffle.io.read.size | Amount of data read in shuffle operations in this stage. | |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: in |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: out |
Consulta Asignación de métricas de OpenTelemetry para obtener más información.
Referencias adicionales
Más enlaces, artículos y documentación útiles:
