Cette page n'est pas encore disponible en français, sa traduction est en cours.
Si vous avez des questions ou des retours sur notre projet de traduction actuel,
n'hésitez pas à nous contacter.
Overview
The Apache Spark receiver allows for collection of Apache Spark metrics and access to the Spark Overview dashboard. Configure the receiver according to the specifications of the latest version of the apachesparkreceiver.
For more information, see the OpenTelemetry project documentation for the Apache Spark receiver.
Setup
To collect Apache Spark metrics with OpenTelemetry for use with Datadog:
- Configure the Apache Spark receiver in your OpenTelemetry Collector configuration.
- Ensure the OpenTelemetry Collector is configured to export to Datadog.
See the Apache Spark receiver documentation for detailed configuration options and requirements.
Data collected
| OTEL | DESCRIPTION | FILTER |
|---|
| spark.driver.block_manager.disk.usage | Disk space used by the BlockManager. | |
| spark.driver.block_manager.memory.usage | Memory usage for the driver’s BlockManager. | |
| spark.driver.dag_scheduler.stage.count | Number of stages the DAGScheduler is either running or needs to run. | |
| spark.executor.disk.usage | Disk space used by this executor for RDD storage. | |
| spark.executor.memory.usage | Storage memory used by this executor. | |
| spark.job.stage.active | Number of active stages in this job. | |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: failed |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: skipped |
| spark.job.stage.result | Number of stages with a specific result in this job. | job_result: completed |
| spark.job.task.active | Number of active tasks in this job. | |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: completed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: failed |
| spark.job.task.result | Number of tasks with a specific result in this job. | job_result: skipped |
| spark.stage.io.records | Number of records written and read in this stage. | direction: in |
| spark.stage.io.records | Number of records written and read in this stage. | direction: out |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: in |
| spark.stage.io.size | Amount of data written and read at this stage. | direction: out |
| spark.stage.shuffle.io.read.size | Amount of data read in shuffle operations in this stage. | |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: in |
| spark.stage.shuffle.io.records | Number of records written or read in shuffle operations in this stage. | direction: out |
For the full mapping between OpenTelemetry and Datadog metric names, see OpenTelemetry Metrics Mapping.
Further reading
Documentation, liens et articles supplémentaires utiles:
