---
title: Apache Spark Metrics
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: Docs > OpenTelemetry in Datadog > Integrations > Apache Spark Metrics
---

# Apache Spark Metrics

## Overview{% #overview %}

{% image
   source="https://docs.dd-static.net/images/opentelemetry/collector_exporter/spark_metrics.530d0f8550d8342ab5cb0073967ff151.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/opentelemetry/collector_exporter/spark_metrics.530d0f8550d8342ab5cb0073967ff151.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="OpenTelemetry Apache Spark metrics in a Spark dashboard" /%}

The [Apache Spark receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver) allows for collection of Apache Spark metrics and access to the [Spark Overview](https://app.datadoghq.com/screen/integration/95/spark---overview) dashboard. Configure the receiver according to the specifications of the latest version of the `apachesparkreceiver`.

For more information, see the OpenTelemetry project documentation for the [Apache Spark receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver).

## Setup{% #setup %}

To collect Apache Spark metrics with OpenTelemetry for use with Datadog:

1. Configure the [Apache Spark receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver) in your OpenTelemetry Collector configuration.
1. Ensure the OpenTelemetry Collector is [configured to export to Datadog](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter.md).

See the [Apache Spark receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver) for detailed configuration options and requirements.

## Data collected{% #data-collected %}

| OTEL                                    | DESCRIPTION                                                            | FILTER                    |
| --------------------------------------- | ---------------------------------------------------------------------- | ------------------------- |
| spark.driver.block_manager.disk.usage   | Disk space used by the BlockManager.                                   |
| spark.driver.block_manager.memory.usage | Memory usage for the driver's BlockManager.                            |
| spark.driver.dag_scheduler.stage.count  | Number of stages the DAGScheduler is either running or needs to run.   |
| spark.executor.disk.usage               | Disk space used by this executor for RDD storage.                      |
| spark.executor.memory.usage             | Storage memory used by this executor.                                  |
| spark.job.stage.active                  | Number of active stages in this job.                                   |
| spark.job.stage.result                  | Number of stages with a specific result in this job.                   | `job_result`: `failed`    |
| spark.job.stage.result                  | Number of stages with a specific result in this job.                   | `job_result`: `skipped`   |
| spark.job.stage.result                  | Number of stages with a specific result in this job.                   | `job_result`: `completed` |
| spark.job.task.active                   | Number of active tasks in this job.                                    |
| spark.job.task.result                   | Number of tasks with a specific result in this job.                    | `job_result`: `completed` |
| spark.job.task.result                   | Number of tasks with a specific result in this job.                    | `job_result`: `failed`    |
| spark.job.task.result                   | Number of tasks with a specific result in this job.                    | `job_result`: `skipped`   |
| spark.stage.io.records                  | Number of records written and read in this stage.                      | `direction`: `in`         |
| spark.stage.io.records                  | Number of records written and read in this stage.                      | `direction`: `out`        |
| spark.stage.io.size                     | Amount of data written and read at this stage.                         | `direction`: `in`         |
| spark.stage.io.size                     | Amount of data written and read at this stage.                         | `direction`: `out`        |
| spark.stage.shuffle.io.read.size        | Amount of data read in shuffle operations in this stage.               |
| spark.stage.shuffle.io.records          | Number of records written or read in shuffle operations in this stage. | `direction`: `in`         |
| spark.stage.shuffle.io.records          | Number of records written or read in shuffle operations in this stage. | `direction`: `out`        |

For the full mapping between OpenTelemetry and Datadog metric names, see [OpenTelemetry Metrics Mapping](https://docs.datadoghq.com/opentelemetry/guide/metrics_mapping.md).

## Further reading{% #further-reading %}

- [Setting Up the OpenTelemetry Collector](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter.md)