---
title: Apache Spark Metrics
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: Docs > OpenTelemetry in Datadog > Integrations > Apache Spark Metrics
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Apache Spark Metrics

## Overview{% #overview %}

{% image
   source="https://docs.dd-static.net/images/opentelemetry/collector_exporter/spark_metrics.530d0f8550d8342ab5cb0073967ff151.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/opentelemetry/collector_exporter/spark_metrics.530d0f8550d8342ab5cb0073967ff151.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="OpenTelemetry Apache Spark metrics in a Spark dashboard" /%}

The [Apache Spark receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver) collects Apache Spark metrics that populate the [Spark Overview](https://app.datadoghq.com/screen/integration/95/spark---overview) dashboard.

For more information, see the OpenTelemetry project documentation for the [Apache Spark receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver).

## Setup{% #setup %}

This example uses component identifiers from OpenTelemetry Collector Contrib v0.154.0. For other versions or distributions, use the identifiers your distribution supports. Earlier versions name this receiver `apachespark`.

The Apache Spark receiver reads the REST API that a running Spark application serves on its web UI port.

Add the following lines to your Collector configuration, and set `endpoint` to the address of your Spark application's web UI:

```yaml
receivers:
  apache_spark:
    endpoint: http://localhost:4040
```

Add `apache_spark` to the `receivers` list of the metrics pipeline in your configuration. Keep the processors already in that pipeline. The Datadog OTLP metrics intake accepts only delta metrics, and this receiver produces cumulative sums, so the pipeline needs `cumulativetodelta`. The [recommended Collector setup](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter.md) includes it.

For all configuration options, see the [Apache Spark receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver).

## Data collected{% #data-collected %}

| OTEL                                    | DESCRIPTION                                                            | FILTER                    |
| --------------------------------------- | ---------------------------------------------------------------------- | ------------------------- |
| spark.driver.block_manager.disk.usage   | Disk space used by the BlockManager.                                   |
| spark.driver.block_manager.memory.usage | Memory usage for the driver's BlockManager.                            |
| spark.driver.dag_scheduler.stage.count  | Number of stages the DAGScheduler is either running or needs to run.   |
| spark.executor.disk.usage               | Disk space used by this executor for RDD storage.                      |
| spark.executor.memory.usage             | Storage memory used by this executor.                                  |
| spark.job.stage.active                  | Number of active stages in this job.                                   |
| spark.job.stage.result                  | Number of stages with a specific result in this job.                   | `job_result`: `failed`    |
| spark.job.stage.result                  | Number of stages with a specific result in this job.                   | `job_result`: `skipped`   |
| spark.job.stage.result                  | Number of stages with a specific result in this job.                   | `job_result`: `completed` |
| spark.job.task.active                   | Number of active tasks in this job.                                    |
| spark.job.task.result                   | Number of tasks with a specific result in this job.                    | `job_result`: `completed` |
| spark.job.task.result                   | Number of tasks with a specific result in this job.                    | `job_result`: `failed`    |
| spark.job.task.result                   | Number of tasks with a specific result in this job.                    | `job_result`: `skipped`   |
| spark.stage.io.records                  | Number of records written and read in this stage.                      | `direction`: `in`         |
| spark.stage.io.records                  | Number of records written and read in this stage.                      | `direction`: `out`        |
| spark.stage.io.size                     | Amount of data written and read at this stage.                         | `direction`: `in`         |
| spark.stage.io.size                     | Amount of data written and read at this stage.                         | `direction`: `out`        |
| spark.stage.shuffle.io.read.size        | Amount of data read in shuffle operations in this stage.               |
| spark.stage.shuffle.io.records          | Number of records written or read in shuffle operations in this stage. | `direction`: `in`         |
| spark.stage.shuffle.io.records          | Number of records written or read in shuffle operations in this stage. | `direction`: `out`        |

For the full mapping between OpenTelemetry and Datadog metric names, see [OpenTelemetry Metrics Mapping](https://docs.datadoghq.com/opentelemetry/guide/metrics_mapping.md).

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Setting Up the OpenTelemetry Collector](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter.md)