Data Jobs Monitoring gives visibility into the performance and reliability of your Apache Spark and Databricks jobs.

Setup

Follow these steps to enable Data Jobs Monitoring for Databricks.

1. Configure the Datadog-Databricks integration for a Databricks workspace.
2. Install the Datadog Agent on your Databricks cluster(s) in the workspace.

Configure the Datadog-Databricks integration

1. In your Databricks workspace, click on your profile in the top right corner and go to Settings. Select Developer in the left side bar. Next to Access tokens, click Manage.

2. Click Generate new token, enter “Datadog Integration” in the Comment field, remove the default value in Lifetime (days), and click Generate. Take note of your token.

   Important:

   • Make sure you delete the default value in Lifetime (days) so that the token doesn’t expire and the integration doesn’t break.
   • Ensure the account generating the token has CAN VIEW access for the Databricks jobs and clusters you want to monitor.

   As an alternative, follow the official Databricks documentation to generate access token for a service principal.

3. In Datadog, open the Databricks integration tile.

4. On the Configure tab, click Add Databricks Workspace.

5. Enter a workspace name, your Databricks workspace URL, and the Databricks token you generated.

6. In the Select products to set up integration section, make sure the Data Jobs Monitoring product is Enabled.

7. In the Datadog Agent Setup section, chooose either

• Managed by Datadog (recommended): Datadog installs and manages the Agent with a global init script in the workspace.
• Manaully: Follow the instructions below to install and manage the init script for installing the Agent globally or on specific Databricks clusters.

Install the Datadog Agent

The Datadog Agent must be installed on Databricks clusters to monitor Databricks jobs that run on all-purpose or job clusters.

export DD_API_KEY=<YOUR API KEY>
export DD_SITE=<YOUR DATADOG SITE>
export DATABRICKS_WORKSPACE="<YOUR WORKSPACE NAME>"

Restart already-running clusters

The init script installs the Agent when clusters start.

Already-running all-purpose clusters or long-lived job clusters must be manually restarted for the init script to install the Datadog Agent.

For scheduled jobs that run on job clusters, the init script installs the Datadog Agent automatically on the next run.

Validation

In Datadog, view the Data Jobs Monitoring page to see a list of all your Databricks jobs.

Troubleshooting

If you don’t see any data in DJM after installing the product, follow those steps.

The init script installs the Datadog Agent. To make sure it is properly installed, ssh into the cluster and run the Agent status command:

sudo datadog-agent status

If the Agent is not installed, view the installation logs located in /tmp/datadog-djm-init.log.

If you need further assistance from Datadog support, add the following environment variable to the init script. This ensures that logs are sent to Datadog when a failure occurs.

export DD_DJM_ADD_LOGS_TO_FAILURE_REPORT=true

Advanced Configuration

Tag spans at runtime

You can set tags on Spark spans at runtime. These tags are applied only to spans that start after the tag is added.

// Add tag for all next Spark computations
sparkContext.setLocalProperty("spark.datadog.tags.key", "value")
spark.read.parquet(...)

To remove a runtime tag:

// Remove tag for all next Spark computations
sparkContext.setLocalProperty("spark.datadog.tags.key", null)

Aggregate cluster metrics from one-time job runs

This configuration is applicable if you want cluster resource utilization data about your jobs and create a new job and cluster for each run via the one-time run API endpoint (common when using orchestration tools outside of Databricks such as Airflow or Azure Data Factory).

If you are submitting Databricks Jobs through the one-time run API endpoint, each job run has a unique job ID. This can make it difficult to group and analyze cluster metrics for jobs that use ephemeral clusters. To aggregate cluster utilization from the same job and assess performance across multiple runs, you must set the DD_JOB_NAME variable inside the spark_env_vars of every new_cluster to the same value as your request payload’s run_name.

Here’s an example of a one-time job run request body:

{
   "run_name": "Example Job",
   "idempotency_token": "8f018174-4792-40d5-bcbc-3e6a527352c8",
   "tasks": [
      {
         "task_key": "Example Task",
         "description": "Description of task",
         "depends_on": [],
         "notebook_task": {
            "notebook_path": "/Path/to/example/task/notebook",
            "source": "WORKSPACE"
         },
         "new_cluster": {
            "num_workers": 1,
            "spark_version": "13.3.x-scala2.12",
            "node_type_id": "i3.xlarge",
            "spark_env_vars": {
               "DD_JOB_NAME": "Example Job"
            }
         }
      }
   ]
}

Set up Data Jobs Monitoring with Databricks Networking Restrictions

With Databricks Networking Restrictions, Datadog may not have access to your Databricks APIs, which is required to collect traces for Databricks job executions along with tags and other metadata.

If you are controlling Databricks API access through IP access lists, allow-listing Datadog’s specific IP addresses allows your cluster to perform all these interactions with Datadog services. Please see Databricks documentation for more details on how to manage IP access lists in Databricks.

If you are using Databricks Private Connectivity, the steps to configure the connection depend on your cloud provider.

For further assistance, contact the Datadog support team.

Further Reading

Additional helpful documentation, links, and articles:

Data Jobs MonitoringDOCUMENTATION