---
title: Databricks
description: >-
  Connect Databricks to Datadog Data Observability to monitor data quality,
  track usage, and detect issues.
breadcrumbs: >-
  Docs > Data Observability Overview > Quality Monitoring > Warehouse
  Integrations > Databricks
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Databricks

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

The Databricks integration connects Datadog to your Databricks workspace to sync metadata and table-level metrics. Use it to monitor data freshness, detect anomalies, and trace lineage across your data stack.

**Note**: The instructions below are for Quality Monitoring. For Jobs Monitoring, see [Enable Data Observability: Jobs Monitoring for Databricks](https://docs.datadoghq.com/data_observability/jobs_monitoring/databricks.md).

## Prerequisites{% #prerequisites %}

If your Databricks workspace restricts network access by IP, add the Datadog webhook IPs to your allowlist. For the list of IPs, see the `webhooks` section of IP ranges list.

## Set up your account in Databricks{% #set-up-your-account-in-databricks %}

### Step 1 - Connect the Databricks integration tile{% #step-1---connect-the-databricks-integration-tile %}

1. Complete the installation instructions in the [Databricks integration documentation](https://docs.datadoghq.com/integrations/databricks.md) using Datadog's integration tile. Take note of the service principal's application ID and save it somewhere safe, because it will be referenced later.

**Note**: Workspace Admin permissions are not required for Quality Monitoring.

1. When configuring the integration, turn on the Data Observability toggle.

1. Click Save Databricks Workspace.

### Step 2 - Grant access{% #step-2---grant-access %}

In Databricks, open the SQL Editor to run the following commands. Use the service principal's application (client) ID, not its display name, wherever `<application_id>` appears.

First, grant access to system schemas for lineage:

```sql
GRANT USE CATALOG ON CATALOG system TO `<application_id>`;
GRANT USE SCHEMA ON CATALOG system TO `<application_id>`;
GRANT SELECT ON CATALOG system TO `<application_id>`;
```

These grants cover the whole `system` catalog, which includes the [query history system table](https://docs.databricks.com/aws/en/admin/system-tables/query-history) (`system.query.history`). Datadog reads query history from this table to build lineage between your tables and to give you visibility into the queries running against them. To read the query text in that table, the service principal also needs the group membership described in Step 3.

Then, grant read-only access to the scope of data you want to monitor:

{% tab title="Full catalog access" %}
Use the full catalog access option for simpler setup. It automatically includes future tables without needing to update permissions.

```sql
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<application_id>`;
GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<application_id>`;
GRANT SELECT ON CATALOG <catalog_name> TO `<application_id>`;
```

{% /tab %}

{% tab title="Specific tables" %}
Use the specific tables option for least-privilege access or if you only need to monitor a subset of your data. You must update permissions when adding new tables.

```sql
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<application_id>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<application_id>`;
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<application_id>`;
```

{% /tab %}

These permissions are needed for the following reasons:

- `GRANT USE CATALOG` is required to navigate into the catalog and discover schemas.
- `GRANT USE SCHEMA` is required to enumerate tables and monitor schema-level health.
- `GRANT SELECT` is required for data quality monitoring, such as custom SQL or distribution checks.

### Step 3 - Grant access to query text{% #step-3---grant-access-to-query-text %}

Databricks masks SQL query text for any principal that is not an account administrator or a member of the `databricks_pii_access` account-level group. For a masked principal, query text is returned as `<Redacted>` in the `statement_text` column of `system.query.history`, the [Query History API](https://docs.databricks.com/api/workspace/queryhistory/list), the [List Queries API](https://docs.databricks.com/api/workspace/queries/list), and audit log events that capture SQL statement text.

Add the service principal to `databricks_pii_access` to use the following capabilities, which read query text:

- **Data Lineage**: Supplemented by parsing query text from Databricks query history.
- **Databricks serverless job monitoring**: Monitoring jobs that run on [serverless compute](https://docs.databricks.com/aws/en/compute/serverless/), where no Datadog Agent runs on the cluster.
- **SQL warehouse & query monitoring**: Visibility into the queries running on your SQL warehouses, and optimization recommendations generated by Datadog.

Table-level metrics such as freshness, row count, and column statistics read table data and metadata rather than query text, so they work without this membership.

Group membership is required in addition to the `system` catalog grants in Step 2. A principal that is in the group but lacks `SELECT` on `CATALOG system` still cannot read query history.

To create the group and add the service principal:

1. The `databricks_pii_access` group does not exist in a Databricks account by default, and workspace administrators are not members of it automatically. Create it with the exact name `databricks_pii_access`, which is case-sensitive.
   - If you do not manage groups with SCIM or an external identity provider, go to Account Console > User Management > Groups > Add Group.
   - If you manage groups with SCIM or an external identity provider, create the group there instead.
1. Add the service principal from Step 1 to the group.

For more details, see the Databricks documentation on [managing account-level groups](https://docs.databricks.com/aws/en/admin/users-groups/groups).

## Next steps{% #next-steps %}

After you configure the integration, Datadog begins syncing your metadata and column-level lineage in the background. Initial syncs can take several hours depending on the size of your Databricks deployment.

After the initial sync completes, create a [Data Observability monitor](https://docs.datadoghq.com/monitors/types/data_observability.md) to start alerting on freshness, row count, column-level metrics, and custom SQL metrics.

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Data Observability Overview](https://docs.datadoghq.com/data_observability.md)
- [Data Observability Monitors](https://docs.datadoghq.com/monitors/types/data_observability.md)