Databricks workspaces deployed using Private Link Connectivity are isolated from the public internet, which prevents Datadog from accessing the Databricks APIs directly. To monitor these workspaces, deploy a Private Action Runner (PAR) in your environment. The runner polls Datadog for requests, executes them against the Databricks API locally, and sends the results back through your existing Datadog VPC Endpoint.

Architecture

Private Link monitoring is built on the Private Action Runner’s polling architecture. Datadog provides a custom version of the Private Action Runner image that includes a script for querying the Databricks API.

The request flow works as follows:

1. The Private Action Runner reaches out to the Datadog backend (either over the public internet or using Private Link) to query for pending requests. If a request is found, details are returned to the runner.
2. Databricks credentials (client ID and secret) are retrieved from secret storage. A token is generated for the session by calling the Databricks API from the runner using those credentials.
3. The runner executes the fetched query against the Databricks API. The results are returned to the runner.
4. The runner forwards the results back to the Datadog backend for processing.

All authentication to Datadog is handled by the underlying Private Action Runner. A locally configured allowlist is set on the runner during installation to restrict the actions that can be performed to only those that are necessary. You can also configure access controls within Datadog to manage which users and teams can use the runner. See Manage access to Private Action Runners for more information.

Installation

Step 1: Datadog prerequisites

1. Establish connectivity to Datadog API endpoints from within the VPC, using one of the following methods:
   • Egress to the public internet (through NAT gateway or other)
   • Egress to Datadog through Private Link VPC Endpoints or services:
     • AWS: Set up VPC Endpoints (available in US1, AP1, and AP2)
     • Azure: Private Link service setup (available in US3 only). This requires approval in the Datadog Azure account. Reach out to Datadog Support as specified in the instructions.
     • GCP: Set up Private Service Connect Endpoints (available in US5 and EU1)
2. Create a Datadog service account:
   1. Go to the Service Accounts page.
   2. Click New Service Account.
   3. Enter a name and email address for your service account.
   4. Under Assign Roles, select the Datadog Standard Role. Alternatively, select a custom role that has the Connections Resolve permission.
   5. Click Create Service Account.
3. Ensure Remote Configuration is enabled for your Datadog account.

Step 2: Databricks prerequisites

1. Create a Databricks service principal (Databricks documentation):
   1. As a workspace admin, log in to the Databricks workspace.
   2. Click your username in the top bar and select Settings.
   3. Click the Identity and access tab.
   4. Next to Service principals, click Manage.
   5. Click Add service principal, then click Add new and enter a name.
      • Alternatively, to reuse a service principal from another workspace, select an existing service principal and skip the next step.
      Datadog does not support Microsoft Entra ID managed Service Principals over Private Link.
   6. Click Add.
2. Generate secrets for the Databricks service principal (Databricks documentation):
   1. Select the service principal created above.
   2. Click the Secrets tab, then click Generate secret.
   3. Set the secret’s lifetime in days (maximum 730 days).
   4. Click Generate.
   5. Make the Databricks credentials available to the runner using one of the following methods:
      • Cloud secret storage (AWS Secrets Manager, AWS Parameter Store, or Azure Key Vault): Store the credentials as a JSON blob:

        {"client_id": "<CLIENT_ID>", "client_secret": "<CLIENT_SECRET>"}
        

        Make note of the secret path (AWS ARN or AKV URL). You enter this in the integration tile or set it as the DATABRICKS_SECRET_PATH environment variable on the runner container or pod.
      • Environment variables: Set DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET directly on the runner container or pod. When using this method, leave the Secret Path field blank in the integration tile.
3. Provision workspace access and entitlements for the service principal. See Permissions for full details.
   1. Grant the service principal Workspace Admin privileges. This allows Datadog to manage init script installations and updates automatically. Alternatively, assign more granular permissions for monitoring jobs, clusters, and queries.
      1. Click the Identity and access tab.
      2. Next to Groups, click Manage.
      3. Select the admins group.
      4. Click Add members and select the service principal added above.
      5. Click Add.
   2. To access Databricks cost data or monitor serverless jobs, grant the service principal CAN USE on the SQL Warehouse and read access to system tables in Unity Catalog. See Permissions for details.

Step 3: Set up the Private Action Runner

Set up your Private Action Runner using one of the following options.

Note: Other mirrors are available if gcr.io is inaccessible:

• docker.io/datadog/dd-data-observability-par
• public.ecr.aws/datadog/dd-data-observability-par
• datadoghq.azurecr.io/dd-data-observability-par
• asia-docker.pkg.dev/datadoghq/asia.gcr.io/dd-data-observability-par

Step 4: Configure Datadog

1. Attach a connection to the Private Action Runner:
   1. Go to the Private Action Runners page.
   2. Select the installed runner.
   3. Select Finish Setup.
   4. Select Script.
   5. Set Path to File to /etc/data-observability/config/script.yaml.
   6. Select Next, Confirm Access.
   7. Add the service account created in Step 1 with Resolver access. This step is required — the runner uses this service account to authenticate Datadog requests, and the integration cannot function without it.
   8. Select Create and Test. Verify the test results do not return an error.
2. Set up the Databricks integration:
   1. Go to the Databricks integration tile.
   2. Under Credentials, select Private Action Runner.
      1. Under Connection, select the connection created in the previous step.
      2. Under Service Account, select the service account created in Step 1.
      3. Under Secret Path, enter the secret path from Step 2. Leave this field blank if you set credentials using the DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET environment variables, or if you set the path using the DATABRICKS_SECRET_PATH environment variable on the runner.
   3. Complete the remainder of the integration setup as described in Configure the Datadog-Databricks integration:
      1. Enter the Workspace Name, Workspace URL, and System Tables SQL Warehouse ID.
      2. Enable the products you are interested in. Note: Model Serving behind Private Link is not supported at this time.
      3. If you enabled Jobs Monitoring, deploy the Datadog Agent init script to monitor jobs and clusters on classic compute. Follow the instructions in Install the Datadog Agent to set up the init script, then restart any running clusters for the changes to take effect.
   4. Click Save Databricks Workspace.
   5. Verify there are no errors on save.

Validation

After completing the setup, jobs and cluster information should appear in Data Observability: Jobs Monitoring shortly.

Further Reading

Additional helpful documentation, links, and articles:

Data Observability: Jobs Monitoring for DatabricksDOCUMENTATION Private Action RunnersDOCUMENTATION