Overview

The BigQuery integration connects Datadog to your Google Cloud project to sync metadata, query history, and table-level metrics. Use it to monitor data freshness, detect anomalies, and trace lineage across your data stack.

Configure the BigQuery integration in Datadog

To configure the BigQuery integration in Datadog:

1. Navigate to Datadog Data Observability > Settings.
2. Click the Configure button for the BigQuery option.

You can now choose to use a service account that you’ve already connected to Datadog or set up a new one specifically for Data Observability.

Using an existing service account

Choose this option if you’ve previously connected a service account to Datadog.

1. Select the Use connected Service Account option.
2. Select the service account from the dropdown list.
3. Click Next.
4. Turn on the Enable Data Observability toggle and click Add Account. Data Observability functionality requires additional roles and API access beyond the basic BigQuery integration.

Additional roles

Apply the following roles to the service account in your GCP IAM console.

Additional APIs

Make sure the following APIs have been enabled in the project associated with the service account.

Using a new service account

1. Select the Add new Service Account option.
2. Choose one of the setup methods. Quick Start is recommended for most users since it requires no manual permissions setup.

After you’ve completed the Quick Start flow, click Next to proceed to the next page and enable the Enable Data Observability toggle before clicking Add Account.

Manual setup

In your Google Cloud console’s IAM page, create a service account with the following roles:

In Datadog, after you’ve selected the Manual setup method:

1. Copy the generated Datadog Principal to your clipboard. If this is your first time configuration GCP you’ll need to first click the Generate Principal button.
2. In the Google Cloud console, under Service Accounts, find the service account you recently created and click on it.
3. Navigate to the Principals with access tab and click the Grant access button.
4. Paste the Datadog Principal into the New principals text box.
5. Under the Assign roles section, select the Service Account Token Creator role. This will allow the Datadog Principal to impersonate the service account you just created.
6. Click the Save button.

Finally, you need to enable the following APIs for every project you want to monitor, including the project where the service account has been created.

Once the service account has been created and the necessary roles and APIs have been applied, you can return to Datadog.

1. Click Verify and save account.
2. Click Next.
3. Turn on the Enable Data Observability toggle and click Add Account.

Next steps

After you configure the integration, Datadog begins syncing your information schema and query history in the background. Initial syncs can take several hours depending on the size of your BigQuery deployment.

Further reading

Additional helpful documentation, links, and articles:

Learn about Data ObservabilityDOCUMENTATION