--- title: Google BigQuery description: >- BigQuery is Google's fully managed, petabyte scale, low cost enterprise data warehouse for analytics. breadcrumbs: Docs > Integrations > Google BigQuery --- # Google BigQuery ## Overview{% #overview %} Datadog's integration with Google BigQuery provides monitoring for your data analytics workloads. By connecting BigQuery with Datadog (using the Google Cloud Platform integration), you gain real-time insights into query performance, resource consumption, and cost drivers. This integration enables you to track critical metrics such as job completion times, slot utilization, and processed bytes, allowing you to identify high latencies in BigQuery jobs execution. Additionally, [Data Observability](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/bigquery/) provides column-level lineage freshness monitoring, and anomaly detection across your BigQuery tables. ## Setup{% #setup %} ### Installation{% #installation %} If you haven't already, set up the [Google Cloud Platform integration first](https://docs.datadoghq.com/integrations/google-cloud-platform/). There are no other installation steps required. ### Resource Monitoring and Optimization{% #resource-monitoring-and-optimization %} Resource Monitoring and Optimization provides granular visibility into your BigQuery environments, including query-level performance metrics and slot usage. See the [Google Cloud integration documentation](https://docs.datadoghq.com/integrations/google-cloud-platform/#expanded-bigquery-monitoring) for detailed setup instructions. ### Data Observability{% #data-observability %} To enable lineage, freshness monitoring, and data quality for BigQuery tables: 1. On the **Configure** tab, select an existing connected service account or add a new one. 1. Enable **Data Observability** for the account. The service account requires the following BigQuery roles: - **BigQuery Data Viewer**: For dataset visibility - **BigQuery Resource Viewer**: For job visibility - **BigQuery Job User**: For running quality queries See the [BigQuery Data Observability documentation](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/bigquery/) for full setup instructions. ### Log collection{% #log-collection %} Google BigQuery logs are collected with Google Cloud Logging and sent to a Dataflow job through a Cloud Pub/Sub topic. If you haven't already, [set up logging with the Datadog Dataflow template](https://docs.datadoghq.com/integrations/google-cloud-platform/#log-collection). Once this is done, export your Google BigQuery logs from Google Cloud Logging to the Pub/Sub topic: 1. Go to the [Google Cloud Logging page](https://console.cloud.google.com/logs/viewer) and filter the Google BigQuery logs. 1. Click **Create Export** and name the sink. 1. Choose "Cloud Pub/Sub" as the destination and select the Pub/Sub topic that was created for that purpose. **Note**: The Pub/Sub topic can be located in a different project. 1. Click **Create** and wait for the confirmation message to show up. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **gcp.bigquery.job.num\_in\_flight**(gauge) | Number of in flight jobs.*Shown as job* | | **gcp.bigquery.query.biengine\_fallback\_count**(count) | The reasons that queries failed BI Engine execution.*Shown as query* | | **gcp.bigquery.query.column\_metadata\_index\_staleness.avg**(gauge) | The average distribution of staleness in milliseconds of the column metadata index for queries that successfully used the column metadata index in the last sampling interval.*Shown as millisecond* | | **gcp.bigquery.query.column\_metadata\_index\_staleness.samplecount**(gauge) | The sample count for distribution of staleness in milliseconds of the column metadata index for queries that successfully used the column metadata index in the last sampling interval.*Shown as millisecond* | | **gcp.bigquery.query.column\_metadata\_index\_staleness.sumsqdev**(gauge) | The sum of squared deviation for distribution of staleness in milliseconds of the column metadata index for queries that successfully used the column metadata index in the last sampling interval.*Shown as millisecond* | | **gcp.bigquery.query.count**(gauge) | Queries in flight.*Shown as query* | | **gcp.bigquery.query.execution\_count**(count) | Number of queries executed.*Shown as query* | | **gcp.bigquery.query.execution\_times.avg**(gauge) | Average of query execution times.*Shown as second* | | **gcp.bigquery.query.execution\_times.samplecount**(count) | Sample Count of query execution times.*Shown as second* | | **gcp.bigquery.query.execution\_times.sumsqdev**(gauge) | Sum of Squared Deviation for query execution times.*Shown as second* | | **gcp.bigquery.query.scanned\_bytes**(rate) | Number of scanned bytes. Note: this metric is available with a six-hour delay.*Shown as byte* | | **gcp.bigquery.query.scanned\_bytes\_billed**(rate) | Number of scanned bytes billed. Note: this metric is available with a six-hour delay.*Shown as byte* | | **gcp.bigquery.query.statement\_scanned\_bytes**(count) | Scanned bytes broken down by statement type. Note: this metric is available with a six-hour delay.*Shown as byte* | | **gcp.bigquery.query.statement\_scanned\_bytes\_billed**(count) | Scanned bytes billed broken down by statement type. Note: this metric is available with a six-hour delay.*Shown as byte* | | **gcp.bigquery.slots.allocated**(gauge) | Number of BigQuery slots currently allocated for project, slot allocation can be broken down based on reservation and job type. | | **gcp.bigquery.slots.allocated\_for\_project**(gauge) | Number of BigQuery slots currently allocated for the project. | | **gcp.bigquery.slots.allocated\_for\_project\_and\_job\_type**(gauge) | Number of BigQuery slots currently allocated for the project and job type. | | **gcp.bigquery.slots.allocated\_for\_reservation**(gauge) | Number of BigQuery slots currently allocated for project in the reservation. | | **gcp.bigquery.slots.assigned**(gauge) | The number of slots assigned to the given project or organization. | | **gcp.bigquery.slots.capacity\_committed**(gauge) | The total slot capacity commitments purchased through this administrator project or organization. | | **gcp.bigquery.slots.max\_assigned**(gauge) | The maximum number of slots assigned to the given project or organization. | | **gcp.bigquery.slots.total\_allocated\_for\_reservation**(gauge) | Number of BigQuery slots currently allocated across all projects in the reservation. | | **gcp.bigquery.storage.insertall\_inserted\_bytes**(count) | The number of bytes uploaded by the project using the InsertAll streaming API.*Shown as byte* | | **gcp.bigquery.storage.insertall\_inserted\_rows**(count) | The number of rows uploaded by the project using the InsertAll streaming API.*Shown as row* | | **gcp.bigquery.storage.stored\_bytes**(gauge) | Number of bytes stored. Note: this metric is available with a three-hour delay.*Shown as byte* | | **gcp.bigquery.storage.table\_count**(gauge) | Number of tables. Note: this metric is available with a three-hour delay.*Shown as table* | | **gcp.bigquery.storage.uploaded\_bytes**(count) | Number of uploaded bytes. Note: this metric is available with a six-hour delay.*Shown as byte* | | **gcp.bigquery.storage.uploaded\_bytes\_billed**(count) | Number of uploaded bytes billed. Note: this metric is available with a six-hour delay.*Shown as byte* | | **gcp.bigquery.storage.uploaded\_row\_count**(count) | Number of uploaded rows. Note: this metric is available with a six-hour delay.*Shown as row* | ### Events{% #events %} The Google BigQuery integration does not include any events. ### Service Checks{% #service-checks %} The Google BigQuery integration does not include any service checks. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).