Overview
BigQuery is Google’s fully managed, petabyte scale, low cost enterprise data warehouse for analytics.
Get metrics from Google BigQuery to:
- Visualize the performance of your BigQuery queries.
- Correlate the performance of your BigQuery queries with your applications.
Setup
Installation
If you haven’t already, set up the Google Cloud Platform integration first
. There are no other installation steps that need to be performed.
Log collection
Google BigQuery logs are collected with Google Cloud Logging and sent to a Cloud pub/sub with an HTTP push forwarder. If you haven’t already, set up a Cloud pub/sub with an HTTP push forwarder
.
Once this is done, export your Google BigQuery logs from Google Cloud Logging to the pub/sub:
- Go to the Google Cloud Logging page
and filter the Google BigQuery logs.
- Click Create Export and name the sink.
- Choose “Cloud Pub/Sub” as the destination and select the pub/sub that was created for that purpose. Note: The pub/sub can be located in a different project.
- Click Create and wait for the confirmation message to show up.
Data Collected
Metrics
gcp.bigquery.job.num_in_flight
(gauge) | Number of in flight jobs
Shown as job |
gcp.bigquery.query.count
(gauge) | Queries in flight
Shown as query |
gcp.bigquery.query.execution_times.avg
(gauge) | Average of query execution times
Shown as second |
gcp.bigquery.query.execution_times.samplecount
(count) | Sample Count of query execution times
Shown as second |
gcp.bigquery.query.execution_times.sumsqdev
(gauge) | Sum of Squared Deviation for query execution times
Shown as second |
gcp.bigquery.query.scanned_bytes
(rate) | Number of scanned bytes. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.query.scanned_bytes_billed
(rate) | Number of scanned bytes billed. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.query.statement_scanned_bytes
(rate) | Scanned bytes broken down by statement type. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.query.statement_scanned_bytes_billed
(rate) | Scanned bytes billed broken down by statement type. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.slots.allocated
(gauge) | Number of BigQuery slots currently allocated for project, slot allocation can be broken down based on reservation and job type |
gcp.bigquery.slots.allocated_for_project
(gauge) | Number of BigQuery slots currently allocated for the project |
gcp.bigquery.slots.allocated_for_project_and_job_type
(gauge) | Number of BigQuery slots currently allocated for the project and job type |
gcp.bigquery.slots.allocated_for_reservation
(gauge) | Number of BigQuery slots currently allocated for project in the reservation |
gcp.bigquery.slots.total_allocated_for_reservation
(gauge) | Number of BigQuery slots currently allocated across all projects in the reservation |
gcp.bigquery.storage.stored_bytes
(gauge) | Number of bytes stored. Note: this metric is available with a 3h delay
Shown as byte |
gcp.bigquery.storage.table_count
(gauge) | Number of tables. Note: this metric is available with a 3h delay
Shown as table |
gcp.bigquery.storage.uploaded_bytes
(rate) | Number of uploaded bytes. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.storage.uploaded_bytes_billed
(rate) | Number of uploaded bytes billed. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.storage.uploaded_row_count
(rate) | Number of uploaded rows. Note: this metric is available with a 6h delay
Shown as row |
Events
The Google BigQuery integration does not include any events.
Service Checks
The Google BigQuery integration does not include any service checks.
Troubleshooting
Need help? Contact Datadog support
.
