Overview
BigQuery is Google’s fully managed, petabyte scale, low cost enterprise data warehouse for analytics.
Get metrics from Google BigQuery to:
- Visualize the performance of your BigQuery queries.
- Correlate the performance of your BigQuery queries with your applications.
Setup
Installation
If you haven’t already, set up the Google Cloud Platform integration first. There are no other installation steps that need to be performed.
Log collection
Google BigQuery logs are collected with Google Cloud Logging and sent to a Cloud pub/sub with an HTTP push forwarder. If you haven’t already, set up a Cloud pub/sub with an HTTP push forwarder.
Once this is done, export your Google BigQuery logs from Google Cloud Logging to the pub/sub:
- Go to the Google Cloud Logging page and filter the Google BigQuery logs.
- Click Create Export and name the sink.
- Choose “Cloud Pub/Sub” as the destination and select the pub/sub that was created for that purpose. Note: The pub/sub can be located in a different project.
- Click Create and wait for the confirmation message to show up.
Data Collected
Metrics
gcp.bigquery.job.num_in_flight
(gauge) | Number of in flight jobs
Shown as job |
gcp.bigquery.query.count
(gauge) | Queries in flight
Shown as query |
gcp.bigquery.query.execution_times.avg
(gauge) | Average of query execution times
Shown as second |
gcp.bigquery.query.execution_times.samplecount
(count) | Sample Count of query execution times
Shown as second |
gcp.bigquery.query.execution_times.sumsqdev
(gauge) | Sum of Squared Deviation for query execution times
Shown as second |
gcp.bigquery.query.scanned_bytes
(rate) | Number of scanned bytes. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.query.scanned_bytes_billed
(rate) | Number of scanned bytes billed. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.query.statement_scanned_bytes
(rate) | Scanned bytes broken down by statement type. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.query.statement_scanned_bytes_billed
(rate) | Scanned bytes billed broken down by statement type. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.slots.allocated
(gauge) | Number of BigQuery slots currently allocated for project, slot allocation can be broken down based on reservation and job type |
gcp.bigquery.slots.allocated_for_project
(gauge) | Number of BigQuery slots currently allocated for the project |
gcp.bigquery.slots.allocated_for_project_and_job_type
(gauge) | Number of BigQuery slots currently allocated for the project and job type |
gcp.bigquery.slots.allocated_for_reservation
(gauge) | Number of BigQuery slots currently allocated for project in the reservation |
gcp.bigquery.slots.total_allocated_for_reservation
(gauge) | Number of BigQuery slots currently allocated across all projects in the reservation |
gcp.bigquery.storage.stored_bytes
(gauge) | Number of bytes stored. Note: this metric is available with a 3h delay
Shown as byte |
gcp.bigquery.storage.table_count
(gauge) | Number of tables. Note: this metric is available with a 3h delay
Shown as table |
gcp.bigquery.storage.uploaded_bytes
(rate) | Number of uploaded bytes. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.storage.uploaded_bytes_billed
(rate) | Number of uploaded bytes billed. Note: this metric is available with a 6h delay
Shown as byte |
gcp.bigquery.storage.uploaded_row_count
(rate) | Number of uploaded rows. Note: this metric is available with a 6h delay
Shown as row |
Events
The Google BigQuery integration does not include any events.
Service Checks
The Google BigQuery integration does not include any service checks.
Troubleshooting
Need help? Contact Datadog support.
