BigQuery Storage API

Integration version1.0.0

To find out if this integration is available in your organization, see your Datadog Integrations page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email support@ddog-gov.com.

Overview

The BigQuery Storage API provides high-throughput, direct read and write access to data stored in BigQuery, using an RPC-based protocol instead of paginated REST calls or batch exports. It supports parallel streams, column projection and filtering, and snapshot-consistent reads, making it well-suited for distributed processing frameworks and data pipelines, such as Dataflow, that need fast, structured access to BigQuery-managed storage.

Use the Datadog Google Cloud Platform integration to collect metrics from the BigQuery Storage API to:

  • Visualize the throughput and latency of your BigQuery Storage API read and write connections.
  • Track uploaded rows and bytes to monitor the volume of data flowing through your Storage Write API streams.
  • Correlate BigQuery Storage API performance with the Dataflow jobs and applications that depend on it.

Setup

Installation

If you haven’t already, set up the Google Cloud Platform integration first. There are no other installation steps.

Data Collected

Metrics

gcp.bigquerystorage.dataflow_write.connection_results_count
(count)		Total accumulation of BigQuery Write API connection results.
gcp.bigquerystorage.dataflow_write.request_count
(count)		Total accumulation of BigQuery Write API request count.
gcp.bigquerystorage.dataflow_write.server_side_latencies.avg
(gauge)		The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse.
gcp.bigquerystorage.dataflow_write.server_side_latencies.samplecount
(count)		The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse.
gcp.bigquerystorage.dataflow_write.server_side_latencies.sumsqdev
(count)		The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse.
gcp.bigquerystorage.dataflow_write.transcoding_latencies.avg
(gauge)		The duration distribution (in milliseconds) in BigQuery Write API frontend taken by transcoding data.
gcp.bigquerystorage.dataflow_write.transcoding_latencies.samplecount
(count)		The duration distribution (in milliseconds) in BigQuery Write API frontend taken by transcoding data.
gcp.bigquerystorage.dataflow_write.transcoding_latencies.sumsqdev
(count)		The duration distribution (in milliseconds) in BigQuery Write API frontend taken by transcoding data.
gcp.bigquerystorage.dataflow_write.uploaded_bytes_count
(count)		Total accumulation of BigQuery Write API ingested bytes.
gcp.bigquerystorage.dataflow_write.uploaded_row_count
(count)		Total accumulation of BigQuery Write API ingested rows.
gcp.bigquerystorage.read.concurrent_streams
(gauge)		Number of concurrent ReadRows streams.
gcp.bigquerystorage.write.concurrent_connections
(gauge)		Concurrent connection per project for BigQuery Write API. To be consistent with concurrent connection quota, the metric is by default on the project that initiated the AppendRowsRequests (associated with the API key or the service account).
gcp.bigquerystorage.write.connection_results_count
(count)		Total accumulation of connection results with BigQuery Write API.
gcp.bigquerystorage.write.server_side_latencies.avg
(gauge)		The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse.
gcp.bigquerystorage.write.server_side_latencies.samplecount
(count)		The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse.
gcp.bigquerystorage.write.server_side_latencies.sumsqdev
(count)		The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse.
gcp.bigquerystorage.write.transcoding_latencies.avg
(gauge)		The transcoding latency distribution (in milliseconds) in BigQuery Write API frontend.
gcp.bigquerystorage.write.transcoding_latencies.samplecount
(count)		The transcoding latency distribution (in milliseconds) in BigQuery Write API frontend.
gcp.bigquerystorage.write.transcoding_latencies.sumsqdev
(count)		The transcoding latency distribution (in milliseconds) in BigQuery Write API frontend.
gcp.bigquerystorage.write.uploaded_bytes_count
(count)		Total accumulation of BigQuery Write API ingested bytes.
gcp.bigquerystorage.write.uploaded_row_count
(count)		Total accumulation of BigQuery Write API ingested rows.

Events

The BigQuery Storage API integration does not include any events.

Service Checks

The BigQuery Storage API integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.