Google Storage

Overview

Google Cloud Storage is unified object storage for developers and enterprises, from live data serving to data analytics/ML to data archiving.

Get metrics from Google Storage to:

  • Visualize the performance of your Storage services.
  • Correlate the performance of your Storage services with your applications.

Setup

Installation

If you haven’t already, set up the Google Cloud Platform integration first. There are no other installation steps.

Configuration

To collect custom Cloud Storage labels as tags, enable the cloud asset inventory permission.

Log collection

Google Cloud Storage logs are collected with Google Cloud Logging and sent to a Dataflow job through a Cloud Pub/Sub topic. If you haven’t already, set up logging with the Datadog Dataflow template.

Once this is done, export your Google Cloud Storage logs from Google Cloud Logging to the Pub/Sub topic:

  1. Go to the Google Cloud Logging page and filter the Google Cloud Storage logs.
  2. Click Create Export and name the sink.
  3. Choose “Cloud Pub/Sub” as the destination and select the Pub/Sub topic that was created for that purpose. Note: The Pub/Sub topic can be located in a different project.
  4. Click Create and wait for the confirmation message to show up.

Data Collected

Metrics

gcp.storage.api.lro_count
(count)
Count of completed long-running operations, since the last sample.
gcp.storage.api.request_count
(count)
The number of API calls.
Shown as request
gcp.storage.authn.authentication_count
(count)
The number of HMAC/RSA signed requests.
Shown as request
gcp.storage.authz.acl_based_object_access_count
(count)
The number of requests that result in an object being granted access solely due to object ACLs.
Shown as request
gcp.storage.authz.acl_operations_count
(count)
The usage of ACL operations.
Shown as operation
gcp.storage.authz.object_specific_acl_mutation_count
(count)
The number of changes made to object specific ACLs.
gcp.storage.autoclass.transition_operation_count
(count)
Total number of storage class transition operations initiated by Autoclass.
gcp.storage.autoclass.transitioned_bytes_count
(count)
Total bytes transitioned by Autoclass.
Shown as byte
gcp.storage.client.grpc.client.attempt.duration.avg
(gauge)
The average of the cumulative time taken to complete an RPC attempt including the time it takes to pick a subchannel.
Shown as second
gcp.storage.client.grpc.client.attempt.duration.samplecount
(gauge)
The sample count for the cumulative time taken to complete an RPC attempt including the time it takes to pick a subchannel.
Shown as second
gcp.storage.client.grpc.client.attempt.duration.sumsqdev
(gauge)
The sum of squared deviation for the cumulative time taken to complete an RPC attempt including the time it takes to pick a subchannel.
Shown as second
gcp.storage.client.grpc.client.attempt.rcvd_total_compressed_message_size.avg
(count)
The average cumulative bytes (compressed but not encrypted) received across all response messages (metadata excluded) per RPC attempt; does not include gRPC or transport framing bytes.
Shown as byte
gcp.storage.client.grpc.client.attempt.rcvd_total_compressed_message_size.samplecount
(count)
The sample count for the cumulative bytes (compressed but not encrypted) received across all response messages (metadata excluded) per RPC attempt; does not include gRPC or transport framing bytes.
Shown as byte
gcp.storage.client.grpc.client.attempt.rcvd_total_compressed_message_size.sumsqdev
(count)
The sum of squared deviation for the cumulative bytes (compressed but not encrypted) received across all response messages (metadata excluded) per RPC attempt; does not include gRPC or transport framing bytes.
Shown as byte
gcp.storage.client.grpc.client.attempt.sent_total_compressed_message_size.avg
(count)
The average cumulative bytes (compressed but not encrypted) sent across all request messages (metadata excluded) per RPC attempt; does not include gRPC or transport framing bytes.
Shown as byte
gcp.storage.client.grpc.client.attempt.sent_total_compressed_message_size.samplecount
(count)
The sample count for the cumulative bytes (compressed but not encrypted) sent across all request messages (metadata excluded) per RPC attempt; does not include gRPC or transport framing bytes.
Shown as byte
gcp.storage.client.grpc.client.attempt.sent_total_compressed_message_size.sumsqdev
(count)
The sum of squared deviation for the cumulative bytes (compressed but not encrypted) sent across all request messages (metadata excluded) per RPC attempt; does not include gRPC or transport framing bytes.
Shown as byte
gcp.storage.client.grpc.client.attempt.started
(count)
The cumulative number of RPC attempts started, including those that have not completed.
gcp.storage.client.grpc.client.call.duration.avg
(gauge)
The average of the cumulative time the gRPC library takes to complete an RPC from the application’s perspective.
Shown as second
gcp.storage.client.grpc.client.call.duration.samplecount
(gauge)
The sample count for the cumulative time the gRPC library takes to complete an RPC from the application’s perspective.
Shown as second
gcp.storage.client.grpc.client.call.duration.sumsqdev
(gauge)
The sum of squared deviation of the cumulative time the gRPC library takes to complete an RPC from the application’s perspective.
Shown as second
gcp.storage.client.grpc.lb.rls.cache_entries
(gauge)
Number of entries in the RLS cache.
Shown as entry
gcp.storage.client.grpc.lb.rls.cache_size
(gauge)
The current size of the RLS cache.
Shown as byte
gcp.storage.client.grpc.lb.rls.default_target_picks
(count)
Cumulative number of LB picks sent to the default target.
gcp.storage.client.grpc.lb.rls.failed_picks
(count)
Cumulative number of LB picks failed due to either a failed RLS request or the RLS channel being throttled.
gcp.storage.client.grpc.lb.rls.target_picks
(count)
Cumulative number of LB picks sent to each RLS target. Note that if the default target is also returned by the RLS server, RPCs sent to that target from the cache are counted in this metric, not in grpc.rls.default_target_picks.
gcp.storage.client.grpc.lb.wrr.endpoint_weight_not_yet_usable
(count)
Cumulative number of endpoints from each scheduler update that do not yet have usable weight information (either the load report has not yet been received, or it is within the blackout period).
gcp.storage.client.grpc.lb.wrr.endpoint_weight_stale
(count)
Cumulative number of endpoints from each scheduler update whose latest weight is older than the expiration period.
gcp.storage.client.grpc.lb.wrr.endpoint_weights.avg
(count)
The average of the cumulative bucket counts. The histogram buckets will be endpoint weight ranges. Each bucket will be a counter that is incremented once for every endpoint whose weight is within that range. Note that endpoints without usable weights will have weight 0.
gcp.storage.client.grpc.lb.wrr.endpoint_weights.samplecount
(count)
The sample count for the cumulative bucket counts. The histogram buckets will be endpoint weight ranges. Each bucket will be a counter that is incremented once for every endpoint whose weight is within that range. Note that endpoints without usable weights will have weight 0.
gcp.storage.client.grpc.lb.wrr.endpoint_weights.sumsqdev
(count)
The sum of squared deviation for the cumulative bucket counts. The histogram buckets will be endpoint weight ranges. Each bucket will be a counter that is incremented once for every endpoint whose weight is within that range. Note that endpoints without usable weights will have weight 0.
gcp.storage.client.grpc.lb.wrr.rr_fallback
(count)
Cumulative number of scheduler updates in which there were not enough endpoints with valid weight, which caused the WRR policy to fall back to RR behavior.
gcp.storage.client.grpc.xds_client.connected
(gauge)
Whether or not the xDS client currently has a working ADS stream to the xDS server. For a given server, this will be set to 1 when the stream is initially created. It will be set to 0 when we have a connectivity failure or when the ADS stream fails without seeing a response message, as per A57. Once set to 0, it will be reset to 1 when we receive the first response on an ADS stream.
gcp.storage.client.grpc.xds_client.resource_updates_invalid
(count)
Cumulative count of resources received that were considered invalid.
Shown as resource
gcp.storage.client.grpc.xds_client.resource_updates_valid
(count)
Cumulative count of resources received that were considered valid. The counter will be incremented even for resources that have not changed.
Shown as resource
gcp.storage.client.grpc.xds_client.resources
(gauge)
Number of xDS resources.
Shown as resource
gcp.storage.client.grpc.xds_client.server_failure
(count)
Cumulative count of xDS servers going from healthy to unhealthy. A server goes unhealthy when we have a connectivity failure or when the ADS stream fails without seeing a response message, as per gRFC A57.
gcp.storage.network.received_bytes_count
(count)
The number of bytes received over the network.
Shown as byte
gcp.storage.network.sent_bytes_count
(count)
The number of bytes sent over the network.
Shown as byte
gcp.storage.quota.dualregion_google_egress_bandwidth.limit
(gauge)
Current limit on quota metric storage.googleapis.com/dualregion_google_egress_bandwidth.
Shown as bit
gcp.storage.quota.dualregion_google_egress_bandwidth.usage
(count)
Current usage on quota metric storage.googleapis.com/dualregion_google_egress_bandwidth.
Shown as bit
gcp.storage.quota.dualregion_internet_egress_bandwidth.limit
(gauge)
Current limit on quota metric storage.googleapis.com/dualregion_internet_egress_bandwidth.
Shown as bit
gcp.storage.quota.dualregion_internet_egress_bandwidth.usage
(count)
Current usage on quota metric storage.googleapis.com/dualregion_internet_egress_bandwidth.
Shown as bit
gcp.storage.replication.meeting_rpo
(gauge)
A boolean value of whether objects are meeting RPO or not. For ASYNC_TURBO, RPO is 15 minutes; For DEFAULT, RPO is 12 hours.
gcp.storage.replication.missing_rpo_minutes_last_30d
(gauge)
Total number of minutes where the Recovery Point Objective (RPO) was missed, as measured over the most recent 30 days. The metric is delayed by storage.googleapis.com/replication/time_since_metrics_updated.
gcp.storage.replication.object_replications_last_30d
(gauge)
Total number of object replications, as measured over the most recent 30 days. The metric can be broken down by whether each object replication was meeting or missing Recovery Point Objective (RPO). The metric is delayed by storage.googleapis.com/replication/time_since_metrics_updated.
gcp.storage.replication.time_since_metrics_updated
(gauge)
Seconds since the last time when storage.googleapis.com/replication/* metrics were calculated.
Shown as second
gcp.storage.replication.turbo_max_delay
(gauge)
Delay represents the age in seconds of the oldest object that has not been replicated. Objects uploaded before this time have been replicated.
Shown as second
gcp.storage.replication.v2.object_replications_last_30d
(gauge)
Total number of object replications, as measured over the most recent 30 days. The metric can be broken down by whether each object replication was meeting or missing the design target. The metric is delayed by storage.googleapis.com/replication/v2/time_since_metrics_updated.
gcp.storage.replication.v2.time_since_metrics_updated
(gauge)
Seconds since the last time when metric storage.googleapis.com/replication/missing_rpo_minutes_last_30d and /replication/v2/object_replications_last_30d were calculated.
Shown as second
gcp.storage.storage.object_count
(gauge)
The total number of objects per bucket.
Shown as object
gcp.storage.storage.total_byte_seconds
(gauge)
The total daily storage in byte seconds used.
Shown as second
gcp.storage.storage.total_bytes
(gauge)
The total size of all objects in the bucket.
Shown as byte
gcp.storage.storage.v2.deleted_bytes
(count)
Delta count of deleted bytes per bucket, grouped by storage class. This value is measured and reported once per day, and there might be a delay after measuring before the value becomes available in Datadog. On days with no deletes, no datapoint is exported for this metric.
Shown as byte
gcp.storage.storage.v2.total_byte_seconds
(gauge)
Total daily storage in byte*seconds used by the bucket, grouped by storage class and type where type can be live-object, noncurrent-object, soft-deleted-object and multipart-upload. This value is measured once per day, and there might be a delay after measuring before the value becomes available in Datadog. Once available, the value is repeated at each sampling interval throughout the day.
gcp.storage.storage.v2.total_bytes
(gauge)
Total size of all objects and multipart-uploads in the bucket, grouped by storage class and type where type can be live-object, noncurrent-object, soft-deleted-object and multipart-upload. This value is measured once per day, and there might be a delay after measuring before the value becomes available in Datadog. Once available, the value is repeated at each sampling interval throughout the day.
Shown as byte
gcp.storage.storage.v2.total_count
(gauge)
Total number of objects and multipart-uploads per bucket, grouped by storage class and type where type can be live-object, noncurrent-object, soft-deleted-object and multipart-upload. This value is measured once per day, and there might be a delay after measuring before the value becomes available in Datadog. Once available, the value is repeated at each sampling interval throughout the day.

Events

The Google Cloud Storage integration does not include any events.

Service Checks

The Google Cloud Storage integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.