Overview

Use this guide to troubleshoot metric discrepancies between Google Cloud and Datadog.

Metric discrepancies

Datadog ingests the most granular raw values from Google Cloud. All aggregation seen in Datadog happens on the Datadog side. Datadog’s metrics intake imports the raw values from Google as gauges, and any further aggregation is performed within Datadog. The following steps reconcile the metric gcp.redis.stats.cpu_utilization between Google Cloud and Datadog.

1. Find the corresponding metric in Google Cloud.

   For the Google Cloud integration, Datadog converts Google Cloud metrics into the format gcp.Google_Cloud_SERVICE_NAME.METRIC_NAME. For the example metric, the Google Cloud service name is redis, and the metric name is stats/cpu_utilization. The full metric name is redis.googleapis.com/stats/cpu_utilization.

2. Find the most granular dimensions.

   These include all the Resource labels: project_id,region, instance_id, node_id, and Metric labels: role, space, relationship. Refer to the Google Cloud documentation for other metrics.

   Resource type is associated with each metric under a Google Cloud service. Below are the supported resource types for redis service. The example metric’s resource type is redis_instance. redis_instance has Resource labels: project_id,region, instance_id, node_id.

   • redis.googleapis.com/Cluster
   • redis_instance
   • redis.googleapis.com/ClusterNode

3. Graph the metric in the Google Cloud Metrics Explorer.

   Search for redis.googleapis.com/stats/cpu_utilization.

   • Time: 1 hour (ideally in UTC)
   • Namespace: Cloud Memorystore Redis Instance
   • Metric name: CPU Seconds
   • Filter: (most granular dimensions) project_id, region, instance_id, node_id, role, space, relationship.
   • Aggregation: Sum (should show the same values when using mean, min, max, sum, or none if using most granular dimensions)
   • Min interval: 1m

4. Graph the metric in the Datadog Metrics Explorer.

   In most cases, after completing steps 1–4, you see the exact same values in both Google Cloud and Datadog. However, in our example, a discrepancy appears at 01:40:00 PM.

   • Datadog: 108.71ms
   • Google Cloud: 0.0018119s

5. Understand Google Cloud alignment functions.

   This discrepancy occurs because by default, Google Cloud applies a rate alignment for this metric. For details, see the Google cloud alignment function documentation. Click on configure aligner to see that the alignment function is automatically set to rate (0.108711 / 60 ≃ 0.0018119).

6. Adjust the alignment function in Google Cloud.

   Change the alignment function to delta, min, max, sum, or mean. Assuming you are using the most granular dimensions, the value in Google Cloud should match the value in Datadog.

Further reading

Documentation, liens et articles supplémentaires utiles:

Google Cloud IntegrationINTEGRATION