Google Cloud Metric Discrepancy

이 페이지는 아직 영어로 제공되지 않습니다. 번역 작업 중입니다.
현재 번역 프로젝트에 대한 질문이나 피드백이 있으신 경우 언제든지 연락주시기 바랍니다.

Overview

Use this guide to troubleshoot metric discrepancies between Google Cloud and Datadog.

Metric discrepancies

Datadog ingests the most granular raw values from Google Cloud. All aggregation seen in Datadog happens on the Datadog side. Datadog’s metrics intake imports the raw values from Google as gauges, and any further aggregation is performed within Datadog. The following steps reconcile the metric gcp.redis.stats.cpu_utilization between Google Cloud and Datadog.

  1. Find the corresponding metric in Google Cloud.

    For the Google Cloud integration, Datadog converts Google Cloud metrics into the format gcp.Google_Cloud_SERVICE_NAME.METRIC_NAME. For the example metric, the Google Cloud service name is redis, and the metric name is stats/cpu_utilization. The full metric name is redis.googleapis.com/stats/cpu_utilization.

  2. Find the most granular dimensions.

    These include all the Resource labels: project_id,region, instance_id, node_id, and Metric labels: role, space, relationship. Refer to the Google Cloud documentation for other metrics.

    labels definition in GCP documentation

    Resource type is associated with each metric under a Google Cloud service. Below are the supported resource types for redis service. The example metric’s resource type is redis_instance. redis_instance has Resource labels: project_id,region, instance_id, node_id.

    redis_instance resource labels

  3. Graph the metric in the Google Cloud Metrics Explorer.

    Search for redis.googleapis.com/stats/cpu_utilization.

    • Time: 1 hour (ideally in UTC)
    • Namespace: Cloud Memorystore Redis Instance
    • Metric name: CPU Seconds
    • Filter: (most granular dimensions) project_id, region, instance_id, node_id, role, space, relationship.
    • Aggregation: Sum (should show the same values when using mean, min, max, sum, or none if using most granular dimensions)
    • Min interval: 1m
    gcp metric explorer

  4. Graph the metric in the Datadog Metrics Explorer.

    datadog metric explorer

    In most cases, after completing steps 1–4, you see the exact same values in both Google Cloud and Datadog. However, in our example, a discrepancy appears at 01:40:00 PM.

    • Datadog: 108.71ms
    • Google Cloud: 0.0018119s
    gcp value
    datadog value

  5. Understand Google Cloud alignment functions.

    This discrepancy occurs because by default, Google Cloud applies a rate alignment for this metric. For details, see the Google cloud alignment function documentation. Click on configure aligner to see that the alignment function is automatically set to rate (0.108711 / 60 ≃ 0.0018119).

    gcp aligner
    gcp rate

  6. Adjust the alignment function in Google Cloud.

    Change the alignment function to delta, min, max, sum, or mean. Assuming you are using the most granular dimensions, the value in Google Cloud should match the value in Datadog.

    gcp delta

Further reading

추가 유용한 문서, 링크 및 기사:

Google Cloud IntegrationINTEGRATION more more