- 필수 기능
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- 디지털 경험
- 소프트웨어 제공
- 보안
- 로그 관리
- 관리
- 인프라스트럭처
- ci
- containers
- csm
- ndm
- otel_guides
- overview
- slos
- synthetics
- tests
- 워크플로
Google Cloud Dataflow is a fully-managed service for transforming and enriching data in stream (real time) and batch (historical) modes with equal reliability and expressiveness.
Use the Datadog Google Cloud integration to collect metrics from Google Cloud Dataflow.
If you haven’t already, set up the Google Cloud Platform integration first. There are no other installation steps.
Google Cloud Dataflow logs are collected with Google Cloud Logging and sent to a Dataflow job through a Cloud Pub/Sub topic. If you haven’t already, set up logging with the Datadog Dataflow template.
Once this is done, export your Google Cloud Dataflow logs from Google Cloud Logging to the Pub/sub:
gcp.dataflow.job.billable_shuffle_data_processed (gauge) | The billable bytes of shuffle data processed by this Dataflow job. Shown as byte |
gcp.dataflow.job.current_num_vcpus (gauge) | The number of vCPUs currently being used by this Dataflow job. Shown as cpu |
gcp.dataflow.job.current_shuffle_slots (gauge) | The current shuffle slots used by this Dataflow job. |
gcp.dataflow.job.data_watermark_age (gauge) | The age (time since event timestamp) of the most recent item of data that has been fully processed by the pipeline. Shown as second |
gcp.dataflow.job.elapsed_time (gauge) | Duration that the current run of this pipeline has been in the Running state so far, in seconds. When a run completes, this stays at the duration of that run until the next run starts. Shown as second |
gcp.dataflow.job.element_count (count) | Number of elements added to the pcollection so far. Shown as item |
gcp.dataflow.job.estimated_byte_count (count) | An estimated number of bytes added to the pcollection so far. Shown as byte |
gcp.dataflow.job.is_failed (gauge) | Has this job failed. |
gcp.dataflow.job.system_lag (gauge) | The current maximum duration that an item of data has been awaiting processing, in seconds. Shown as second |
gcp.dataflow.job.total_memory_usage_time (gauge) | The total GB seconds of memory allocated to this Dataflow job. Shown as gibibyte |
gcp.dataflow.job.total_pd_usage_time (gauge) | The total GB seconds for all persistent disk used by all workers associated with this Dataflow job. Shown as gibibyte |
gcp.dataflow.job.total_shuffle_data_processed (gauge) | The total bytes of shuffle data processed by this Dataflow job. Shown as byte |
gcp.dataflow.job.total_streaming_data_processed (gauge) | The total bytes of streaming data processed by this Dataflow job. Shown as byte |
gcp.dataflow.job.total_vcpu_time (gauge) | The total vCPU seconds used by this Dataflow job. |
gcp.dataflow.job.user_counter (gauge) | A user-defined counter metric. |
gcp.dataflow.quota.region_endpoint_shuffle_slot.exceeded (count) | Number of attempts to exceed the limit on quota metric dataflow.googleapis.com/regionendpointshuffle_slot. |
gcp.dataflow.quota.region_endpoint_shuffle_slot.limit (gauge) | Current limit on quota metric dataflow.googleapis.com/regionendpointshuffle_slot. |
gcp.dataflow.quota.region_endpoint_shuffle_slot.usage (gauge) | Current usage on quota metric dataflow.googleapis.com/regionendpointshuffle_slot. |
The Google Cloud Dataflow integration does not include any events.
The Google Cloud Dataflow integration does not include any service checks.
Need help? Contact Datadog support.
Additional helpful documentation, links, and articles: