- 필수 기능
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- 디지털 경험
- 소프트웨어 제공
- 보안
- 로그 관리
- 관리
- 인프라스트럭처
- ci
- containers
- csm
- ndm
- otel_guides
- overview
- slos
- synthetics
- tests
- 워크플로
Google Cloud Dataproc is a fast, easy-to-use, fully-managed cloud service for running Apache Spark and Apache Hadoop clusters in a simpler, more cost-efficient way.
Use the Datadog Google Cloud Platform integration to collect metrics from Google Cloud Dataproc.
If you haven’t already, set up the Google Cloud Platform integration first. There are no other installation steps.
Google Cloud Dataproc logs are collected with Google Cloud Logging and sent to a Dataflow job through a Cloud Pub/Sub topic. If you haven’t already, set up logging with the Datadog Dataflow template.
Once this is done, export your Google Cloud Dataproc logs from Google Cloud Logging to the Pub/Sub topic:
gcp.dataproc.batch.spark.executors (gauge) | Indicates the number of Batch Spark executors. Shown as worker |
gcp.dataproc.cluster.hdfs.datanodes (gauge) | Indicates the number of HDFS DataNodes that are running inside a cluster. Shown as node |
gcp.dataproc.cluster.hdfs.storage_capacity (gauge) | Indicates capacity of HDFS system running on a cluster in GB. Shown as gibibyte |
gcp.dataproc.cluster.hdfs.storage_utilization (gauge) | The percentage of HDFS storage currently used. Shown as percent |
gcp.dataproc.cluster.hdfs.unhealthy_blocks (gauge) | Indicates the number of unhealthy blocks inside the cluster. Shown as block |
gcp.dataproc.cluster.job.completion_time.avg (gauge) | The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed. Shown as millisecond |
gcp.dataproc.cluster.job.completion_time.samplecount (count) | Sample count for cluster job completion time. Shown as millisecond |
gcp.dataproc.cluster.job.completion_time.sumsqdev (gauge) | Sum of squared deviation for cluster job completion time. Shown as second |
gcp.dataproc.cluster.job.duration.avg (gauge) | The time jobs have spent in a given state. Shown as millisecond |
gcp.dataproc.cluster.job.duration.samplecount (count) | Sample count for cluster job duration. Shown as millisecond |
gcp.dataproc.cluster.job.duration.sumsqdev (gauge) | Sum of squared deviation for cluster job duration. Shown as second |
gcp.dataproc.cluster.job.failed_count (count) | Indicates the number of jobs that have failed on a cluster. Shown as job |
gcp.dataproc.cluster.job.running_count (gauge) | Indicates the number of jobs that are running on a cluster. Shown as job |
gcp.dataproc.cluster.job.submitted_count (count) | Indicates the number of jobs that have been submitted to a cluster. Shown as job |
gcp.dataproc.cluster.nodes.expected (gauge) | Indicates the number of nodes that are expected in a cluster. Shown as node |
gcp.dataproc.cluster.nodes.failed_count (count) | Indicates the number of nodes that have failed in a cluster. Shown as node |
gcp.dataproc.cluster.nodes.recovered_count (count) | Indicates the number of nodes that are detected as failed and have been successfully removed from cluster. Shown as node |
gcp.dataproc.cluster.nodes.running (gauge) | Indicates the number of nodes in running state. Shown as node |
gcp.dataproc.cluster.operation.completion_time.avg (gauge) | The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed. Shown as millisecond |
gcp.dataproc.cluster.operation.completion_time.samplecount (count) | Sample count for cluster operation completion time. Shown as millisecond |
gcp.dataproc.cluster.operation.completion_time.sumsqdev (gauge) | Sum of squared deviation for cluster operation completion time. Shown as second |
gcp.dataproc.cluster.operation.duration.avg (gauge) | The time operations have spent in a given state. Shown as millisecond |
gcp.dataproc.cluster.operation.duration.samplecount (count) | Sample count for cluster operation duration. Shown as millisecond |
gcp.dataproc.cluster.operation.duration.sumsqdev (gauge) | Sum of squared deviation for cluster operation duration. Shown as second |
gcp.dataproc.cluster.operation.failed_count (count) | Indicates the number of operations that have failed on a cluster. Shown as operation |
gcp.dataproc.cluster.operation.running_count (gauge) | Indicates the number of operations that are running on a cluster. Shown as operation |
gcp.dataproc.cluster.operation.submitted_count (count) | Indicates the number of operations that have been submitted to a cluster. Shown as operation |
gcp.dataproc.cluster.yarn.allocated_memory_percentage (gauge) | The percentage of YARN memory is allocated. Shown as percent |
gcp.dataproc.cluster.yarn.apps (gauge) | Indicates the number of active YARN applications. |
gcp.dataproc.cluster.yarn.containers (gauge) | Indicates the number of YARN containers. Shown as container |
gcp.dataproc.cluster.yarn.memory_size (gauge) | Indicates the YARN memory size in GB. Shown as gibibyte |
gcp.dataproc.cluster.yarn.nodemanagers (gauge) | Indicates the number of YARN NodeManagers running inside cluster. |
gcp.dataproc.cluster.yarn.pending_memory_size (gauge) | The current memory request, in GB, that is pending to be fulfilled by the scheduler. Shown as gibibyte |
gcp.dataproc.cluster.yarn.virtual_cores (gauge) | Indicates the number of virtual cores in YARN. Shown as core |
gcp.dataproc.job.state (gauge) | Indicates whether job is currently in a particular state or not. |
gcp.dataproc.session.spark.executors (gauge) | Indicates the number of Session Spark executors. Shown as worker |
The Google Cloud Dataproc integration does not include any events.
The Google Cloud Dataproc integration does not include any service checks.
Need help? Contact Datadog support.