Google Cloud Composer

이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Overview

Google Cloud Composer is a fully managed workflow orchestration service that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.

Use the Datadog Google Cloud Platform integration to collect metrics from Google Cloud Composer.

Setup

Installation

If you haven’t already, set up the Google Cloud Platform integration first. There are no other installation steps.

Log collection

Google Cloud Composer logs are collected with Google Cloud Logging and sent to a Dataflow job through a Cloud Pub/Sub topic. If you haven’t already, set up logging with the Datadog Dataflow template.

Once this is done, export your Google Cloud Composer logs from Google Cloud Logging to the Pub/Sub topic:

  1. Go to the Google Cloud Logging page and filter the Google Cloud Composer logs.
  2. Click Create Export and name the sink.
  3. Choose “Cloud Pub/Sub” as the destination and select the Pub/Sub topic that was created for that purpose. Note: The Pub/Sub topic can be located in a different project.
  4. Click Create and wait for the confirmation message to show up.

Data Collected

Metrics

gcp.composer.environment.api.request_count
(count)		Number of Composer API requests seen so far.
Shown as request
gcp.composer.environment.api.request_latencies.avg
(gauge)		Distribution of Composer API call latencies.
Shown as millisecond
gcp.composer.environment.api.request_latencies.samplecount
(count)		Sample count for API request latencies
Shown as millisecond
gcp.composer.environment.api.request_latencies.sumsqdev
(gauge)		Sum of squared deviation for API request latencies
Shown as second
gcp.composer.environment.dag_processing.parse_error_count
(count)		Number of errors raised during parsing DAG files
Shown as error
gcp.composer.environment.dag_processing.processes
(gauge)		Number of currently running DAG parsing processes
Shown as process
gcp.composer.environment.dag_processing.processesor_timeout_count
(count)		Number of file processors terminated due to processing timeout.
Shown as timeout
gcp.composer.environment.dag_processing.total_parse_time
(gauge)		Number of seconds taken to scan and import all DAG files once
Shown as second
gcp.composer.environment.dagbag_size
(gauge)		The current DAG bag size
gcp.composer.environment.database.cpu.reserved_cores
(gauge)		Number of cores reserved for the database instance
Shown as core
gcp.composer.environment.database.cpu.usage_time
(count)		CPU usage time of the database instance, in seconds
Shown as second
gcp.composer.environment.database.cpu.utilization
(gauge)		CPU utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.disk.bytes_used
(gauge)		Used disk space on the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.quota
(gauge)		Maximum data disk size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.utilization
(gauge)		Disk quota usage ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.memory.bytes_used
(gauge)		Memory usage of the database instance in bytes
Shown as byte
gcp.composer.environment.database.memory.quota
(gauge)		Maximum RAM size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.memory.utilization
(gauge)		Memory utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.network.received_bytes_count
(count)		Number of bytes received by the database instance
Shown as byte
gcp.composer.environment.database.network.sent_bytes_count
(count)		Number of bytes sent by the database instance
Shown as byte
gcp.composer.environment.database_health
(gauge)		Healthiness of Composer Airflow database
gcp.composer.environment.executor.open_slots
(gauge)		Number of open slots on executor
gcp.composer.environment.executor.queued_tasks
(gauge)		Number of queued tasks on executor
Shown as task
gcp.composer.environment.executor.running_tasks
(gauge)		Number of running tasks on executor
Shown as task
gcp.composer.environment.finished_task_instance_count
(count)		Overall number of finished task instances
Shown as instance
gcp.composer.environment.healthy
(gauge)		Healthiness of Composer environment.
gcp.composer.environment.num_celery_workers
(gauge)		Number of Celery workers.
Shown as worker
gcp.composer.environment.num_workflows
(gauge)		Number of workflows.
gcp.composer.environment.scheduler_heartbeat_count
(count)		Scheduler heartbeats
gcp.composer.environment.task_queue_length
(gauge)		Number of tasks in queue.
Shown as task
gcp.composer.environment.unfinished_task_instances
(gauge)		Overall task instances in not finished state
Shown as instance
gcp.composer.environment.web_server.cpu.reserved_cores
(gauge)		Number of cores reserved for the web server instance
Shown as core
gcp.composer.environment.web_server.cpu.usage_time
(count)		CPU usage time of the web server instance, in seconds
Shown as second
gcp.composer.environment.web_server.health
(gauge)		Healthiness of Airflow web server
gcp.composer.environment.web_server.memory.bytes_used
(gauge)		Memory usage of the web server instance in bytes
Shown as byte
gcp.composer.environment.web_server.memory.quota
(gauge)		Maximum RAM size of the web server instance, in bytes
Shown as byte
gcp.composer.environment.worker.max_workers
(gauge)		Maximum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.min_workers
(gauge)		Minimum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.pod_eviction_count
(count)		Number of Airflow worker pods evictions
Shown as eviction
gcp.composer.environment.worker.scale_factor_target
(gauge)		Scale factor for Airflow workers count
gcp.composer.environment.zombie_task_killed_count
(count)		Number of zombie tasks killed
Shown as task
gcp.composer.workflow.run_count
(count)		Number of workflow runs completed so far.
gcp.composer.workflow.run_duration
(gauge)		Duration of workflow run completion.
Shown as second
gcp.composer.workflow.task.run_count
(count)		Number of workflow tasks completed so far.
Shown as task
gcp.composer.workflow.task.run_duration
(gauge)		Duration of task completion.
Shown as second

Events

The Google Cloud Composer integration does not include any events.

Service Checks

The Google Cloud Composer integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.