Google Cloud Composer

개요

Google Cloud Composer는 클라우드 및 온프레미스 데이터센터 전반에 걸쳐 파이프라인을 작성, 예약 및 모니터링할 수 있도록 도와드리는 완전관리형 워크플로우 오케스트레이션 서비스입니다.

Datadog Google Cloud Platform 통합을 사용하여 Google Cloud Composer에서 메트릭을 수집합니다.

설정

설치

아직 설치하지 않았다면 먼저 Google 클라우드 플랫폼 통합을 설정합니다. 그 외 다른 설치가 필요하지 않습니다.

로그 수집

Google Cloud Composer 로그는 Google Cloud Logging으로 수집하여 클라우드 Pub/Sub 토픽을 통해 데이터 플로우 작업으로 전송됩니다. 아직 설정하지 않았다면 Datadog 데이터 플로우 템플릿으로 로깅을 설정하세요.

해당 작업이 완료되면 Google Cloud Logging에서 Google Cloud Composer 로그를 다음 Pub/Sub 주제로 내보냅니다.

  1. Google Cloud Logging 페이지로 이동해 Google Cloud Composer 로그를 필터링하세요.
  2. Create Export를 클릭하고 싱크 이름을 지정하세요.
  3. “Cloud Pub/Sub"를 대상으로 선택하고 해당 목적으로 생성된 Pub/Sub 주제를 선택합니다. 참고: Pub/Sub 주제는 다른 프로젝트에 있을 수 있습니다.
  4. Create를 클릭하고 확인 메시지가 나타날 때까지 기다립니다.

수집한 데이터

메트릭

gcp.composer.environment.api.request_count
(count)		Number of Composer API requests seen so far.
Shown as request
gcp.composer.environment.api.request_latencies.avg
(gauge)		Distribution of Composer API call latencies.
Shown as millisecond
gcp.composer.environment.api.request_latencies.samplecount
(count)		Sample count for API request latencies
Shown as millisecond
gcp.composer.environment.api.request_latencies.sumsqdev
(gauge)		Sum of squared deviation for API request latencies
Shown as second
gcp.composer.environment.dag_processing.parse_error_count
(count)		Number of errors raised during parsing DAG files
Shown as error
gcp.composer.environment.dag_processing.processes
(gauge)		Number of currently running DAG parsing processes
Shown as process
gcp.composer.environment.dag_processing.processesor_timeout_count
(count)		Number of file processors terminated due to processing timeout.
Shown as timeout
gcp.composer.environment.dag_processing.total_parse_time
(gauge)		Number of seconds taken to scan and import all DAG files once
Shown as second
gcp.composer.environment.dagbag_size
(gauge)		The current DAG bag size
gcp.composer.environment.database.cpu.reserved_cores
(gauge)		Number of cores reserved for the database instance
Shown as core
gcp.composer.environment.database.cpu.usage_time
(count)		CPU usage time of the database instance, in seconds
Shown as second
gcp.composer.environment.database.cpu.utilization
(gauge)		CPU utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.disk.bytes_used
(gauge)		Used disk space on the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.quota
(gauge)		Maximum data disk size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.utilization
(gauge)		Disk quota usage ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.memory.bytes_used
(gauge)		Memory usage of the database instance in bytes
Shown as byte
gcp.composer.environment.database.memory.quota
(gauge)		Maximum RAM size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.memory.utilization
(gauge)		Memory utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.network.received_bytes_count
(count)		Number of bytes received by the database instance
Shown as byte
gcp.composer.environment.database.network.sent_bytes_count
(count)		Number of bytes sent by the database instance
Shown as byte
gcp.composer.environment.database_health
(gauge)		Healthiness of Composer Airflow database
gcp.composer.environment.executor.open_slots
(gauge)		Number of open slots on executor
gcp.composer.environment.executor.queued_tasks
(gauge)		Number of queued tasks on executor
Shown as task
gcp.composer.environment.executor.running_tasks
(gauge)		Number of running tasks on executor
Shown as task
gcp.composer.environment.finished_task_instance_count
(count)		Overall number of finished task instances
Shown as instance
gcp.composer.environment.healthy
(gauge)		Healthiness of Composer environment.
gcp.composer.environment.num_celery_workers
(gauge)		Number of Celery workers.
Shown as worker
gcp.composer.environment.num_workflows
(gauge)		Number of workflows.
gcp.composer.environment.scheduler_heartbeat_count
(count)		Scheduler heartbeats
gcp.composer.environment.task_queue_length
(gauge)		Number of tasks in queue.
Shown as task
gcp.composer.environment.unfinished_task_instances
(gauge)		Overall task instances in not finished state
Shown as instance
gcp.composer.environment.web_server.cpu.reserved_cores
(gauge)		Number of cores reserved for the web server instance
Shown as core
gcp.composer.environment.web_server.cpu.usage_time
(count)		CPU usage time of the web server instance, in seconds
Shown as second
gcp.composer.environment.web_server.health
(gauge)		Healthiness of Airflow web server
gcp.composer.environment.web_server.memory.bytes_used
(gauge)		Memory usage of the web server instance in bytes
Shown as byte
gcp.composer.environment.web_server.memory.quota
(gauge)		Maximum RAM size of the web server instance, in bytes
Shown as byte
gcp.composer.environment.worker.max_workers
(gauge)		Maximum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.min_workers
(gauge)		Minimum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.pod_eviction_count
(count)		Number of Airflow worker pods evictions
Shown as eviction
gcp.composer.environment.worker.scale_factor_target
(gauge)		Scale factor for Airflow workers count
gcp.composer.environment.zombie_task_killed_count
(count)		Number of zombie tasks killed
Shown as task
gcp.composer.workflow.run_count
(count)		Number of workflow runs completed so far.
gcp.composer.workflow.run_duration
(gauge)		Duration of workflow run completion.
Shown as second
gcp.composer.workflow.task.run_count
(count)		Number of workflow tasks completed so far.
Shown as task
gcp.composer.workflow.task.run_duration
(gauge)		Duration of task completion.
Shown as second

이벤트

Google Cloud Composer 통합은 이벤트를 포함하지 않습니다.

서비스 점검

Google Cloud Composer 통합은 서비스 점검을 포함하지 않습니다.

트러블슈팅

도움이 필요하신가요? Datadog 지원팀에 문의하세요.