Google Cloud Composer

Crawler Crawler

概要

Google Cloud Composer は、複数のクラウドやオンプレミスデータセンターにまたがるパイプラインの作成、スケジューリング、監視を可能にする、フルマネージド型のワークフローオーケストレーションサービスです。

Datadog Google Cloud Platform インテグレーションを使用して、Google Cloud Composer からメトリクスを収集できます。

セットアップ

インストール

Google Cloud Platform インテグレーションをまだセットアップしていない場合は、最初にセットアップします。それ以上のインストール手順はありません。

ログの収集

Google Cloud Composer のログは Google Cloud Logging により収集され、HTTP プッシュフォワーダーを使用して Cloud Pub/Sub へ送信されます。HTTP プッシュフォワーダーを使用した Cloud Pub/Sub をまだセットアップしていない場合は、これをセットアップしてください。

これが完了したら、Google Cloud Composer のログを Google Cloud Logging から Pub/Sub へエクスポートします。

  1. Google Cloud Logging のページに移動し、Google Cloud Composer のログを絞り込みます。
  2. Create Export をクリックし、シンクに名前を付けます。
  3. エクスポート先として「Cloud Pub/Sub」を選択し、エクスポート用に作成された Pub/Sub を選択します。: この Pub/Sub は別のプロジェクト内に配置することもできます。
  4. 作成をクリックし、確認メッセージが表示されるまで待ちます。

収集データ

メトリクス

gcp.composer.environment.api.request_count
(count)
Number of Composer API requests seen so far.
Shown as request
gcp.composer.environment.api.request_latencies.avg
(gauge)
Distribution of Composer API call latencies.
Shown as millisecond
gcp.composer.environment.api.request_latencies.samplecount
(count)
Sample count for API request latencies
Shown as millisecond
gcp.composer.environment.api.request_latencies.sumsqdev
(gauge)
Sum of squared deviation for API request latencies
Shown as second
gcp.composer.environment.dag_processing.parse_error_count
(count)
Number of errors raised during parsing DAG files
Shown as error
gcp.composer.environment.dag_processing.processes
(gauge)
Number of currently running DAG parsing processes
Shown as process
gcp.composer.environment.dag_processing.processesor_timeout_count
(count)
Number of file processors terminated due to processing timeout.
Shown as timeout
gcp.composer.environment.dag_processing.total_parse_time
(gauge)
Number of seconds taken to scan and import all DAG files once
Shown as second
gcp.composer.environment.dagbag_size
(gauge)
The current DAG bag size
gcp.composer.environment.database.cpu.reserved_cores
(gauge)
Number of cores reserved for the database instance
Shown as core
gcp.composer.environment.database.cpu.usage_time
(count)
CPU usage time of the database instance, in seconds
Shown as second
gcp.composer.environment.database.cpu.utilization
(gauge)
CPU utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.disk.bytes_used
(gauge)
Used disk space on the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.quota
(gauge)
Maximum data disk size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.utilization
(gauge)
Disk quota usage ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.memory.bytes_used
(gauge)
Memory usage of the database instance in bytes
Shown as byte
gcp.composer.environment.database.memory.quota
(gauge)
Maximum RAM size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.memory.utilization
(gauge)
Memory utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.network.received_bytes_count
(count)
Number of bytes received by the database instance
Shown as byte
gcp.composer.environment.database.network.sent_bytes_count
(count)
Number of bytes sent by the database instance
Shown as byte
gcp.composer.environment.database_health
(gauge)
Healthiness of Composer Airflow database
gcp.composer.environment.executor.open_slots
(gauge)
Number of open slots on executor
gcp.composer.environment.executor.queued_tasks
(gauge)
Number of queued tasks on executor
Shown as task
gcp.composer.environment.executor.running_tasks
(gauge)
Number of running tasks on executor
Shown as task
gcp.composer.environment.finished_task_instance_count
(count)
Overall number of finished task instances
Shown as instance
gcp.composer.environment.healthy
(gauge)
Healthiness of Composer environment.
gcp.composer.environment.num_celery_workers
(gauge)
Number of Celery workers.
Shown as worker
gcp.composer.environment.num_workflows
(gauge)
Number of workflows.
gcp.composer.environment.scheduler_heartbeat_count
(count)
Scheduler heartbeats
gcp.composer.environment.task_queue_length
(gauge)
Number of tasks in queue.
Shown as task
gcp.composer.environment.unfinished_task_instances
(gauge)
Overall task instances in not finished state
Shown as instance
gcp.composer.environment.web_server.cpu.reserved_cores
(gauge)
Number of cores reserved for the web server instance
Shown as core
gcp.composer.environment.web_server.cpu.usage_time
(count)
CPU usage time of the web server instance, in seconds
Shown as second
gcp.composer.environment.web_server.health
(gauge)
Healthiness of Airflow web server
gcp.composer.environment.web_server.memory.bytes_used
(gauge)
Memory usage of the web server instance in bytes
Shown as byte
gcp.composer.environment.web_server.memory.quota
(gauge)
Maximum RAM size of the web server instance, in bytes
Shown as byte
gcp.composer.environment.worker.max_workers
(gauge)
Maximum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.min_workers
(gauge)
Minimum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.pod_eviction_count
(count)
Number of Airflow worker pods evictions
Shown as eviction
gcp.composer.environment.worker.scale_factor_target
(gauge)
Scale factor for Airflow workers count
gcp.composer.environment.zombie_task_killed_count
(count)
Number of zombie tasks killed
Shown as task
gcp.composer.workflow.run_count
(count)
Number of workflow runs completed so far.
gcp.composer.workflow.run_duration
(gauge)
Duration of workflow run completion.
Shown as second
gcp.composer.workflow.task.run_count
(count)
Number of workflow tasks completed so far.
Shown as task
gcp.composer.workflow.task.run_duration
(gauge)
Duration of task completion.
Shown as second

イベント

Google Cloud Composer インテグレーションには、イベントは含まれません。

サービスのチェック

Google Cloud Composer インテグレーションには、サービスのチェック機能は含まれません。

トラブルシューティング

ご不明な点は、Datadog のサポートチームまでお問合せください。