概要

Google Cloud Composer は、複数のクラウドやオンプレミスデータセンターにまたがるパイプラインの作成、スケジューリング、監視を可能にする、フルマネージド型のワークフローオーケストレーションサービスです。

Datadog Google Cloud Platform インテグレーションを使用して、Google Cloud Composer からメトリクスを収集できます。

セットアップ

インストール

Google Cloud Platform インテグレーションをまだセットアップしていない場合は、最初にセットアップします。それ以上のインストール手順はありません。

収集データ

Google Cloud Composer のログは Google Cloud Logging で収集され、Cloud Pub/Sub トピックを通じて Dataflow ジョブに送信されます。まだの場合は、Datadog Dataflow テンプレートでロギングをセットアップしてください

これが完了したら、Google Cloud Composer のログを Google Cloud Logging から Pub/Sub トピックへエクスポートします。

  1. Google Cloud Logging のページに移動し、Google Cloud Composer のログを絞り込みます。
  2. Create Export をクリックし、シンクに名前を付けます。
  3. 宛先として “Cloud Pub/Sub” を選択し、その目的で作成された Pub/Sub トピックを選択します。: Pub/Sub トピックは別のプロジェクトに配置できます。
  4. 作成をクリックし、確認メッセージが表示されるまで待ちます。

収集データ

メトリクス

gcp.composer.environment.api.request_count
(count)
Number of Composer API requests seen so far.
Shown as request
gcp.composer.environment.api.request_latencies.avg
(gauge)
Distribution of Composer API call latencies.
Shown as millisecond
gcp.composer.environment.api.request_latencies.samplecount
(count)
Sample count for API request latencies
Shown as millisecond
gcp.composer.environment.api.request_latencies.sumsqdev
(gauge)
Sum of squared deviation for API request latencies
Shown as second
gcp.composer.environment.dag_processing.parse_error_count
(count)
Number of errors raised during parsing DAG files
Shown as error
gcp.composer.environment.dag_processing.processes
(gauge)
Number of currently running DAG parsing processes
Shown as process
gcp.composer.environment.dag_processing.processesor_timeout_count
(count)
Number of file processors terminated due to processing timeout.
Shown as timeout
gcp.composer.environment.dag_processing.total_parse_time
(gauge)
Number of seconds taken to scan and import all DAG files once
Shown as second
gcp.composer.environment.dagbag_size
(gauge)
The current DAG bag size
gcp.composer.environment.database.cpu.reserved_cores
(gauge)
Number of cores reserved for the database instance
Shown as core
gcp.composer.environment.database.cpu.usage_time
(count)
CPU usage time of the database instance, in seconds
Shown as second
gcp.composer.environment.database.cpu.utilization
(gauge)
CPU utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.disk.bytes_used
(gauge)
Used disk space on the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.quota
(gauge)
Maximum data disk size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.utilization
(gauge)
Disk quota usage ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.memory.bytes_used
(gauge)
Memory usage of the database instance in bytes
Shown as byte
gcp.composer.environment.database.memory.quota
(gauge)
Maximum RAM size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.memory.utilization
(gauge)
Memory utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.network.received_bytes_count
(count)
Number of bytes received by the database instance
Shown as byte
gcp.composer.environment.database.network.sent_bytes_count
(count)
Number of bytes sent by the database instance
Shown as byte
gcp.composer.environment.database_health
(gauge)
Healthiness of Composer Airflow database
gcp.composer.environment.executor.open_slots
(gauge)
Number of open slots on executor
gcp.composer.environment.executor.queued_tasks
(gauge)
Number of queued tasks on executor
Shown as task
gcp.composer.environment.executor.running_tasks
(gauge)
Number of running tasks on executor
Shown as task
gcp.composer.environment.finished_task_instance_count
(count)
Overall number of finished task instances
Shown as instance
gcp.composer.environment.healthy
(gauge)
Healthiness of Composer environment.
gcp.composer.environment.num_celery_workers
(gauge)
Number of Celery workers.
Shown as worker
gcp.composer.environment.num_workflows
(gauge)
Number of workflows.
gcp.composer.environment.scheduler_heartbeat_count
(count)
Scheduler heartbeats
gcp.composer.environment.task_queue_length
(gauge)
Number of tasks in queue.
Shown as task
gcp.composer.environment.unfinished_task_instances
(gauge)
Overall task instances in not finished state
Shown as instance
gcp.composer.environment.web_server.cpu.reserved_cores
(gauge)
Number of cores reserved for the web server instance
Shown as core
gcp.composer.environment.web_server.cpu.usage_time
(count)
CPU usage time of the web server instance, in seconds
Shown as second
gcp.composer.environment.web_server.health
(gauge)
Healthiness of Airflow web server
gcp.composer.environment.web_server.memory.bytes_used
(gauge)
Memory usage of the web server instance in bytes
Shown as byte
gcp.composer.environment.web_server.memory.quota
(gauge)
Maximum RAM size of the web server instance, in bytes
Shown as byte
gcp.composer.environment.worker.max_workers
(gauge)
Maximum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.min_workers
(gauge)
Minimum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.pod_eviction_count
(count)
Number of Airflow worker pods evictions
Shown as eviction
gcp.composer.environment.worker.scale_factor_target
(gauge)
Scale factor for Airflow workers count
gcp.composer.environment.zombie_task_killed_count
(count)
Number of zombie tasks killed
Shown as task
gcp.composer.workflow.run_count
(count)
Number of workflow runs completed so far.
gcp.composer.workflow.run_duration
(gauge)
Duration of workflow run completion.
Shown as second
gcp.composer.workflow.task.run_count
(count)
Number of workflow tasks completed so far.
Shown as task
gcp.composer.workflow.task.run_duration
(gauge)
Duration of task completion.
Shown as second

イベント

Google Cloud Composer インテグレーションには、イベントは含まれません。

サービスチェック

Google Cloud Composer インテグレーションには、サービスのチェック機能は含まれません。

トラブルシューティング

ご不明な点は、Datadog のサポートチームまでお問い合わせください。