Network Performance Monitoring is now generally available! Network Monitoring is now available!

Google Cloud Composer

Crawler Crawler

Overview

Google Cloud Composer is a fully managed workflow orchestration service that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.

Use the Datadog Google Cloud Platform integration to collect metrics from Google Cloud Composer.

Setup

Installation

If you haven’t already, set up the Google Cloud Platform integration first. There are no other installation steps.

Log collection

Google Cloud Composer logs are collected with Stackdriver and sent to a Cloud pub/sub with an HTTP push forwarder. If you haven’t already, set up a Cloud pub/sub with an HTTP push forwarder.

Once this is done, export your Google Cloud Composer logs from Stackdriver to the pub/sub:

  1. Go to the Stackdriver page and filter the Google Cloud Composer logs.
  2. Click Create Export and name the sink.
  3. Choose “Cloud Pub/Sub” as the destination and select the pub/sub that was created for that purpose. Note: The pub/sub can be located in a different project.
  4. Click Create and wait for the confirmation message to show up.

Data Collected

Metrics

gcp.composer.environment.api.request_count
(count)
Number of Composer API requests seen so far.
Shown as request
gcp.composer.environment.api.request_latencies.avg
(gauge)
Distribution of Composer API call latencies.
Shown as millisecond
gcp.composer.environment.api.request_latencies.samplecount
(count)
Sample count for API request latencies
Shown as millisecond
gcp.composer.environment.api.request_latencies.sumsqdev
(gauge)
Sum of squared deviation for API request latencies
Shown as second
gcp.composer.environment.healthy
(gauge)
Healthiness of Composer environment.
gcp.composer.environment.num_celery_workers
(gauge)
Number of Celery workers.
Shown as worker
gcp.composer.environment.num_workflows
(gauge)
Number of workflows.
gcp.composer.environment.task_queue_length
(gauge)
Number of tasks in queue.
Shown as task
gcp.composer.workflow.run_count
(count)
Number of workflow runs completed so far.
gcp.composer.workflow.run_duration
(gauge)
Duration of workflow run completion.
Shown as second
gcp.composer.workflow.task.run_count
(count)
Number of workflow tasks completed so far.
Shown as task
gcp.composer.workflow.task.run_duration
(gauge)
Duration of task completion.
Shown as second

Events

The Google Cloud Composer integration does not include any events.

Service Checks

The Google Cloud Composer integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.