Google Cloud Composer

Información general

Google Cloud Composer es un servicio de orquestación de flujos de trabajo totalmente gestionado que te permite crear, programar y monitorizar pipelines a través de centros de datos on-premise y en la nube.

Utiliza la integración de Google Cloud Platform con Datadog para recopilar métricas de Google Cloud Composer.

Configuración

Instalación

Si aún no lo has hecho, primero configura la integración Google Cloud Platform. No es necesario realizar ningún otro paso de instalación.

APM

Los logs de Google Cloud Composer se recopilan con Google Cloud Logging y se envían a una tarea de Dataflow a través de un tema Cloud Pub/Sub. Si aún no lo has hecho, configura la generación de logs con la plantilla Dataflow de Datadog.

Una vez hecho esto, exporta tus logs de Google Cloud Composer de Google Cloud Logging al tema Pub/Sub:

  1. Ve a la página de Google Cloud Logging y filtra logs de Google Cloud Composer.
  2. Haz clic en Create Export (Crear exportación) y asigna un nombre al sumidero.
  3. Elige “Cloud Pub/Sub” como destino y selecciona el tema Pub/Sub creado para tal fin. Nota: El tema Pub/Sub puede encontrarse en un proyecto diferente.
  4. Haz clic en Create (Crear) y espera a que aparezca el mensaje de confirmación.

Datos recopilados

Métricas

gcp.composer.environment.api.request_count
(count)
Number of Composer API requests seen so far.
Shown as request
gcp.composer.environment.api.request_latencies.avg
(gauge)
Distribution of Composer API call latencies.
Shown as millisecond
gcp.composer.environment.api.request_latencies.samplecount
(count)
Sample count for API request latencies
Shown as millisecond
gcp.composer.environment.api.request_latencies.sumsqdev
(gauge)
Sum of squared deviation for API request latencies
Shown as second
gcp.composer.environment.dag_processing.parse_error_count
(count)
Number of errors raised during parsing DAG files
Shown as error
gcp.composer.environment.dag_processing.processes
(gauge)
Number of currently running DAG parsing processes
Shown as process
gcp.composer.environment.dag_processing.processesor_timeout_count
(count)
Number of file processors terminated due to processing timeout.
Shown as timeout
gcp.composer.environment.dag_processing.total_parse_time
(gauge)
Number of seconds taken to scan and import all DAG files once
Shown as second
gcp.composer.environment.dagbag_size
(gauge)
The current DAG bag size
gcp.composer.environment.database.cpu.reserved_cores
(gauge)
Number of cores reserved for the database instance
Shown as core
gcp.composer.environment.database.cpu.usage_time
(count)
CPU usage time of the database instance, in seconds
Shown as second
gcp.composer.environment.database.cpu.utilization
(gauge)
CPU utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.disk.bytes_used
(gauge)
Used disk space on the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.quota
(gauge)
Maximum data disk size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.utilization
(gauge)
Disk quota usage ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.memory.bytes_used
(gauge)
Memory usage of the database instance in bytes
Shown as byte
gcp.composer.environment.database.memory.quota
(gauge)
Maximum RAM size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.memory.utilization
(gauge)
Memory utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.network.received_bytes_count
(count)
Number of bytes received by the database instance
Shown as byte
gcp.composer.environment.database.network.sent_bytes_count
(count)
Number of bytes sent by the database instance
Shown as byte
gcp.composer.environment.database_health
(gauge)
Healthiness of Composer Airflow database
gcp.composer.environment.executor.open_slots
(gauge)
Number of open slots on executor
gcp.composer.environment.executor.queued_tasks
(gauge)
Number of queued tasks on executor
Shown as task
gcp.composer.environment.executor.running_tasks
(gauge)
Number of running tasks on executor
Shown as task
gcp.composer.environment.finished_task_instance_count
(count)
Overall number of finished task instances
Shown as instance
gcp.composer.environment.healthy
(gauge)
Healthiness of Composer environment.
gcp.composer.environment.num_celery_workers
(gauge)
Number of Celery workers.
Shown as worker
gcp.composer.environment.num_workflows
(gauge)
Number of workflows.
gcp.composer.environment.scheduler_heartbeat_count
(count)
Scheduler heartbeats
gcp.composer.environment.task_queue_length
(gauge)
Number of tasks in queue.
Shown as task
gcp.composer.environment.unfinished_task_instances
(gauge)
Overall task instances in not finished state
Shown as instance
gcp.composer.environment.web_server.cpu.reserved_cores
(gauge)
Number of cores reserved for the web server instance
Shown as core
gcp.composer.environment.web_server.cpu.usage_time
(count)
CPU usage time of the web server instance, in seconds
Shown as second
gcp.composer.environment.web_server.health
(gauge)
Healthiness of Airflow web server
gcp.composer.environment.web_server.memory.bytes_used
(gauge)
Memory usage of the web server instance in bytes
Shown as byte
gcp.composer.environment.web_server.memory.quota
(gauge)
Maximum RAM size of the web server instance, in bytes
Shown as byte
gcp.composer.environment.worker.max_workers
(gauge)
Maximum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.min_workers
(gauge)
Minimum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.pod_eviction_count
(count)
Number of Airflow worker pods evictions
Shown as eviction
gcp.composer.environment.worker.scale_factor_target
(gauge)
Scale factor for Airflow workers count
gcp.composer.environment.zombie_task_killed_count
(count)
Number of zombie tasks killed
Shown as task
gcp.composer.workflow.run_count
(count)
Number of workflow runs completed so far.
gcp.composer.workflow.run_duration
(gauge)
Duration of workflow run completion.
Shown as second
gcp.composer.workflow.task.run_count
(count)
Number of workflow tasks completed so far.
Shown as task
gcp.composer.workflow.task.run_duration
(gauge)
Duration of task completion.
Shown as second

Eventos

La integración Google Cloud Composer no incluye eventos.

Checks de servicio

La integración Google Cloud Composer no incluye checks de servicio.

Resolución de problemas

¿Necesitas ayuda? Ponte en contacto con el servicio de asistencia de Datadog.