Google Cloud Dataflow

Documentos > Integraciones > Google Cloud Dataflow

Información general

Google Cloud Dataflow es un servicio totalmente gestionado para transformar y enriquecer datos en los modos flujo (stream) (en tiempo real) y batch (histórico) con la misma fiabilidad y expresividad.

Utiliza la integración de Google Cloud con Datadog para recopilar métricas de Google Cloud Dataflow.

Configuración

Recopilación de métricas

Instalación

Si aún no lo has hecho, primero configura la integración Google Cloud Platform. No es necesario realizar ningún otro paso de instalación.

APM

Los logs de Google Cloud Dataflow se recopilan con Google Cloud Logging y se envían a una tarea de Dataflow a través de un tema Cloud Pub/Sub. Si aún no lo has hecho, configura la generación de logs con la plantilla Dataflow de Datadog.

Una vez hecho esto, exporta tus logs de Google Cloud Dataflow desde Google Cloud Logging a Pub/Sub:

Ve a la página de Google Cloud Logging y filtra logs de Google Cloud Dataflow.
Haz clic en Create sink (Crear sumidero) y asigna al sumidero el nombre correspondiente.
Elige “Cloud Pub/Sub” como destino y selecciona el tema Pub/Sub creado para tal fin. Nota: El tema Pub/Sub puede encontrarse en un proyecto diferente.
Haz clic en Create (Crear) y espera a que aparezca el mensaje de confirmación.

Datos recopilados

Métricas

gcp.dataflow.job.billable_shuffle_data_processed (gauge)	The billable bytes of shuffle data processed by this Dataflow job. Shown as byte
gcp.dataflow.job.current_num_vcpus (gauge)	The number of vCPUs currently being used by this Dataflow job. Shown as cpu
gcp.dataflow.job.current_shuffle_slots (gauge)	The current shuffle slots used by this Dataflow job.
gcp.dataflow.job.data_watermark_age (gauge)	The age (time since event timestamp) of the most recent item of data that has been fully processed by the pipeline. Shown as second
gcp.dataflow.job.elapsed_time (gauge)	Duration that the current run of this pipeline has been in the Running state so far, in seconds. When a run completes, this stays at the duration of that run until the next run starts. Shown as second
gcp.dataflow.job.element_count (count)	Number of elements added to the pcollection so far. Shown as item
gcp.dataflow.job.estimated_byte_count (count)	An estimated number of bytes added to the pcollection so far. Shown as byte
gcp.dataflow.job.is_failed (gauge)	Has this job failed.
gcp.dataflow.job.system_lag (gauge)	The current maximum duration that an item of data has been awaiting processing, in seconds. Shown as second
gcp.dataflow.job.total_memory_usage_time (gauge)	The total GB seconds of memory allocated to this Dataflow job. Shown as gibibyte
gcp.dataflow.job.total_pd_usage_time (gauge)	The total GB seconds for all persistent disk used by all workers associated with this Dataflow job. Shown as gibibyte
gcp.dataflow.job.total_shuffle_data_processed (gauge)	The total bytes of shuffle data processed by this Dataflow job. Shown as byte
gcp.dataflow.job.total_streaming_data_processed (gauge)	The total bytes of streaming data processed by this Dataflow job. Shown as byte
gcp.dataflow.job.total_vcpu_time (gauge)	The total vCPU seconds used by this Dataflow job.
gcp.dataflow.job.user_counter (gauge)	A user-defined counter metric.
gcp.dataflow.quota.region_endpoint_shuffle_slot.exceeded (count)	Number of attempts to exceed the limit on quota metric dataflow.googleapis.com/regionendpointshuffle_slot.
gcp.dataflow.quota.region_endpoint_shuffle_slot.limit (gauge)	Current limit on quota metric dataflow.googleapis.com/regionendpointshuffle_slot.
gcp.dataflow.quota.region_endpoint_shuffle_slot.usage (gauge)	Current usage on quota metric dataflow.googleapis.com/regionendpointshuffle_slot.

Al utilizar Google Cloud Dataflow para monitorizar métricas de pipelines Apache Beam, ten en cuenta que las métricas generadas a partir de métodos Gauge estáticos no se recopilan. Si necesitas monitorizar estas métricas, puedes utilizar Micrometer.