Google Cloud TPU

To find out if this integration is available in your organization, see your Datadog Integrations page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email support@ddog-gov.com.

Información general

Los productos Google Cloud TPU ponen las ventajas de las Tensor Processing Units (TPU) a disposición de todos los investigadores de ML, ingenieros de ML, desarrolladores y científicos de datos que ejecutan modelos de ML de última generación, a través de un recurso informático en la nube escalable y fácil de utilizar.

Utiliza la integración de Google Cloud Platform con Datadog para recopilar métricas de Google Cloud TPU.

Configuración

Instalación

Para utilizar Google Cloud TPU, sólo necesitas configurar la integración Google Cloud Platform.

Recopilación de logs

Los logs de integración se recopilan con Google Cloud Logging y se envían a una tarea de Dataflow a través de un tema Cloud Pub/Sub. Si aún no lo has hecho, configura la generación de logs con la plantilla Dataflow de Datadog.

Una vez hecho esto, exporta tus logs de Google Cloud TPU de Google Cloud Logging al tema Pub/Sub:

  1. Ve a la página de Google Cloud Logging y filtra logs de Google Cloud TPU.
  2. Haz clic en Create Export (Crear exportación) y asigna un nombre al sumidero.
  3. Elige “Cloud Pub/Sub” como destino y selecciona el tema Pub/Sub creado para tal fin. Nota: El tema Pub/Sub puede estar ubicado en un proyecto diferente.
  4. Haz clic en Crear y espera a que aparezca el mensaje de confirmación.

Recopilación de datos

Métricas

gcp.tpu.cpu.utilization
(gauge)		Current CPU utilization on the TPU worker, represented as a percentage. Values are typically numbers between 0.0 and 100.0, but might exceed 100.0.
Shown as percent
gcp.tpu.memory.usage
(gauge)		Memory usage in bytes.
Shown as byte
gcp.tpu.network.received_bytes_count
(count)		Cumulative bytes of data this server has received over the network.
Shown as byte
gcp.tpu.network.sent_bytes_count
(count)		Cumulative bytes of data this server has sent over the network.
Shown as byte
gcp.tpu.accelerator.duty_cycle
(count)		Percentage of time over the sample period during which the accelerator was actively processing
Shown as percent
gcp.tpu.instance.uptime_total
(count)		Elapsed time since the VM was started, in seconds.
Shown as second
gcp.gke.node.accelerator.tensorcore_utilization
(count)		Current percentage of the Tensorcore that is utilized.
Shown as percent
gcp.gke.node.accelerator.duty_cycle
(count)		Percent of time over the past sample period (10s) during which the accelerator was actively processing.
Shown as percent
gcp.gke.node.accelerator.memory_used
(count)		Total accelerator memory allocated in bytes.
Shown as byte
gcp.gke.node.accelerator.memory_total
(count)		Total accelerator memory in bytes.
Shown as byte
gcp.gke.node.accelerator.memory_bandwidth_utilization
(count)		Current percentage of the accelerator memory bandwidth that is being used.
Shown as percent
gcp.gke.container.accelerator.tensorcore_utilization
(count)		Current percentage of the Tensorcore that is utilized.
Shown as percent
gcp.gke.container.accelerator.duty_cycle
(count)		Percent of time over the past sample period (10s) during which the accelerator was actively processing.
Shown as percent
gcp.gke.container.accelerator.memory_used
(count)		Total accelerator memory allocated in bytes.
Shown as byte
gcp.gke.container.accelerator.memory_total
(count)		Total accelerator memory in bytes.
Shown as byte
gcp.gke.container.accelerator.memory_bandwidth_utilization
(count)		Current percentage of the accelerator memory bandwidth that is being used.
Shown as percent
gcp.tpu.accelerator.memory_bandwidth_utilization
(gauge)		Current percentage of the accelerator memory bandwidth that is being used. Computed by dividing the memory bandwidth used over a sample period by the maximum supported bandwidth over the same sample period.
Shown as percent
gcp.tpu.accelerator.memory_total
(gauge)		Total accelerator memory currently allocated in bytes.
Shown as byte
gcp.tpu.accelerator.memory_used
(gauge)		Total accelerator memory currently used in bytes.
Shown as byte
gcp.tpu.accelerator.tensorcore_utilization
(gauge)		Current percentage of the Tensorcore that is utilized. Computed by dividing the Tensorcore operations that were performed over a sample period by the supported number of Tensorcore operations over the same sample period.
Shown as percent
gcp.tpu.multislice.accelerator.device_to_host_transfer_latencies.avg
(gauge)		Cumulative distribution of device to host transfer latency for each chunk of data. A latency starts when the request for data to be transferred to the host is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond
gcp.tpu.multislice.accelerator.host_to_device_transfer_latencies.avg
(gauge)		Cumulative distribution of host to device transfer latency for each chunk of data of multislice traffic. A latency starts when the request for data to be transferred to the device is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond
gcp.tpu.multislice.network.collective_end_to_end_latencies.avg
(gauge)		Cumulative distribution of end to end collective latency for multislice traffic. A latency starts when the request for the collective is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond
gcp.tpu.multislice.network.dcn_transfer_latencies.avg
(gauge)		Cumulative distribution of network-transfer latencies for multislice traffic. A latency starts when the request for data to be transferred over the DCN is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond

Eventos

La integración Google Cloud TPU no incluye eventos.

Checks de servicios

La integración Google Cloud TPU no incluye checks de servicio.

Solucionar problemas

¿Necesitas ayuda? Ponte en contacto con el servicio de asistencia de Datadog.