Google Cloud TPU

To find out if this integration is available in your organization, see your Datadog Integrations page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email support@ddog-gov.com.

Présentation

Avec Google Cloud TPU, tous les chercheurs, ingénieurs et développeurs en ML, ainsi que tous les data scientists exécutant des modèles de ML novateurs, peuvent profiter des avantages offerts par les Tensor Processing Units (TPU) grâce à des ressources de cloud computing évolutives et faciles à utiliser.

Utilisez l’intégration Datadog/Google Cloud Platform pour recueillir des métriques de Google Cloud TPU.

Configuration

Installation

Si vous ne l’avez pas déjà fait, configurez d’abord l’intégration Google Cloud Platform. Aucune autre procédure d’installation n’est requise.

Collecte de logs

Les logs Google Cloud TPU sont recueillis avec Google Cloud Logging et envoyés à un Cloud Pub/Sub via un forwarder Push HTTP. Si vous ne l’avez pas déjà fait, configurez un Cloud Pub/Sub à l’aide d’un forwarder Push HTTP.

Une fois cette opération effectuée, exportez vos logs Google Cloud TPU depuis Google Cloud Logging vers le Pub/Sub :

  1. Accédez à la page Google Cloud Logging et filtrez les logs Google Cloud TPU.
  2. Cliquez sur Create Export et nommez le récepteur.
  3. Choisissez Cloud Pub/Sub comme destination et sélectionnez le Pub/Sub créé à cette fin. Remarque : le Pub/Sub peut se situer dans un autre projet.
  4. Cliquez sur Create et attendez que le message de confirmation s’affiche.

Données collectées

Métriques

gcp.tpu.cpu.utilization
(gauge)
Current CPU utilization on the TPU worker, represented as a percentage. Values are typically numbers between 0.0 and 100.0, but might exceed 100.0.
Shown as percent
gcp.tpu.memory.usage
(gauge)
Memory usage in bytes.
Shown as byte
gcp.tpu.network.received_bytes_count
(count)
Cumulative bytes of data this server has received over the network.
Shown as byte
gcp.tpu.network.sent_bytes_count
(count)
Cumulative bytes of data this server has sent over the network.
Shown as byte
gcp.tpu.accelerator.duty_cycle
(count)
Percentage of time over the sample period during which the accelerator was actively processing
Shown as percent
gcp.tpu.instance.uptime_total
(count)
Elapsed time since the VM was started, in seconds.
Shown as second
gcp.gke.node.accelerator.tensorcore_utilization
(count)
Current percentage of the Tensorcore that is utilized.
Shown as percent
gcp.gke.node.accelerator.duty_cycle
(count)
Percent of time over the past sample period (10s) during which the accelerator was actively processing.
Shown as percent
gcp.gke.node.accelerator.memory_used
(count)
Total accelerator memory allocated in bytes.
Shown as byte
gcp.gke.node.accelerator.memory_total
(count)
Total accelerator memory in bytes.
Shown as byte
gcp.gke.node.accelerator.memory_bandwidth_utilization
(count)
Current percentage of the accelerator memory bandwidth that is being used.
Shown as percent
gcp.gke.container.accelerator.tensorcore_utilization
(count)
Current percentage of the Tensorcore that is utilized.
Shown as percent
gcp.gke.container.accelerator.duty_cycle
(count)
Percent of time over the past sample period (10s) during which the accelerator was actively processing.
Shown as percent
gcp.gke.container.accelerator.memory_used
(count)
Total accelerator memory allocated in bytes.
Shown as byte
gcp.gke.container.accelerator.memory_total
(count)
Total accelerator memory in bytes.
Shown as byte
gcp.gke.container.accelerator.memory_bandwidth_utilization
(count)
Current percentage of the accelerator memory bandwidth that is being used.
Shown as percent
gcp.tpu.accelerator.memory_bandwidth_utilization
(gauge)
Current percentage of the accelerator memory bandwidth that is being used. Computed by dividing the memory bandwidth used over a sample period by the maximum supported bandwidth over the same sample period.
Shown as percent
gcp.tpu.accelerator.memory_total
(gauge)
Total accelerator memory currently allocated in bytes.
Shown as byte
gcp.tpu.accelerator.memory_used
(gauge)
Total accelerator memory currently used in bytes.
Shown as byte
gcp.tpu.accelerator.tensorcore_utilization
(gauge)
Current percentage of the Tensorcore that is utilized. Computed by dividing the Tensorcore operations that were performed over a sample period by the supported number of Tensorcore operations over the same sample period.
Shown as percent
gcp.tpu.multislice.accelerator.device_to_host_transfer_latencies.avg
(gauge)
Cumulative distribution of device to host transfer latency for each chunk of data. A latency starts when the request for data to be transferred to the host is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond
gcp.tpu.multislice.accelerator.host_to_device_transfer_latencies.avg
(gauge)
Cumulative distribution of host to device transfer latency for each chunk of data of multislice traffic. A latency starts when the request for data to be transferred to the device is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond
gcp.tpu.multislice.network.collective_end_to_end_latencies.avg
(gauge)
Cumulative distribution of end to end collective latency for multislice traffic. A latency starts when the request for the collective is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond
gcp.tpu.multislice.network.dcn_transfer_latencies.avg
(gauge)
Cumulative distribution of network-transfer latencies for multislice traffic. A latency starts when the request for data to be transferred over the DCN is issued and ends when an acknowledgement is received that the transfer of data has completed.
Shown as microsecond

Événements

L’intégration Google Cloud TPU n’inclut aucun événement.

Checks de service

L’intégration Google Cloud TPU n’inclut aucun check de service.

Dépannage

Besoin d’aide ? Contactez l’assistance Datadog.