---
title: Google Cloud Managed Service for Apache Kafka
description: >-
  Monitor cluster, broker, topic, partition, and Kafka Connect metrics for
  Google Cloud Managed Service for Apache Kafka.
breadcrumbs: Docs > Integrations > Google Cloud Managed Service for Apache Kafka
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Google Cloud Managed Service for Apache Kafka
Integration version1.0.0
{% callout %}
# Important note for users on the following Datadog sites: us2.ddog-gov.com

{% alert level="info" %}
To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator.

To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

[Google Cloud Managed Service for Apache Kafka](https://cloud.google.com/products/managed-service-for-apache-kafka) is a fully managed service that lets you run Apache Kafka clusters on Google Cloud without having to manage the underlying infrastructure. It provides Kafka-compatible clusters, topics, and Kafka Connect for streaming data pipelines.

Use the Datadog Google Cloud Platform integration to collect metrics from Google Cloud Managed Service for Apache Kafka, including cluster, broker, topic, partition, consumer group, and Kafka Connect connector metrics.

## Setup{% #setup %}

### Installation{% #installation %}

If you haven't already, set up the [Google Cloud Platform integration](https://docs.datadoghq.com/integrations/google-cloud-platform.md) first. There are no other installation steps.

## Data Collected{% #data-collected %}

### Metrics{% #metrics %}

|  |
|  |
| **gcp.managedkafka.broker.under_min_isr_partitions**(gauge)              | Number of partitions whose in-sync replica (ISR) count is below min.insync.replicas. This comes from the MBean kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount.                                                                                  |
| **gcp.managedkafka.broker.under_replicated_partitions**(gauge)            | Number of under-replicated partitions, or the number of partitions that have replicas not in sync. This comes from the MBean kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions.                                                                    |
| **gcp.managedkafka.byte_in_count**(count)                                 | The total number of bytes from clients sent to the topic. This comes from the MBean kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=([-.\w]+).*Shown as byte*                                                                                      |
| **gcp.managedkafka.byte_out_count**(count)                                | The total number of bytes sent to clients. This comes from the MBean kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=([-.\w]+).*Shown as byte*                                                                                                    |
| **gcp.managedkafka.byte_size**(gauge)                                      | The size of the partition on disk in bytes. This comes from the MBean kafka.log:type=Log,name=Size,topic=([-.\w]+),partition=([0-9]+).*Shown as byte*                                                                                                            |
| **gcp.managedkafka.cluster_byte_in_count**(count)                        | The total number of bytes from clients sent to all topics. This comes from the MBean kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec.*Shown as byte*                                                                                                     |
| **gcp.managedkafka.cluster_byte_out_count**(count)                       | The total number of bytes sent to clients from all topics. This comes from the MBean kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec.*Shown as byte*                                                                                                    |
| **gcp.managedkafka.cluster_message_in_count**(count)                     | The total number of messages that have been published to all topics. This comes from the MBean kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec.                                                                                                       |
| **gcp.managedkafka.connector_deadletterqueue_record_count**(count)       | Total number of attempted or failed writes to the dead letter queue. This comes from the MBean kafka.connect<type=task-error-metrics, connector=(.+), task=(.+)><>(deadletterqueue-produce-failures                                                              |
| **gcp.managedkafka.connector_incoming_byte_rate**(gauge)                 | Average number of incoming bytes read per second from all sockets from client connection. This comes from the MBean kafka.connect<type=connect-metrics, client-id=(.+)><>incoming-byte-rate.*Shown as byte*                                                      |
| **gcp.managedkafka.connector_outgoing_byte_rate**(gauge)                 | Average number of outgoing bytes sent per second to all servers from client connection. This comes from the MBean kafka.connect<type=connect-metrics, client-id=(.+)><>outgoing-byte-rate.*Shown as byte*                                                        |
| **gcp.managedkafka.connector_request_rate**(gauge)                        | Average number of requests sent per second from client connection. This comes from the MBean kafka.connect<type=connect-metrics, client-id=(.+)><>request-rate.                                                                                                  |
| **gcp.managedkafka.connector_response_rate**(gauge)                       | Average number of responses received and sent per second from client connection. This comes from the MBean kafka.connect<type=connect-metrics, client-id=(.+)><>response-rate.                                                                                   |
| **gcp.managedkafka.connector_sink_active_records**(gauge)                | Most recent number of records read from Kafka but not completely committed by the sink task. This comes from the MBean kafka.connect<type=sink-task-metrics, connector=(.+), task=(.+)><>sink-record-active-count.                                               |
| **gcp.managedkafka.connector_sink_record_rate**(gauge)                   | Average record read or send rate from the sink connector. This comes from the MBean kafka.connect<type=sink-task-metrics, connector=(.+), task=(.+)><>(sink-record-read-rate                                                                                     |
| **gcp.managedkafka.connector_sink_record_read_total**(count)            | Total number of records read from Kafka by the sink task before transformations.                                                                                                                                                                                 |
| **gcp.managedkafka.connector_source_active_records**(gauge)              | Most recent number of records polled by the task but not completely written to Kafka. This comes from the MBean kafka.connect<type=source-task-metrics, connector=(.+), task=(.+)><>source-record-active-count.                                                  |
| **gcp.managedkafka.connector_source_record_poll_total**(count)          | Total number of records polled by the source task before transformations.                                                                                                                                                                                        |
| **gcp.managedkafka.connector_source_record_rate**(gauge)                 | Average record write or poll rate from source connector. This comes from the MBean kafka.connect<type=source-task-metrics, connector=(.+), task=(.+)><>(source-record-write-rate                                                                                 |
| **gcp.managedkafka.connector_tasks**(gauge)                                | Number of connector tasks in different states. This comes from the MBean kafka.connect<type=connect-worker-metrics, connector=(.+)><>(connector-destroyed-task-count                                                                                             |
| **gcp.managedkafka.connector_total_error_record_count**(count)          | Total number of task record errors. This comes from the MBean kafka.connect<type=task-error-metrics, connector=(.+), task=(.+)><>(total-record-errors                                                                                                            |
| **gcp.managedkafka.connector_total_retries_count**(count)                | Total number of connector task retries. This comes from the MBean kafka.connect<type=task-error-metrics, connector=(.+), task=(.+)><>total-retries.                                                                                                              |
| **gcp.managedkafka.consumer_groups**(gauge)                                | The current number of consumer groups consuming from the broker. This comes from the MBean kafka.server:type=GroupMetadataManager,name=NumGroups.                                                                                                                |
| **gcp.managedkafka.consumer_lag**(gauge)                                   | Lag in messages per follower replica. This comes from the MBean kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+).                                                                                      |
| **gcp.managedkafka.cpu.core_usage_time**(count)                           | Total CPU usage of the broker in vCPU seconds.*Shown as second*                                                                                                                                                                                                  |
| **gcp.managedkafka.cpu.limit**(gauge)                                       | Current CPU limit of the broker in vCPUs.                                                                                                                                                                                                                        |
| **gcp.managedkafka.cpu.worker_core_usage_time**(count)                   | Total CPU usage of the Kafka Connect worker in vCPU seconds.*Shown as second*                                                                                                                                                                                    |
| **gcp.managedkafka.disk.limit**(gauge)                                      | The disk limit of the broker in bytes.*Shown as byte*                                                                                                                                                                                                            |
| **gcp.managedkafka.disk.used_bytes**(gauge)                                | Used bytes of the broker's disk.*Shown as byte*                                                                                                                                                                                                                  |
| **gcp.managedkafka.first_offset**(gauge)                                   | The first offset in the partition. This comes from the MBean kafka.log:type=Log,name=LogStartOffset,topic=([-.\w]+),partition=([0-9]+).                                                                                                                          |
| **gcp.managedkafka.jvm.memory_area.bytes_committed**(gauge)               | Committed bytes of a given JVM memory area.*Shown as byte*                                                                                                                                                                                                       |
| **gcp.managedkafka.jvm.memory_area.bytes_used**(gauge)                    | Used bytes of a given JVM memory area.*Shown as byte*                                                                                                                                                                                                            |
| **gcp.managedkafka.last_offset**(gauge)                                    | The last offset in the partition. This comes from the MBean kafka.log:type=Log,name=LogEndOffset,topic=([-.\w]+),partition=([0-9]+).                                                                                                                             |
| **gcp.managedkafka.log_segments**(gauge)                                   | The current number of log segments. This comes from the MBean kafka.log:type=Log,name=NumLogSegments,topic=([-.\w]+),partition=([0-9]+).                                                                                                                         |
| **gcp.managedkafka.memory.limit**(gauge)                                    | The memory limit of the broker in bytes.*Shown as byte*                                                                                                                                                                                                          |
| **gcp.managedkafka.memory.usage**(gauge)                                    | The memory usage of the broker in bytes.*Shown as byte*                                                                                                                                                                                                          |
| **gcp.managedkafka.memory.worker_usage**(gauge)                            | The memory usage of the Kafka Connect worker in bytes.*Shown as byte*                                                                                                                                                                                            |
| **gcp.managedkafka.message_in_count**(count)                              | The total number of messages that have been published to the topic. This comes from the MBean kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=([-.\w]+).                                                                                        |
| **gcp.managedkafka.mirror_source_connector_byte_rate**(gauge)           | The average number of bytes per second in replicated records. This comes from the MBean kafka.connect.mirror<type=MirrorSourceConnector, target=(.+), topic=(.+), partition=(.+)><>byte-rate.*Shown as byte*                                                     |
| **gcp.managedkafka.mirror_source_connector_bytes**(gauge)                | The number of bytes in replicated records. This comes from the MBean kafka.connect.mirror<type=MirrorSourceConnector, target=(.+), topic=(.+), partition=(.+)><>byte-count.*Shown as byte*                                                                       |
| **gcp.managedkafka.mirror_source_connector_record_age**(gauge)          | The age of a record when it is replicated. This comes from the MBean kafka.connect.mirror<type=MirrorSourceConnector, target=(.+), topic=(.+), partition=(.+)><>record-age-ms.*Shown as millisecond*                                                             |
| **gcp.managedkafka.mirror_source_connector_record_rate**(gauge)         | The rate of records replicated from source to target cluster. This comes from the MBean kafka.connect.mirror<type=MirrorSourceConnector, target=(.+), topic=(.+), partition=(.+)><>record-rate.                                                                  |
| **gcp.managedkafka.mirror_source_connector_records**(gauge)              | The number of records replicated from source to target cluster. This comes from the MBean kafka.connect.mirror<type=MirrorSourceConnector, target=(.+), topic=(.+), partition=(.+)><>record-count.                                                               |
| **gcp.managedkafka.mirror_source_connector_replication_latency**(gauge) | The time it takes a record to propagate from source to target. This comes from the MBean kafka.connect.mirror<type=MirrorSourceConnector, target=(.+), topic=(.+), partition=(.+)><>replication-latency-ms.*Shown as millisecond*                                |
| **gcp.managedkafka.mtls_truststore_update_count**(count)                 | A cumulative count of mTLS truststore update attempts for the Kafka cluster, categorized by update status. This metric increments each time an mTLS truststore update attempt concludes.                                                                         |
| **gcp.managedkafka.offline_partitions**(gauge)                             | The number of offline topic partitions as observed by the controller. This comes from the MBean kafka.controller:type=KafkaController,name=OfflinePartitionsCount.                                                                                               |
| **gcp.managedkafka.offset_lag**(gauge)                                     | The number of messages that the consumer group has not yet committed on the partition.                                                                                                                                                                           |
| **gcp.managedkafka.partitions**(gauge)                                      | The current number of partitions handled by this broker. This comes from the MBean kafka.server:type=ReplicaManager,name=PartitionCount.                                                                                                                         |
| **gcp.managedkafka.request_byte_count**(count)                            | The total size, in bytes, of requests made to the cluster. This comes from the MBean kafka.network:type=RequestMetrics,name=RequestBytes,request=([-.\w]+).*Shown as byte*                                                                                       |
| **gcp.managedkafka.request_count**(count)                                  | The total number of requests made to the broker. This comes from the MBean kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce                                                                                                                |
| **gcp.managedkafka.request_latencies**(gauge)                              | The number of milliseconds taken for each request, at various percentiles. This comes from the MBean kafka.network:type=RequestMetrics,name=TotalTimeMs,request={Produce                                                                                         |
| **gcp.managedkafka.storage.long_term_usage**(gauge)                       | The long-term storage usage for the cluster.*Shown as byte*                                                                                                                                                                                                      |
| **gcp.managedkafka.topic_error_count**(count)                             | The total number of failed requests made to the topic. This comes from the MBeans kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec,topic=([-.\w]+) and kafka.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec,topic=([-.\w]+). |
| **gcp.managedkafka.topic_request_count**(count)                           | The total number of requests made to the topic. This comes from the MBeans kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=([-.\w]+) and kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=([-.\w]+).          |

### Events{% #events %}

The Google Cloud Managed Service for Apache Kafka integration does not include any events.

### Service Checks{% #service-checks %}

The Google Cloud Managed Service for Apache Kafka integration does not include any service checks.

## Troubleshooting{% #troubleshooting %}

Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).