Overview

This Agent integration collects message offset metrics from your Kafka consumers. This check fetches the highwater offsets from the Kafka brokers, consumer offsets that are stored in Kafka (or Zookeeper for old-style consumers), and then calculates consumer lag (which is the difference between the broker offset and the consumer offset).

Note:

• This integration ensures that consumer offsets are checked before broker offsets; in the worst case, consumer lag may be a little overstated. Checking these offsets in the reverse order can understate consumer lag to the point of having negative values, which is a dire scenario usually indicating messages are being skipped.
• If you want to collect JMX metrics from your Kafka brokers or Java-based consumers/producers, see the Kafka Broker integration.

Minimum Agent version: 6.0.0

Setup

Installation

The Agent’s Kafka consumer check is included in the Datadog Agent package. No additional installation is needed on your Kafka nodes.

Configuration

instances:
  - kafka_connect_str: <KAFKA_CONNECT_STR>
    consumer_groups:
      # Monitor all topics for consumer <CONSUMER_NAME>
      <CONSUMER_NAME>: {}

Cluster Monitoring (Preview)

When enable_cluster_monitoring is enabled, the integration collects cluster-wide metrics for Data Streams Monitoring in addition to consumer lag:

• Brokers: Configuration and health metrics.
• Topics and partitions: Sizes, offsets, and replication status.
• Consumer groups: Member details, group state, rebalance detection, membership-change counting, and metadata exposed as tags (partition_assignor, consumer_group_type, is_simple_consumer_group, and group_instance_id). Empty groups are visible through the consumer_group_state:EMPTY tag on kafka.consumer_group.members.
• Schema registry: Schema metadata (requires schema_registry_url).

Batched collection

Broker configurations, topic configurations, and schema registry version checks are collected in batches across multiple agent runs rather than all at once. This reduces load on large Kafka clusters but means that not all metrics are emitted in every check run. On a cluster with many brokers, topics, or schema subjects, the integration spreads the work over successive runs so that each run stays fast and does not overload the cluster.

Example configuration:

instances:
  - kafka_connect_str: localhost:9092
    enable_cluster_monitoring: true
    schema_registry_url: http://localhost:8081  # optional

Kafka ACL Permissions

Cluster (kafka-cluster)

• DESCRIBE
• DESCRIBE_CONFIGS (cluster monitoring only)

Topic (*)

• DESCRIBE
• DESCRIBE_CONFIGS (cluster monitoring only)
• READ, WRITE (Kafka messages only)

Consumer group (*)

• DESCRIBE
• READ

Validation

1. Run the Agent’s status subcommand and look for kafka_consumer under the Checks section.
2. Ensure the metric kafka.consumer_lag is generated for the appropriate consumer_group.

Data Collected

Metrics

Kafka messages

This integration is used by Data Streams Monitoring to retrieve messages from Kafka on demand.

Events

consumer_lag:
The Datadog Agent emits an event when the value of the consumer_lag metric goes below 0, tagging it with topic, partition and consumer_group.

Service Checks

The Kafka-consumer check does not include any service checks.

Troubleshooting

• Troubleshooting and Deep Dive for Kafka
• Agent failed to retrieve RMIServer stub

Kerberos GSSAPI Authentication

Depending on your Kafka cluster’s Kerberos setup, you may need to configure the following:

• Kafka client configured for the Datadog Agent to connect to the Kafka broker. The Kafka client should be added as a Kerberos principal and added to a Kerberos keytab. The Kafka client should also have a valid kerberos ticket.
• TLS certificate to authenticate a secure connection to the Kafka broker.
  • If JKS keystore is used, a certificate needs to be exported from the keystore and the file path should be configured with the applicable tls_cert and/or tls_ca_cert options.
  • If a private key is required to authenticate the certificate, it should be configured with the tls_private_key option. If applicable, the private key password should be configured with the tls_private_key_password.
• KRB5_CLIENT_KTNAME environment variable pointing to the Kafka client’s Kerberos keytab location if it differs from the default path (for example, KRB5_CLIENT_KTNAME=/etc/krb5.keytab)
• KRB5CCNAME environment variable pointing to the Kafka client’s Kerberos credentials ticket cache if it differs from the default path (for example, KRB5CCNAME=/tmp/krb5cc_xxx)
• If the Datadog Agent is unable to access the environment variables, configure the environment variables in a Datadog Agent service configuration override file for your operating system. The procedure for modifying the Datadog Agent service unit file may vary for different Linux operating systems. For example, in a Linux systemd environment:

Linux Systemd Example

1. Configure the environment variables in an environment file. For example: /path/to/environment/file

   KRB5_CLIENT_KTNAME=/etc/krb5.keytab
   KRB5CCNAME=/tmp/krb5cc_xxx
   

2. Create a Datadog Agent service configuration override file: sudo systemctl edit datadog-agent.service

3. Configure the following in the override file:

   [Service]
   EnvironmentFile=/path/to/environment/file
   

4. Run the following commands to reload the systemd daemon, datadog-agent service, and Datadog Agent:

   sudo systemctl daemon-reload
   sudo systemctl restart datadog-agent.service
   sudo service datadog-agent restart
   

Further Reading

Additional helpful documentation, links, and articles:

Monitoring Kafka performance metricsBLOG Collecting Kafka performance metricsBLOG Monitoring Kafka with DatadogBLOG