Overview

Connect Kafka to Datadog to:

• Visualize the performance of your cluster in real time.
• Correlate the performance of Kafka with the rest of your applications.

This check has a limit of 350 metrics per instance. The number of returned metrics is indicated in the Agent status output. Specify the metrics you are interested in by editing the configuration below. For more detailed instructions on customizing the metrics to collect, see the JMX Checks documentation.

To collect Kafka consumer metrics, see the kafka_consumer check.

If you would benefit from visualizing the topology of your streaming data pipelines, or from investigating localized bottlenecks within your data streams setup, check out Data Streams Monitoring.

Note: This integration attached sample configuration works only for Kafka >= 0.8.2. If you are running a version older than that, see the Agent v5.2.x released sample files.

Setup

Installation

The Agent’s Kafka check is included in the Datadog Agent package, so you don’t need to install anything else on your Kafka nodes.

The check collects metrics from JMX with JMXFetch. A JVM is needed on each kafka node so the Agent can run JMXFetch. The same JVM that Kafka uses can be used for this.

Note: The Kafka check cannot be used with Managed Streaming for Apache Kafka (Amazon MSK). Use the Amazon MSK integration instead.

Configuration

Validation

Run the Agent’s status subcommand and look for kafka under the JMXFetch section:

========
JMXFetch
========
  Initialized checks
  ==================
    kafka
      instance_name : kafka-localhost-9999
      message :
      metric_count : 46
      service_check_count : 0
      status : OK

Data Collected

Metrics

Events

The Kafka check does not include any events.

Service Checks

kafka.can_connect
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored Kafka instance, WARNING if no metrics are collected, and OK otherwise.
Statuses: ok, critical, warning

Troubleshooting

• Troubleshooting and Deep Dive for Kafka
• Agent failed to retrieve RMIServer stub

Further Reading

• Monitoring Kafka performance metrics
• Collecting Kafka performance metrics
• Monitoring Kafka with Datadog
• Kafka Overview on the Knowledge Center

Kafka Consumer Integration

Overview

This Agent check only collects metrics for message offsets. If you want to collect JMX metrics from the Kafka brokers or Java-based consumers/producers, see the Kafka check.

If you would benefit from visualizing the topology of your streaming data pipelines, or from investigating localized bottlenecks within your data streams setup, check out Data Streams Monitoring.

This check fetches the highwater offsets from the Kafka brokers, consumer offsets that are stored in Kafka or Zookeeper (for old-style consumers), and the calculated consumer lag (which is the difference between the broker offset and the consumer offset).

Note: This integration ensures that consumer offsets are checked before broker offsets; in the worst case, consumer lag may be a little overstated. Checking these offsets in the reverse order can understate consumer lag to the point of having negative values, which is a dire scenario usually indicating messages are being skipped.

Setup

Installation

The Agent’s Kafka consumer check is included in the Datadog Agent package. No additional installation is needed on your Kafka nodes.

Configuration

Host

To configure this check for an Agent running on a host:

Metric collection

1. Edit the kafka_consumer.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory. See the sample kafka_consumer.d/conf.yaml for all available configuration options.

2. Restart the Agent.

Log collection

This check does not collect additional logs. To collect logs from Kafka brokers, see log collection instructions for Kafka.

Containerized

For containerized environments, see the Autodiscovery Integration Templates for guidance on applying the parameters below.

Metric collection

Log collection

This check does not collect additional logs. To collect logs from Kafka brokers, see log collection instructions for Kafka.

Validation

Run the Agent’s status subcommand and look for kafka_consumer under the Checks section.

Data Collected

Metrics

Events

consumer_lag:
The Datadog Agent emits an event when the value of the consumer_lag metric goes below 0, tagging it with topic, partition and consumer_group.

Service Checks

The Kafka-consumer check does not include any service checks.

Troubleshooting

• Troubleshooting and Deep Dive for Kafka
• Agent failed to retrieve RMIServer stub

Kerberos GSSAPI Authentication

Depending on your Kafka cluster’s Kerberos setup, you may need to configure the following:

• Kafka client configured for the Datadog Agent to connect to the Kafka broker. The Kafka client should be added as a Kerberos principal and added to a Kerberos keytab. The Kafka client should also have a valid kerberos ticket.
• TLS certificate to authenticate a secure connection to the Kafka broker.
  • If JKS keystore is used, a certificate needs to be exported from the keystore and the file path should be configured with the applicable tls_cert and/or tls_ca_cert options.
  • If a private key is required to authenticate the certificate, it should be configured with the tls_private_key option. If applicable, the private key password should be configured with the tls_private_key_password.
• KRB5_CLIENT_KTNAME environment variable pointing to the Kafka client’s Kerberos keytab location if it differs from the default path (for example, KRB5_CLIENT_KTNAME=/etc/krb5.keytab)
• KRB5CCNAME environment variable pointing to the Kafka client’s Kerberos credentials ticket cache if it differs from the default path (for example, KRB5CCNAME=/tmp/krb5cc_xxx)
• If the Datadog Agent is unable to access the environment variables, configure the environment variables in a Datadog Agent service configuration override file for your operating system. The procedure for modifying the Datadog Agent service unit file may vary for different Linux operating systems. For example, in a Linux systemd environment:

Linux Systemd Example

1. Configure the environment variables in an environment file. For example: /path/to/environment/file

KRB5_CLIENT_KTNAME=/etc/krb5.keytab
KRB5CCNAME=/tmp/krb5cc_xxx

2. Create a Datadog Agent service configuration override file: sudo systemctl edit datadog-agent.service

3. Configure the following in the override file:

[Service]
EnvironmentFile=/path/to/environment/file

4. Run the following commands to reload the systemd daemon, datadog-agent service, and Datadog Agent:

sudo systemctl daemon-reload
sudo systemctl restart datadog-agent.service
sudo service datadog-agent restart

Further Reading

• Monitoring Kafka performance metrics
• Collecting Kafka performance metrics
• Monitoring Kafka with Datadog