Amazon MSK

Overview

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data.

This integration uses a crawler that collects metrics from CloudWatch. Read the Amazon MSK (Agent) page for information about monitoring MSK through the Datadog Agent.

Setup

Enable the Amazon MSK crawler to see MSK metrics from CloudWatch in Datadog.

Installation

If you haven’t already, set up the Amazon Web Services integration first.

Metric collection

  1. In the AWS integration page, ensure that Kafka is enabled under the Metric Collection tab.

  2. Install the Amazon MSK integration.

Log collection

Enable logging

Configure Amazon MSK to send logs either to a S3 bucket or to CloudWatch.

Notes:

  • If you log to a S3 bucket, make sure that amazon_msk is set as Target prefix.
  • If you log to a CloudWatch log group, make sure its name contains the substring msk.

Send logs to Datadog

  1. If you haven’t already, set up the Datadog Forwarder Lambda function.

  2. Once the Lambda function is installed, manually add a trigger on the S3 bucket or CloudWatch log group that contains your Amazon MSK logs in the AWS console:

Data Collected

Metrics

See metric-spec.yaml for a list of metrics provided by this integration.

Events

The Amazon MSK crawler does not include any events.

Service Checks

The Amazon MSK integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles:

Monitor Amazon Managed Streaming for Apache Kafka with DatadogBLOG more more