Prerequisites

To start with Data Streams Monitoring, you need recent versions of the Datadog Agent and Java libraries:

• Datadog Agent v7.34.0 or later
• APM enabled with the Java Agent
  • Kafka and RabbitMQ: v1.9.0 or later
  • Amazon SQS: v1.27.0 or later

Installation

Java uses auto-instrumentation to inject and extract additional metadata required by Data Streams Monitoring for measuring end-to-end latencies and the relationship between queues and services. To enable Data Streams Monitoring, set the DD_DATA_STREAMS_ENABLED environment variable to true on services sending messages to (or consuming messages from) Kafka, SQS or RabbitMQ.

Also, set the DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED variable to true so that DD_SERVICE is used as the service name in traces.

For example:

environment:
  - DD_DATA_STREAMS_ENABLED: "true"
  - DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED: "true"

As an alternative, you can set the -Ddd.data.streams.enabled=true system property by running the following when you start your Java application:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.data.streams.enabled=true -Ddd.trace.remove.integration.service.names.enabled=true -jar path/to/your/app.jar

One-Click Installation

To set up Data Streams Monitoring from the Datadog UI without needing to restart your service, use Configuration at Runtime. Navigate to the APM Service Page and Enable DSM.

Supported libraries

Data Streams Monitoring supports the confluent-kafka library.

Monitoring SQS pipelines

Data Streams Monitoring uses one message attribute to track a message’s path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.

Further Reading

Additional helpful documentation, links, and articles:

Kafka IntegrationDOCUMENTATION Service CatalogDOCUMENTATION