Setup Data Streams Monitoring for Java

Docs > Data Streams Monitoring > Setup Data Streams Monitoring for Java

Prerequisites

Datadog Agent v7.34.0 or later

Supported libraries

Technology	Library	Minimal tracer version	Recommended tracer version
Kafka	kafka-clients (v3.7 is not fully supported)	1.9.0	1.43.0 or later
RabbitMQ	amqp-client	1.9.0	1.42.2 or later
Amazon SQS	aws-java-sdk-sqs (v1)	1.27.0	1.42.2 or later
Amazon SQS	sqs (v2)	1.27.0	1.42.2 or later
Amazon Kinesis	Kinesis (v1)	1.22.0	1.42.2 or later
Amazon Kinesis	Kinesis (v2)	1.22.0	1.42.2 or later
Amazon SNS	SNS (v1)	1.31.0	1.42.2 or later
Amazon SNS	SNS (v2)	1.31.0	1.42.2 or later
Google PubSub	Google Cloud Pub/Sub	1.25.0	1.42.2 or later

Installation

Java uses auto-instrumentation to inject and extract additional metadata required by Data Streams Monitoring for measuring end-to-end latencies and the relationship between queues and services. To enable Data Streams Monitoring, set the DD_DATA_STREAMS_ENABLED environment variable to true on services sending messages to (or consuming messages from) Kafka, SQS or RabbitMQ.

Also, set the DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED variable to true so that DD_SERVICE is used as the service name in traces.

For example:

environment:
  - DD_DATA_STREAMS_ENABLED: "true"
  - DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED: "true"

As an alternative, you can set the -Ddd.data.streams.enabled=true system property by running the following when you start your Java application:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.data.streams.enabled=true -Ddd.trace.remove.integration-service-names.enabled=true -jar path/to/your/app.jar

One-Click Installation

To set up Data Streams Monitoring from the Datadog UI without needing to restart your service, use Configuration at Runtime. Navigate to the APM Service Page and Enable DSM.

Enable the Data Streams Monitoring from the Dependencies section of the APM Service Page

Monitoring SQS pipelines

Data Streams Monitoring uses one message attribute to track a message’s path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.

To monitor a data pipeline where Amazon SNS talks directly to Amazon SQS, you must perform the following additional configuration steps:

Set the environment variable DD_TRACE_SQS_BODY_PROPAGATION_ENABLED to true.

For example:

environment:
  - DD_DATA_STREAMS_ENABLED: "true"
  - DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED: "true"
  - DD_TRACE_SQS_BODY_PROPAGATION_ENABLED: "true"

Ensure that you are using Java tracer v1.44.0+.

Enable Amazon SNS raw message delivery.

Monitoring Kinesis pipelines

There are no message attributes in Kinesis to propagate context and track a message’s full path through a Kinesis stream. As a result, Data Streams Monitoring’s end-to-end latency metrics are approximated based on summing latency on segments of a message’s path, from the producing service through a Kinesis Stream, to a consumer service. Throughput metrics are based on segments from the producing service through a Kinesis Stream, to the consumer service. The full topology of data streams can still be visualized through instrumenting services.

Manual instrumentation

Data Streams Monitoring propagates context through message headers. If you are using a message queue technology that is not supported by DSM, a technology without headers (such as Kinesis), or Lambdas, use manual instrumentation to set up DSM.