Setup Data Streams Monitoring for Java

Prerequisites

Supported libraries

TechnologyLibraryMinimal tracer versionRecommended tracer version
Kafkakafka-clients (v3.7 is not fully supported)1.9.01.43.0 or later
RabbitMQamqp-client1.9.01.42.2 or later
Amazon SQSaws-java-sdk-sqs (v1)1.27.01.42.2 or later
Amazon SQSsqs (v2)1.27.01.42.2 or later
Amazon KinesisKinesis (v1)1.22.01.42.2 or later
Amazon KinesisKinesis (v2)1.22.01.42.2 or later
Amazon SNSSNS (v1)1.31.01.42.2 or later
Amazon SNSSNS (v2)1.31.01.42.2 or later
Google PubSubGoogle Cloud Pub/Sub1.25.01.42.2 or later

Installation

To enable Data Streams Monitoring, set the following environment variables to true on services that are sending or consuming messages:

  • DD_DATA_STREAMS_ENABLED
  • DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED
    environment:
  - DD_DATA_STREAMS_ENABLED: "true"
  - DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED: "true"

    Run the following when you start your Java application:

    java -javaagent:/path/to/dd-java-agent.jar -Ddd.data.streams.enabled=true -Ddd.trace.remove.integration-service-names.enabled=true -jar path/to/your/app.jar

    One-Click Installation

    To set up Data Streams Monitoring from the Datadog UI without needing to restart your service, use Configuration at Runtime. Navigate to the APM Service Page and Enable DSM.

    Enable the Data Streams Monitoring from the Dependencies section of the APM Service Page
    Setup

    Use Datadog’s Java tracer, dd-trace-java, to collect information from your Kafka Connect workers.

    1. Add the dd-java-agent.jar file to your Kafka Connect workers. Ensure that you are using dd-trace-java v1.44+.
    2. Modify your Java options to include the Datadog Java tracer on your worker nodes. For example, on Strimzi, modify STRIMZI_JAVA_OPTS to add -javaagent:/path/to/dd-java-agent.jar.

    Monitoring SQS pipelines

    Data Streams Monitoring uses one message attribute to track a message’s path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.

    Monitoring SNS-to-SQS pipelines

    To monitor a data pipeline where Amazon SNS talks directly to Amazon SQS, you must perform the following additional configuration steps:

      • Set the environment variable DD_TRACE_SQS_BODY_PROPAGATION_ENABLED to true.

        For example:

        environment:
  - DD_DATA_STREAMS_ENABLED: "true"
  - DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED: "true"
  - DD_TRACE_SQS_BODY_PROPAGATION_ENABLED: "true"

      • Ensure that you are using Java tracer v1.44.0+.

      Enable Amazon SNS raw message delivery.

      Monitoring Kinesis pipelines

      There are no message attributes in Kinesis to propagate context and track a message’s full path through a Kinesis stream. As a result, Data Streams Monitoring’s end-to-end latency metrics are approximated based on summing latency on segments of a message’s path, from the producing service through a Kinesis Stream, to a consumer service. Throughput metrics are based on segments from the producing service through a Kinesis Stream, to the consumer service. The full topology of data streams can still be visualized through instrumenting services.

      Manual instrumentation

      Data Streams Monitoring propagates context through message headers. If you are using a message queue technology that is not supported by DSM, a technology without headers (such as Kinesis), or Lambdas, use manual instrumentation to set up DSM.

      Monitoring connectors

      Confluent Cloud connectors

      Data Streams Monitoring can automatically discover your Confluent Cloud connectors and visualize them within the context of your end-to-end streaming data pipeline.

      Setup

      1. Install and configure the Datadog-Confluent Cloud integration.

      2. In Datadog, open the Confluent Cloud integration tile.

        The Confluent Cloud integration tile in Datadog, on the Configure tab. Under an Actions heading, a table titled '13 Resources autodiscovered' containing a list of resources and checkboxes for each resource.

        Under Actions, a list of resources populates with detected clusters and connectors. Datadog attempts to discover new connectors every time you view this integration tile.

      3. Select the resources you want to add.

      4. Click Add Resources.

      5. Navigate to Data Streams Monitoring to visualize the connectors and track connector status and throughput.

      Self-hosted Kafka connectors

      This feature is in Preview.

      Data Streams Monitoring can collect information from your self-hosted Kafka connectors. In Datadog, these connectors are shown as services connected to Kafka topics. Datadog collects throughput to and from all Kafka topics. Datadog does not collect connector status or sinks and sources from self-hosted Kafka connectors.

      Further reading

      Additional helpful documentation, links, and articles:

      Kafka IntegrationDOCUMENTATION more more Software CatalogDOCUMENTATION more more Autodiscover Confluent Cloud connectors and easily monitor performance in Data Streams MonitoringBLOG more more