Requisitos previos

• Datadog Agent v7.34.0 o posterior

Bibliotecas compatibles

Instalación

.NET utiliza la instrumentación automática para inyectar y extraer los metadatos adicionales que necesita Data Streams Monitoring para medir las latencias de extremo a extremo y la relación entre colas y servicios.

entorno:
  - DD_DATA_STREAMS_ENABLED: "true"
  - DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED: "true"

Monitoring Kafka Pipelines

Data Streams Monitoring uses message headers to propagate context through Kafka streams. If log.message.format.version is set in the Kafka broker configuration, it must be set to 0.11.0.0 or higher. Data Streams Monitoring is not supported for versions lower than this.

Monitoring SQS pipelines

Data Streams Monitoring uses one message attribute to track a message’s path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.

Monitoring RabbitMQ pipelines

The RabbitMQ integration can provide detailed monitoring and metrics of your RabbitMQ deployments. For full compatibility with Data Streams Monitoring, Datadog recommends configuring the integration as follows:

instances:
  - prometheus_plugin:
      url: http://<HOST>:15692
      unaggregated_endpoint: detailed?family=queue_coarse_metrics&family=queue_consumer_count&family=channel_exchange_metrics&family=channel_queue_exchange_metrics&family=node_coarse_metrics

This ensures that all RabbitMQ graphs populate, and that you see detailed metrics for individual exchanges as well as queues.

Monitoring SNS-to-SQS pipelines

To monitor a data pipeline where Amazon SNS talks directly to Amazon SQS, you must enable Amazon SNS raw message delivery.

Monitoring Azure Service Bus

Setting up Data Streams Monitoring for Azure Service Bus applications requires additional configuration for the instrumented application.

1. Either set the environment variable AZURE_EXPERIMENTAL_ENABLE_ACTIVITY_SOURCE to true, or in your application code set the Azure.Experimental.EnableActivitySource context switch to true. This instructs the Azure Service Bus library to generate tracing information. See Azure SDK documentation for more details.
2. Set the DD_TRACE_OTEL_ENABLED environment variable to true. This instructs the .NET auto-instrumentation to listen to the tracing information generated by the Azure Service Bus Library and enables the inject and extract operations required for Data Streams Monitoring.

Monitorización de conectores

Conectores de Confluent Cloud

Data Streams Monitoring can automatically discover your Confluent Cloud connectors and visualize them within the context of your end-to-end streaming data pipeline.

Setup

1. Install and configure the Datadog-Confluent Cloud integration.

2. In Datadog, open the Confluent Cloud integration tile.

   

   Under Actions, a list of resources populates with detected clusters and connectors. Datadog attempts to discover new connectors every time you view this integration tile.

3. Select the resources you want to add.

4. Click Add Resources.

5. Navigate to Data Streams Monitoring to visualize the connectors and track connector status and throughput.

Tamaño de los mensajes

Cuando Data Streams Monitoring está activado en el modo por defecto, algunos mensajes no se instrumentan cuando son demasiado pequeños, o demasiado grandes.

Los siguientes umbrales de tamaño se aplican cuando Data Streams Monitoring está activado en el modo por defecto:

• Kafka

  • Los mensajes de menos de 34 bytes no se instrumentan por defecto.

• RabbitMQ

  • Los mensajes de más de 128 kilobytes no se instrumentan por defecto.

• Amazon Kinesis

  • Los mensajes de menos de 34 bytes no se instrumentan por defecto.

Referencias adicionales

Más enlaces, artículos y documentación útiles:

Detectar de forma automática conectores de Confluent Cloud y monitorizar el rendimiento fácilmente en Data Streams MonitoringBLOG