Redpanda

Supported OS Linux Mac OS Windows

Intégration1.1.0

Présentation

Redpanda est une plateforme de diffusion compatible avec l’API Kafka permettant de gérer des workloads essentiels.

Associez Datadog à Redpanda pour consulter vos métriques clés et ajouter des groupes de métriques supplémentaires basés sur des besoins spécifiques des utilisateurs.

Configuration

Installation

Téléchargez et lancez l’Agent Datadog.
Installez manuellement l’intégration Redpanda. Consultez la section Utiliser les intégrations de la communauté pour découvrir plus de détails en fonction de votre environnement.

Host

Pour configurer ce check pour un Agent en cours d’exécution sur un host, exécutez la commande datadog-agent integration install -t datadog-redpanda==<VERSION_INTÉGRATION>.

Environnement conteneurisé

Pour utiliser au mieux cette intégration avec l’Agent Docker dans des environnements conteneurisés, générez une image de l’Agent avec l’intégration Redpanda installée.

Pour générer une image à jour de l’Agent, procédez comme suit :

Utilisez le Dockerfile suivant :

FROM gcr.io/datadoghq/agent:latest

ARG INTEGRATION_VERSION=1.0.0

RUN agent integration install -r -t datadog-redpanda==${INTEGRATION_VERSION}

Créez l’image et envoyez-la vers votre registre Docker privé.
Mettez à niveau l’image du conteneur de l’Agent Datadog. Si vous utilisez un chart Helm, modifiez la section agents.image du fichier values.yaml afin de remplacer l’image de l’Agent par défaut :

agents:
  enabled: true
  image:
    tag: <NOUVEAU_TAG>
    repository: <VOTRE_RÉFÉRENTIEL_PRIVÉ>/<NOM_AGENT>

Utilisez le nouveau fichier values.yaml pour mettre à niveau l’Agent :

helm upgrade -f values.yaml <NOM_VERSION> datadog/datadog

Configuration

Host

Collecte de métriques

Pour commencer à recueillir vos données de performance Redpanda, procédez comme suit :

Modifiez le fichier redpanda.d/conf.yaml dans le dossier conf.d/ à la racine du répertoire de configuration de votre Agent. Consultez le fichier d’exemple redpanda.d/conf.yaml.example pour découvrir toutes les options de configuration disponibles.
Redémarrez l’Agent.

Collecte de logs

Par défaut, la collecte de logs est désactivée dans l’Agent Datadog. Elle est disponible à partir de la version 6.0 de l’Agent.

Pour activer les logs, ajoutez ce qui suit à votre fichier datadog.yaml :
```
logs_enabled: true
```
Ajoutez les lignes suivantes à votre fichier redpanda.d/conf.yaml pour commencer à recueillir vos logs Redpanda :
```
 logs:
 - type: journald
   source: redpanda
```

Environnement conteneurisé

Collecte de métriques

Dans les environnements conteneurisés, Autodiscovery est configuré par défaut après l’intégration du check Redpanda à l’image de l’Agent Datadog.

Les métriques sont automatiquement recueillies sur le serveur de Datadog. Pour en savoir plus, consultez la section relative aux modèles d’intégration Autodiscovery.

Collecte de logs

Par défaut, la collecte de logs est désactivée dans l’Agent Datadog. Elle est disponible à partir de la version 6.0 de l’Agent.

Pour activer les logs, consultez la section relative à la collecte de logs Kubernetes.

Paramètre	Valeur
`<CONFIG_LOG>`	`{"source": "redpanda", "service": "redpanda_cluster"}`

Validation

Lancez la sous-commande status de l’Agent et cherchez redpanda dans la section Checks.

Données collectées

Métriques


redpanda.application.build (gauge)	Redpanda build information
redpanda.application.uptime (gauge)	Redpanda uptime in seconds Shown as second
redpanda.controller.log_limit_requests_available (gauge)	Controller log rate limiting. Available rps for group Shown as request
redpanda.controller.log_limit_requests_dropped (count)	Controller log rate limiting. Amount of requests that are dropped due to exceeding limit in group Shown as request
redpanda.partitions.moving_from_node (gauge)	Amount of partitions that are moving from node
redpanda.partitions.moving_to_node (gauge)	Amount of partitions that are moving to node
redpanda.partitions.node_cancelling_movements (gauge)	Amount of cancelling partition movements for node
redpanda.reactor.cpu_busy_seconds (gauge)	Total CPU busy time in seconds Shown as second
redpanda.io_queue.total_read_ops (count)	Total read operations passed in the queue Shown as operation
redpanda.io_queue.total_write_ops (count)	Total write operations passed in the queue Shown as operation
redpanda.kafka.group_offset (gauge)	Consumer group committed offset
redpanda.kafka.group_count (gauge)	Number of consumers in a group
redpanda.kafka.group_topic_count (gauge)	Number of topics in a group
redpanda.kafka.group_lag_max (gauge)	Maximum lag for any partition in the group
redpanda.kafka.group_lag_sum (gauge)	Sum of lag for all partitions in the group
redpanda.cluster.partitions (gauge)	Configured number of partitions for the topic
redpanda.cluster.replicas (gauge)	Configured number of replicas for the topic
redpanda.kafka.request_latency_seconds (gauge)	Internal latency of kafka produce requests Shown as second
redpanda.kafka.under_replicated_replicas (gauge)	Number of under replicated replicas (i.e. replicas that are live, but not at the latest offest)
redpanda.memory.allocated_memory (gauge)	Allocated memory size in bytes Shown as byte
redpanda.memory.available_memory_low_water_mark (gauge)	The low-water mark for available_memory from process start Shown as byte
redpanda.memory.available_memory (gauge)	Total shard memory potentially available in bytes (free_memory plus reclaimable) Shown as byte
redpanda.memory.free_memory (gauge)	Free memory size in bytes Shown as byte
redpanda.node_status.rpcs_received (gauge)	Number of node status RPCs received by this node Shown as request
redpanda.node_status.rpcs_sent (gauge)	Number of node status RPCs sent by this node Shown as request
redpanda.node_status.rpcs_timed_out (gauge)	Number of timed out node status RPCs from this node Shown as request
redpanda.raft.leadership_changes (count)	Number of leadership changes across all partitions of a given topic
redpanda.raft.recovery_bandwidth (gauge)	Bandwidth available for partition movement. bytes/sec
redpanda.pandaproxy.request_errors (count)	Total number of rest_proxy server errors Shown as error
redpanda.pandaproxy.request_latency (gauge)	Internal latency of request for rest_proxy Shown as millisecond
redpanda.rpc.active_connections (gauge)	Count of currently active connections Shown as connection
redpanda.rpc.request_errors (count)	Number of rpc errors Shown as error
redpanda.rpc.request_latency_seconds (gauge)	RPC latency Shown as second
redpanda.scheduler.runtime_seconds (count)	Accumulated runtime of task queue associated with this scheduling group Shown as second
redpanda.schema_registry.errors (count)	Total number of schema_registry server errors Shown as error
redpanda.schema_registry_latency_seconds (gauge)	Internal latency of request for schema_registry Shown as second
redpanda.storage.disk_free_bytes (count)	Disk storage bytes free. Shown as byte
redpanda.storage.disk_free_space_alert (gauge)	Status of low storage space alert. 0-OK, 1-Low Space 2-Degraded
redpanda.storage.disk_total_bytes (count)	Total size of attached storage, in bytes. Shown as byte
redpanda.cloud.client_backoff (count)	Total number of requests that backed off
redpanda.cloud.client_download_backoff (count)	Total number of download requests that backed off
redpanda.cloud.client_downloads (count)	Total number of requests that downloaded an object from cloud storage
redpanda.cloud.client_not_found (count)	Total number of requests for which the object was not found
redpanda.cloud.client_upload_backoff (count)	Total number of upload requests that backed off
redpanda.cloud.client_uploads (count)	Total number of requests that uploaded an object to cloud storage
redpanda.cloud.storage.active_segments (gauge)	Number of remote log segments currently hydrated for read
redpanda.cloud.storage.cache_op_hit (count)	Number of get requests for objects that are already in cache.
redpanda.cloud.storage.op_in_progress_files (gauge)	Number of files that are being put to cache.
redpanda.cloud.storage.cache_op_miss (count)	Number of get requests that are not satisfied from the cache.
redpanda.cloud.storage.op_put (count)	Number of objects written into cache. Shown as operation
redpanda.cloud.storage.cache_space_files (gauge)	Number of objects in cache.
redpanda.cloud.storage.cache_space_size_bytes (gauge)	Sum of size of cached objects. Shown as byte
redpanda.cloud.storage.deleted_segments (count)	Number of segments that have been deleted from S3 for the topic. This may grow due to retention or non compacted segments being replaced with their compacted equivalent.
redpanda.cloud.storage.errors (count)	Number of transmit errors Shown as error
redpanda.cloud.storage.housekeeping.drains (gauge)	Number of times upload housekeeping queue was drained
redpanda.cloud.storage.housekeeping.jobs_completed (count)	Number of executed housekeeping jobs
redpanda.cloud.storage.housekeeping.jobs_failed (count)	Number of failed housekeeping jobs Shown as error
redpanda.cloud.storage.housekeeping.jobs_skipped (count)	Number of skipped housekeeping jobs
redpanda.cloud.storage.housekeeping.pauses (gauge)	Number of times upload housekeeping was paused
redpanda.cloud.storage.housekeeping.resumes (gauge)	Number of times upload housekeeping was resumed
redpanda.cloud.storage.housekeeping.rounds (count)	Number of upload housekeeping rounds
redpanda.cloud.storage.jobs.cloud_segment_reuploads (gauge)	Number of segment reuploads from cloud storage sources (cloud storage cache or direct download from cloud storage)
redpanda.cloud.storage.jobs.local_segment_reuploads (gauge)	Number of segment reuploads from local data directory
redpanda.cloud.storage.jobs.manifest_reuploads (gauge)	Number of manifest reuploads performed by all housekeeping jobs
redpanda.cloud.storage.jobs.metadata_syncs (gauge)	Number of archival configuration updates performed by all housekeeping jobs
redpanda.cloud.storage.jobs.segment_deletions (gauge)	Number of segments deleted by all housekeeping jobs
redpanda.cloud.storage.readers (gauge)	Total number of segments pending deletion from the cloud for the topic
redpanda.cloud.storage.segments (gauge)	Total number of uploaded bytes for the topic
redpanda.cloud.storage.segments_pending_deletion (gauge)	Number of read cursors for hydrated remote log segments
redpanda.cloud.storage.uploaded_bytes (count)	Total number of accounted segments in the cloud for the topic Shown as byte
redpanda.cluster.brokers (gauge)	Number of configured brokers in the cluster
redpanda.cluster.controller_log_limit_requests_dropped (count)	Controller log rate limiting. Amount of requests that are dropped due to exceeding limit in group
redpanda.cluster.partition_num_with_broken_rack_constraint (gauge)	Number of partitions that don’t satisfy the rack awareness constraint
redpanda.cluster.topics (gauge)	Number of topics in the cluster
redpanda.cluster.unavailable_partitions (gauge)	Number of partitions that lack quorum among replicants
redpanda.kafka.partition_committed_offset (gauge)	Latest committed offset for the partition (i.e. the offset of the last message safely persisted on most replicas)
redpanda.kafka.partitions (gauge)	Configured number of partitions for the topic
redpanda.kafka.replicas (gauge)	Configured number of replicas for the topic
redpanda.kafka.request_bytes (count)	Total number of bytes produced per topic Shown as byte

Événements

L’intégration Redpanda n’inclut aucun événement.

Checks de service

redpanda.openmetrics.health

Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.

Statuses: ok, critical

Dépannage

Besoin d’aide ? Contactez l’assistance Datadog.