MapR

Supported OS Linux

Integration version1.11.0

Información general

Este check monitoriza MapR 6.1+ a través del Datadog Agent.

Configuración

Sigue las instrucciones de abajo para instalar y configurar este check para un Agent que se ejecuta en un host.

Instalación

El check de MapR está incluido en el paquete del Datadog Agent, pero requiere que se realicen operaciones de configuración adicionales.

Requisitos previos

Pasos de instalación para cada nodo:

  1. Instala el Agent.

  2. Instala la biblioteca librdkafka, que se requiere para mapr-streams-library, siguiendo estas instrucciones.

  3. Instala la biblioteca mapr-streams-library con el siguiente comando:

    sudo -u dd-agent /opt/datadog-agent/embedded/bin/pip install --global-option=build_ext --global-option="--library-dirs=/opt/mapr/lib" --global-option="--include-dirs=/opt/mapr/include/" mapr-streams-python.

    Si utilizas Python 3 con el Agent v7, reemplaza pip por pip3.

  4. Añade /opt/mapr/lib/ a /etc/ld.so.conf (o un archivo en /etc/ld.so.conf.d/). Esto es necesario para que la mapr-streams-library que utiliza el Agent encuentre las bibliotecas compartidas de MapR.

  5. Vuelve a cargar las bibliotecas ejecutando sudo ldconfig.

  6. Configura la integración mediante la especificación de la localización del ticket.

Notas adicionales

  • Si no tienes habilitada la “seguridad” en el clúster, puedes continuar sin un ticket.
  • Si tu entorno de producción no admite herramientas de compilación como gcc (necesarias para compilar la mapr-streams-library), es posible generar un archivo wheel compilado de la biblioteca en una instancia de desarrollo y distribuir el archivo wheel compilado a producción. Los hosts de desarrollo y producción tienen que ser lo suficientemente similares para que el archivo wheel compilado sea compatible. Puedes ejecutar sudo -u dd-agent /opt/datadog-agent/embedded/bin/pip wheel --global-option=build_ext --global-option="--library-dirs=/opt/mapr/lib" --global-option="--include-dirs=/opt/mapr/include/" mapr-streams-python para crear el archivo de wheel en la máquina de desarrollo. Luego, debes ejecutar sudo -u dd-agent /opt/datadog-agent/embedded/bin/pip install <THE_WHEEL_FILE> en la máquina de producción.
  • Si utilizas Python 3 con el Agent v7, asegúrate de reemplazar pip por pip3 al instalar la mapr-streams-library.

Configuración

Recopilación de métricas

  1. Edita el archivo mapr.d/conf.yaml, que se encuentra en la carpeta conf.d/ en la raíz del directorio de configuración del Agent, para empezar a recopilar los datos de rendimiento de MapR. Consulta el archivo de ejemplo mapr.d/conf.yaml para conocer todas las opciones de configuración disponibles.
  2. Define el parámetro ticket_location de la configuración como la ruta del ticket de larga duración que creaste.
  3. Reinicia el Agent.

Recopilación de logs

MapR utiliza FluentD para los logs. Debes usar el complemento de FluentD para Datadog para recopilar los logs de MapR. El siguiente comando descarga e instala el complemento en el directorio correcto.

curl https://raw.githubusercontent.com/DataDog/fluent-plugin-datadog/master/lib/fluent/plugin/out_datadog.rb -o /opt/mapr/fluentd/fluentd-<VERSION>/lib/fluentd-<VERSION>-linux-x86_64/lib/app/lib/fluent/plugin/out_datadog.rb

Luego, actualiza /opt/mapr/fluentd/fluentd-<VERSION>/etc/fluentd/fluentd.conf con la siguiente sección.

<match *>
  @type copy
  <store> # Esta sección está aquí de manera predeterminada y envía los logs a ElasticCache for Kibana.
    @include /opt/mapr/fluentd/fluentd-<VERSION>/etc/fluentd/es_config.conf
    include_tag_key true
    tag_key service_name
  </store>
  <store> # Esta sección también reenvía todos los logs a Datadog:
    @type datadog
    @id dd_agent
    include_tag_key true
    dd_source mapr  # Establece "source: mapr" en cada log para permitir el parseo automático en Datadog.
    dd_tags "<KEY>:<VALUE>"
    service <YOUR_SERVICE_NAME>
    api_key <YOUR_API_KEY>
  </store>

Consulta el fluent_datadog_plugin para obtener más detalles sobre las opciones que puedes utilizar.

Validación

Ejecuta el subcomando de estado del Agent y busca mapr en la sección Checks.

Datos recopilados

Métricas

mapr.alarms.alarm_raised
(gauge)
The number of threads that are waiting to be executed. This can occur when a thread must wait for another thread to perform an action before proceeding.
Shown as thread
mapr.cache.lookups_data
(count)
The number of cache lookups in the block cache.
Shown as operation
mapr.cache.lookups_dir
(count)
The number of cache lookups in the table LRU cache. The table LRU is used for storing internal B-Tree leaf pages.
Shown as operation
mapr.cache.lookups_inode
(count)
The number of cache lookups in the inode cache.
mapr.cache.lookups_largefile
(count)
The number of cache lookups in the large file LRU cache. The large file LRU is used for storing files with size greater than 64K and MapR database data pages.
Shown as operation
mapr.cache.lookups_meta
(count)
The number of cache lookups on the meta LRU cache. The meta LRU is used for storing internal B-Tree pages.
Shown as operation
mapr.cache.lookups_smallfile
(count)
The number of cache lookups on the small file LRU cache. This LRU is used for storing files with size less than 64K and MapR database index pages.
Shown as operation
mapr.cache.lookups_table
(count)
The number of cache lookups in the table LRU cache. The table LRU is used for storing internal B-Tree leaf pages.
Shown as operation
mapr.cache.misses_data
(count)
The number of cache misses in the block cache.
Shown as miss
mapr.cache.misses_dir
(count)
The number of cache misses on the table LRU cache.
Shown as miss
mapr.cache.misses_inode
(count)
The number of cache misses in the inode cache.
Shown as miss
mapr.cache.misses_largefile
(count)
The number of cache misses on the large file LRU cache.
Shown as miss
mapr.cache.misses_meta
(count)
The number of cache misses on the meta LRU cache.
Shown as miss
mapr.cache.misses_smallfile
(count)
The number of cache misses on the small file LRU cache.
Shown as miss
mapr.cache.misses_table
(count)
The number of cache misses on the table LRU cache.
Shown as miss
mapr.cldb.cluster_cpu_total
(gauge)
The number of physical CPUs in the cluster.
Shown as cpu
mapr.cldb.cluster_cpubusy_percent
(gauge)
The aggregate percentage of busy CPUs in the cluster.
Shown as percent
mapr.cldb.cluster_disk_capacity
(gauge)
The storage capacity for MapR disks in GB.
Shown as gibibyte
mapr.cldb.cluster_diskspace_used
(gauge)
The amount of MapR disks used in GB.
Shown as gibibyte
mapr.cldb.cluster_memory_capacity
(gauge)
The memory capacity in MB.
Shown as mebibyte
mapr.cldb.cluster_memory_used
(gauge)
The amount of used memory in MB.
Shown as mebibyte
mapr.cldb.containers
(gauge)
The number of containers currently in the cluster.
Shown as container
mapr.cldb.containers_created
(count)
The cumulative number of containers created in the cluster. This value includes containers that have been deleted.
Shown as container
mapr.cldb.containers_unusable
(gauge)
The number of containers that are no longer usable. The CLDB marks a container as unusable when the node that stores the container is offline for 1 hour or more.
Shown as container
mapr.cldb.disk_space_available
(gauge)
The amount of disk space available in GB.
Shown as gibibyte
mapr.cldb.nodes_in_cluster
(gauge)
The number of nodes in the cluster.
Shown as node
mapr.cldb.nodes_offline
(gauge)
The number of nodes in the cluster that are offline.
Shown as node
mapr.cldb.rpc_received
(count)
The number of RPCs received.
Shown as operation
mapr.cldb.rpcs_failed
(count)
The number of RPCs failed.
Shown as operation
mapr.cldb.storage_pools_cluster
(gauge)
The number of storage pools.
mapr.cldb.storage_pools_offline
(gauge)
The number of offline storage pools.
mapr.cldb.volumes
(gauge)
The number of volumes created, including system volumes.
Shown as volume
mapr.db.append_bytes
(count)
The number of bytes written by append RPCs
Shown as byte
mapr.db.append_rpcrows
(count)
The number of rows written by append RPCs
Shown as object
mapr.db.append_rpcs
(count)
The number of MapR Database append RPCs completed
Shown as operation
mapr.db.cdc.pending_bytes
(gauge)
The number of bytes of CDC data remaining to be sent
Shown as byte
mapr.db.cdc.sent_bytes
(count)
The number of bytes of CDC data sent
Shown as byte
mapr.db.checkandput_bytes
(count)
The number of bytes written by check and put RPCs
Shown as byte
mapr.db.checkandput_rpcrows
(count)
The number of rows written by check and put RPCs
Shown as object
mapr.db.checkandput_rpcs
(count)
The number of MapR Database check and put RPCs completed
Shown as operation
mapr.db.flushes
(count)
The number of flushes that reorganize data from bucket files (unsorted data) to spill files (sorted data) when the bucket size exceeds a threshold.
Shown as flush
mapr.db.forceflushes
(count)
The number of flushes that reorganize data from bucket files (unsorted data) to spill files (sorted data) when the in-memory bucket file cache fills up.
Shown as flush
mapr.db.fullcompacts
(count)
The number of compactions that combine multiple MapR Database data files containing sorted data (known as spills) into a single spill file.
Shown as operation
mapr.db.get_bytes
(count)
The number of bytes read by get RPCs
Shown as byte
mapr.db.get_currpcs
(gauge)
The number of MapR Database get RPCs in progress
Shown as operation
mapr.db.get_readrows
(count)
The number of rows read by get RPCs
Shown as object
mapr.db.get_resprows
(count)
The number of rows returned from get RPCs
Shown as object
mapr.db.get_rpcs
(count)
The number of MapR database get RPCs completed
Shown as operation
mapr.db.increment_bytes
(count)
The number of bytes written by increment RPCs
Shown as byte
mapr.db.increment_rpcrows
(count)
The number of rows written by increment RPCs
Shown as object
mapr.db.increment_rpcs
(count)
The number of MapR Database increment RPCs completed
Shown as operation
mapr.db.index.pending_bytes
(gauge)
The number of bytes of secondary index data remaining to be sent
Shown as byte
mapr.db.minicompacts
(count)
The number of compactions that combine multiple small data files containing sorted data (known as spills) into a single spill file.
Shown as operation
mapr.db.put_bytes
(count)
The number of bytes written by put RPCs
Shown as byte
mapr.db.put_currpcs
(gauge)
The number of MapR Database put RPCs in progress
Shown as operation
mapr.db.put_readrows
(count)
The number of rows read by put RPCs
Shown as object
mapr.db.put_rpcrows
(count)
The number of rows written by put RPCs. Each MapR Database put RPC can include multiple put rows.
Shown as object
mapr.db.put_rpcs
(count)
The number of MapR Database put RPCs completed
Shown as operation
mapr.db.repl.pending_bytes
(gauge)
The number of bytes of replication data remaining to be sent
Shown as byte
mapr.db.repl.sent_bytes
(count)
The number of bytes sent to replicate data
Shown as byte
mapr.db.scan_bytes
(count)
The number of bytes read by scan RPCs
Shown as byte
mapr.db.scan_currpcs
(gauge)
The number of MapR Database scan RPCs in progress
Shown as operation
mapr.db.scan_readrows
(count)
The number of rows read by scan RPCs
Shown as object
mapr.db.scan_resprows
(count)
The number of rows returned from scan RPCs.
Shown as object
mapr.db.scan_rpcs
(count)
The number of MapR Database scan RPCs completed
Shown as operation
mapr.db.table.latency
(gauge)
The latency of RPC operations on tables,represented as a histogram. Endpoints identify histogram bucket boundaries.
Shown as millisecond
mapr.db.table.read_bytes
(count)
The number of bytes read from tables
Shown as byte
mapr.db.table.read_rows
(count)
The number of rows read from tables
Shown as object
mapr.db.table.resp_rows
(count)
The number of rows returned from tables
Shown as object
mapr.db.table.rpcs
(count)
The number of RPC calls completed on tables
Shown as operation
mapr.db.table.value_cache_hits
(count)
The number of MapR Database operations on tables that utilized the MapR Database value cache
Shown as operation
mapr.db.table.value_cache_lookups
(count)
The number of MapR Database operations on tables that performed a lookup on the MapR Database value cache
Shown as operation
mapr.db.table.write_bytes
(count)
The number of bytes written to tables
Shown as byte
mapr.db.table.write_rows
(count)
The number of rows written to tables
Shown as object
mapr.db.ttlcompacts
(count)
The number of compactions that result in reclamation of disk space due to removal of stale data.
Shown as operation
mapr.db.updateandget_bytes
(count)
The number of bytes written by update and get RPCs
Shown as byte
mapr.db.updateandget_rpcrows
(count)
The number of rows written by update and get RPCs
Shown as object
mapr.db.updateandget_rpcs
(count)
The number of MapR Database update and get RPCs completed
Shown as operation
mapr.db.valuecache_hits
(count)
The number of MapR Database operations that utilized the MapR Database value cache
Shown as operation
mapr.db.valuecache_lookups
(count)
The number of MapR Database operations that performed a lookup on the MapR Database value cache
Shown as operation
mapr.db.valuecache_usedSize
(gauge)
The MapR Database value cache size in MB
Shown as mebibyte
mapr.drill.allocator_root_peak
(gauge)
The peak amount of memory used in bytes by the internal memory allocator.
Shown as byte
mapr.drill.allocator_root_used
(gauge)
The amount of memory used in bytes by the internal memory allocator.
Shown as byte
mapr.drill.blocked_count
(gauge)
The number of threads that are blocked because they are waiting for a monitor lock.
Shown as thread
mapr.drill.count
(gauge)
The number of live threads (including both daemon and non-daemon threads).
Shown as thread
mapr.drill.fd_usage
(gauge)
The ratio of used to total file descriptors.
mapr.drill.fragments_running
(gauge)
The number of query fragments currently running in the drillbit.
Shown as byte
mapr.drill.heap_used
(gauge)
The amount of heap memory used in bytes by the JVM.
Shown as byte
mapr.drill.non_heap_used
(gauge)
The amount of non-heap memory used in bytes by the JVM.
Shown as byte
mapr.drill.queries_completed
(count)
The number of completed, canceled or failed queries for which this drillbit is the foreman.
Shown as byte
mapr.drill.queries_running
(gauge)
The number of running queries for which this drillbit is the foreman.
Shown as byte
mapr.drill.runnable_count
(gauge)
The number of threads executing in the JVM.
Shown as thread
mapr.drill.waiting_count
(gauge)
The number of threads that are waiting to be executed. This can occur when a thread must wait for another thread to perform an action before proceeding.
Shown as thread
mapr.fs.bulk_writes
(count)
The number of bulk-write operations. Bulk-write operations occur when the MapR filesystem container master aggregates multiple file writes from one or more clients into one RPC before replicating the writes.
Shown as write
mapr.fs.bulk_writesbytes
(count)
The number of bytes written by bulk-write operations. Bulk-write operations occur when the MapR filesystem container master aggregates multiple file writes from one or more clients into one RPC before replicating the writes.
Shown as byte
mapr.fs.kvstore_delete
(count)
The number of delete operations on key-value store files which are used by the CLDB and MapR database.
Shown as operation
mapr.fs.kvstore_insert
(count)
The number of insert operations on key-value store files which are used by the CLDB and MapR database.
Shown as operation
mapr.fs.kvstore_lookup
(count)
The number of lookup operations on key-value store files which are used by the CLDB and MapR database.
Shown as operation
mapr.fs.kvstore_scan
(count)
The number of scan operations on key-value store files which are used by the CLDB and MapR database.
Shown as operation
mapr.fs.local_readbytes
(count)
The number of bytes read by applications that are running on the MapR filesystem node.
Shown as byte
mapr.fs.local_reads
(count)
The number of file read operations by applications that are running on the MapR filesystem node.
Shown as read
mapr.fs.local_writebytes
(count)
The number of bytes written by applications that are running on the MapR filesystem node.
Shown as byte
mapr.fs.local_writes
(count)
The number of file write operations by applications that are running on the MapR filesystem node.
Shown as operation
mapr.fs.read_bytes
(count)
The amount of data read remotely in MB.
Shown as mebibyte
mapr.fs.read_cachehits
(count)
The number of cache hits for file reads. This value includes pages that the MapR filesystem populates using readahead mechanism.
Shown as hit
mapr.fs.read_cachemisses
(count)
The number of cache misses for file read operations.
Shown as miss
mapr.fs.reads
(count)
The number of remote reads.
Shown as read
mapr.fs.statstype_create
(count)
The number of file create operations.
Shown as operation
mapr.fs.statstype_lookup
(count)
The number of lookup operations.
Shown as operation
mapr.fs.statstype_read
(count)
The number of file read operations.
Shown as read
mapr.fs.statstype_write
(count)
The number of file write operations.
Shown as write
mapr.fs.write_bytes
(count)
The amount of data written remotely in MB.
Shown as mebibyte
mapr.fs.writes
(count)
The number of remote writes.
Shown as write
mapr.io.read_bytes
(gauge)
The number of MB read from disk.
Shown as mebibyte
mapr.io.reads
(gauge)
The number of MapR Filesystem disk read operations.
Shown as read
mapr.io.write_bytes
(count)
The number of MB written to disk.
Shown as mebibyte
mapr.io.writes
(count)
The number of MapR Filesystem disk write operations.
Shown as write
mapr.metrics.submitted
(gauge)
Number of metrics submitted every check run.
mapr.process.context_switch_involuntary
(count)
The number of involuntary context switches for MapR processes.
Shown as operation
mapr.process.context_switch_voluntary
(count)
The number of voluntary context switches for MapR processes.
Shown as process
mapr.process.cpu_percent
(gauge)
The percentage of CPU used for MapR processes.
Shown as percent
mapr.process.cpu_time.syst
(count)
The amount of time measured in seconds that the process has been in kernel mode.
Shown as second
mapr.process.cpu_time.user
(count)
The amount of time measured in seconds that the process has been in user mode
Shown as second
mapr.process.data
(gauge)
The amount memory in MB used by the data segments of MapR processes.
Shown as mebibyte
mapr.process.disk_octets.read
(count)
The number of bytes read from disk for MapR processes.
Shown as byte
mapr.process.disk_octets.write
(count)
The number of bytes written to disk for MapR processes.
Shown as byte
mapr.process.disk_ops.read
(count)
The number of read operations for MapR processes.
Shown as read
mapr.process.disk_ops.write
(count)
The number of write operations for MapR processes.
Shown as write
mapr.process.mem_percent
(gauge)
The percentage of total system memory (not capped by MapR processes) used for MapR processes.
Shown as percent
mapr.process.page_faults.majflt
(count)
The number of major MapR process faults that required loading a memory page from disk.
Shown as error
mapr.process.page_faults.minflt
(count)
The number of minor MapR process faults that required loading a memory page from disk.
Shown as error
mapr.process.rss
(gauge)
The actual amount of memory in MB used by MapR processes.
Shown as mebibyte
mapr.process.vm
(gauge)
The amount of virtual memory in MB used by MapR processes.
Shown as mebibyte
mapr.rpc.bytes_recd
(count)
The number of bytes received by the MapR Filesystem over RPC.
Shown as byte
mapr.rpc.bytes_sent
(count)
The number of bytes sent by the MapR filesystem over RPC.
Shown as byte
mapr.rpc.calls_recd
(count)
The number of RPC calls received by the MapR filesystem.
Shown as message
mapr.streams.listen_bytes
(count)
The number of megabytes consumed by Streams messages.
Shown as mebibyte
mapr.streams.listen_currpcs
(gauge)
The number of concurrent Stream consumer RPCs.
Shown as object
mapr.streams.listen_msgs
(count)
The number of Streams messages read by the consumer.
Shown as object
mapr.streams.listen_rpcs
(count)
The number of Streams consumer RPCs.
Shown as object
mapr.streams.produce_bytes
(count)
The number of megabytes produced by Streams messages.
Shown as mebibyte
mapr.streams.produce_msgs
(count)
The number of Streams messages produced.
Shown as object
mapr.streams.produce_rpcs
(count)
The number of Streams producer RPCs.
Shown as object
mapr.topology.disks_total_capacity
(gauge)
The disk capacity in gigabytes.
Shown as gibibyte
mapr.topology.disks_used_capacity
(gauge)
The amount disk space used in gigabytes.
Shown as gibibyte
mapr.topology.utilization
(gauge)
The aggregate percentage of CPU utilization.
Shown as percent
mapr.volmetrics.read_latency
(gauge)
The per volume read latency in milliseconds
Shown as millisecond
mapr.volmetrics.read_ops
(count)
A count of the read operations per volume
Shown as operation
mapr.volmetrics.read_throughput
(gauge)
The per volume read throughput in KB
Shown as kibibyte
mapr.volmetrics.write_latency
(gauge)
The per volume write latency in milliseconds
Shown as millisecond
mapr.volmetrics.write_ops
(count)
A count of the write operations per volume
Shown as operation
mapr.volmetrics.write_throughput
(gauge)
The per volume write throughput in KB
Shown as kibibyte
mapr.volume.logical_used
(gauge)
The number of MBs used for logical volumes before compression is applied to the files.
Shown as mebibyte
mapr.volume.quota
(gauge)
The number of megabytes(MB) used for volume quota.
Shown as mebibyte
mapr.volume.snapshot_used
(gauge)
The number of MBs used for snapshots.
Shown as mebibyte
mapr.volume.total_used
(gauge)
The number of MB used for volumes and snapshots.
Shown as mebibyte
mapr.volume.used
(gauge)
The number of MB used for volumes after compression is applied to the files.
Shown as mebibyte

Eventos

El check de MapR no incluye eventos.

Checks de servicio

mapr.can_connect
Returns CRITICAL if the Agent fails to connect and subscribe to the stream topic, OK otherwise.
Statuses: ok, critical

Solucionar problemas

  • El Agent entra en un bucle de error luego de configurar la integración de MapR.

    Ha habido algunos casos en los que la biblioteca de C dentro de mapr-streams-python genera errores de segmentación debido a problemas de permisos. Asegúrate de que el usuario del dd-agent tenga permiso de lectura en el archivo del ticket y que dicho usuario del dd-agent pueda ejecutar comandos de la maprcli cuando la variable de entornoMAPR_TICKETFILE_LOCATION esté apuntando hacia el ticket.

  • La integración parece funcionar correctamente, pero no envía ninguna métrica.

    Asegúrate de dejar que el Agent se ejecute durante al menos un par de minutos, ya que la integración extrae datos de un tema y MapR necesita introducir datos en ese tema. Si eso no funciona, pero al ejecutar el Agent manualmente con sudo se muestran datos, se trata de un problema con los permisos. Revisa todo. El usuario de Linux del dd-agent debe ser capaz de utilizar un ticket almacenado localmente, lo que le permite ejecutar consultas en MapR como usuario X (que puede o no ser el mismo dd-agent). Además, el usuario X debe tener el permiso consume en el flujo /var/mapr/mapr.monitoring/metricstreams.

  • Ves el mensaje confluent_kafka was not imported correctly ...

    El entorno integrado del Agent no pudo ejecutar el comando import confluent_kafka. Esto significa que la mapr-streams-library no se instaló dentro del entorno integrado o que no puede encontrar las bibliotecas mapr-core. El mensaje de error debería incluir más detalles.

¿Necesitas más ayuda? Ponte en contacto con el servicio de asistencia de Datadog.