Hive
Rapport de recherche Datadog : Bilan sur l'adoption de l'informatique sans serveur Rapport : Bilan sur l'adoption de l'informatique sans serveur

Hive

Agent Check Check de l'Agent

Supported OS: Linux Mac OS Windows

Présentation

Ce check surveille deux composants de Hive : Hive Metastore et HiveServer2.

Configuration

Installation

Le check Hive est inclus avec le paquet de l’Agent Datadog. Vous n’avez donc rien d’autre à installer sur votre serveur.

Configuration

Configuration de Hive

  1. Modifiez le fichier de configuration de Hive dans HIVE_HOME/conf/hive-site.xml pour activer les métriques de Hive Metastore et de HiveServer2 en ajoutant les propriétés suivantes :

    <property>
     <name>hive.metastore.metrics.enabled</name>
     <value>true</value>
    </property>
    <property>
     <name>hive.server2.metrics.enabled</name>
     <value>true</value>
    </property>
  2. Activez une connexion JMX à distance pour HiveServer2 et/ou Hive Metastore. Définissez par exemple la variable d’environnement HADOOP_CLIENT_OPTS :

    export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.port=8808"

    Redémarrez ensuite HiveServer2 ou Hive Metastore. Hive Metastore et HiveServer2 ne peuvent pas partager la même connexion JMX.

Host

Suivez les instructions ci-dessous pour installer et configurer ce check lorsque l’Agent est exécuté sur un host. Consultez la section Environnement conteneurisé pour en savoir plus sur les environnements conteneurisés.

Collecte de métriques
  1. Modifiez le fichier hive.d/conf.yaml dans le dossier conf.d/ à la racine du répertoire de configuration de votre Agent pour commencer à recueillir vos données de performance Hive. Consultez le fichier d’exemple hive.d/conf.yaml pour découvrir toutes les options de configuration disponibles.

    Ce check prévoit une limite de 350 métriques par instance. Le nombre de métriques renvoyées est indiqué sur la page d’informations. Choisissez les métriques qui vous intéressent en modifiant la configuration ci-dessous. Pour découvrir comment modifier la liste des métriques à recueillir, consultez la documentation relative aux checks JMX afin d’obtenir des instructions détaillées. Si vous souhaitez surveiller davantage de métriques, contactez l’assistance Datadog.

  2. Redémarrez l’Agent.

Collecte de logs

Disponible à partir des versions > 6.0 de l’Agent

  1. La collecte de logs est désactivée par défaut dans l’Agent Datadog. Vous devez l’activer dans datadog.yaml :

    logs_enabled: true
  2. Ajoutez ce bloc de configuration à votre fichier hive.d/conf.yaml pour commencer à recueillir vos logs Hive :

     logs:
       - type: file
         path: /tmp/<USER>/hive.log
         source: hive
         service: '<SERVICE_NAME>'
         log_processing_rules:
           - type: multi_line
             name: new_log_start_with_date
             pattern: \d{4}\-\d{2}\-\d{2}

    Modifiez les valeurs des paramètres path et service et configurez-les pour votre environnement. Consultez le fichier d’exemple hive.d/conf.yaml pour découvrir toutes les options de configuration disponibles.

  3. Redémarrez l’Agent.

Environnement conteneurisé

Consultez la documentation relative aux modèles d’intégration Autodiscovery pour découvrir comment appliquer les paramètres ci-dessous à un environnement conteneurisé.

Collecte de métriques

Pour recueillir des métriques avec l’intégration Datadog/Hive, consultez le guide Autodiscovery avec JMX.

Collecte de logs

Disponible à partir des versions > 6.0 de l’Agent

La collecte des logs est désactivée par défaut dans l’Agent Datadog. Pour l’activer, consultez la section Collecte de logs avec Kubernetes.

ParamètreValeur
<CONFIG_LOG>{"source": "hive", "service": "<NOM_SERVICE>", "log_processing_rules":{"type":"multi_line","name":"new_log_start_with_date", "pattern":"\d{4}\-\d{2}\-\d{2}"}}

Validation

Lancez la sous-commande status de l’Agent et cherchez Hive dans la section Checks.

Données collectées

Métriques

hive.metastore.open_connections
(gauge)
Number of connection opened.
Shown as connection
hive.metastore.db.created
(rate)
Total number of created database.
Shown as item
hive.metastore.db.deleted
(rate)
Total number of deleted database.
Shown as item
hive.metastore.db.init
(rate)
Number of initialized database.
Shown as item
hive.metastore.table.created
(rate)
Total number of created table.
Shown as table
hive.metastore.table.deleted
(rate)
Total number of deleted table.
Shown as table
hive.metastore.table.init
(rate)
Number of initialized table.
Shown as table
hive.metastore.partition.created
(rate)
Total number of created partition.
Shown as item
hive.metastore.partition.deleted
(rate)
Total number of deleted partition.
Shown as item
hive.metastore.partition.init
(rate)
Number of initialized partition.
Shown as item
hive.metastore.directsql_errors
(gauge)
Number of SQL error.
Shown as unit
hive.metastore.memory.heap.init
(gauge)
Memory used at the initialization by the metastore.
Shown as byte
hive.metastore.memory.heap.used
(gauge)
Memory used by the metastore.
Shown as byte
hive.metastore.memory.heap.max
(gauge)
Maximum memory that can be used by the metastore.
Shown as byte
hive.metastore.memory.heap.committed
(gauge)
Committed memory for the metastore.
Shown as byte
hive.metastore.memory.non_heap.init
(gauge)
Memory used at the initialization by the metastore.
Shown as byte
hive.metastore.memory.non_heap.used
(gauge)
Memory used by the metastore.
Shown as byte
hive.metastore.memory.non_heap.max
(gauge)
Maximum memory that can be used by the metastore.
Shown as byte
hive.metastore.memory.non_heap.committed
(gauge)
Committed memory for the metastore
Shown as byte
hive.metastore.memory.total.init
(gauge)
Total memory at the initialization by the metastore.
Shown as byte
hive.metastore.memory.total.used
(gauge)
Total memory used by the metastore.
Shown as byte
hive.metastore.memory.total.max
(gauge)
Total maximum memory that can be used for the metastore.
Shown as byte
hive.metastore.memory.total.committed
(gauge)
Total committed memory for the metastore.
Shown as byte
hive.metastore.api.create_table
(rate)
API call to create a table.
Shown as task
hive.metastore.api.create_table.active_call
(gauge)
Active API call to create a table.
Shown as unit
hive.metastore.api.get_table
(rate)
API call to get a table.
Shown as task
hive.metastore.api.get_table.active_call
(gauge)
Active API call to get a table.
Shown as unit
hive.metastore.api.drop_table
(rate)
API call to drop a table.
Shown as task
hive.metastore.api.drop_table.active_call
(gauge)
Active API call to drop a table.
Shown as unit
hive.metastore.api.init
(rate)
API initialization.
Shown as task
hive.metastore.api.init.active_call
(gauge)
Active API initialization.
Shown as unit
hive.metastore.api.get_all_databases
(rate)
API call to get all databases.
Shown as task
hive.metastore.api.get_all_databases.active_call
(gauge)
Active API call to get all databases.
Shown as unit
hive.metastore.api.get_database
(rate)
API call to get a database.
Shown as task
hive.metastore.api.get_database.active_call
(gauge)
Active API call to get a database.
Shown as unit
hive.metastore.api.get_all_tables
(rate)
API call to get all tables.
Shown as task
hive.metastore.api.get_all_tables.active_call
(gauge)
Active API call to get all tables.
Shown as unit
hive.metastore.api.shutdown
(rate)
API shutdown.
Shown as task
hive.metastore.api.shutdown.active_call
(gauge)
Active API shutdown.
Shown as unit
hive.metastore.api.flushcache
(rate)
API flushcache.
Shown as task
hive.metastore.api.flushcache.active_call
(gauge)
Active API flushcache.
Shown as unit
hive.metastore.api.alter_table
(rate)
API call to alter table.
Shown as task
hive.metastore.api.alter_table.active_call
(gauge)
Active API call to alter table.
Shown as unit
hive.metastore.api.get_all_functions
(rate)
API call to get all functions.
Shown as task
hive.metastore.api.get_all_functions.active_call
(gauge)
Active API call to get all functions.
Shown as unit
hive.metastore.api.get_table_req
(rate)

Shown as task
hive.metastore.api.get_table_req.active_call
(gauge)

Shown as unit
hive.server.open_operations
(gauge)
Operation opened in the HiveServer2.
Shown as operation
hive.server.session.active
(gauge)
Number of active session.
Shown as session
hive.server.session.active.time_mean
(gauge)
Average time a session has been active.
Shown as millisecond
hive.server.session.open
(gauge)
Number of opened session.
Shown as session
hive.server.session.open.time_mean
(gauge)
Average time a session has been opened.
Shown as millisecond
hive.server.queries.submitted.active_call
(gauge)
Active submitted queries.
Shown as query
hive.server.queries.submitted.max
(gauge)
Max time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.mean
(gauge)
Mean time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.min
(gauge)
Min time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.75percentile
(gauge)
P75 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.95percentile
(gauge)
P95 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.count
(rate)
Number of submitted query.
Shown as query
hive.server.queries.submitted.meanrate
(gauge)
Submitted query rate.
Shown as query
hive.server.queries.compiling.active_call
(gauge)
Active compiling queries.
Shown as query
hive.server.queries.compiling.max
(gauge)
Max time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.mean
(gauge)
Mean time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.min
(gauge)
Min time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.75percentile
(gauge)
P75 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.95percentile
(gauge)
P95 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.count
(rate)
Number of compiled query.
Shown as query
hive.server.queries.compiling.meanrate
(gauge)
Compiling query rate.
Shown as query
hive.server.api.queries.executing.active_call
(gauge)
Active executing queries.
Shown as query
hive.server.queries.executing.max
(gauge)
Max time for executing a query.
Shown as millisecond
hive.server.queries.executing.mean
(gauge)
Mean time for executing a query.
Shown as millisecond
hive.server.queries.executing.min
(gauge)
Min time for executing a query.
Shown as millisecond
hive.server.queries.executing.75percentile
(gauge)
P75 time for executing a query.
Shown as millisecond
hive.server.queries.executing.95percentile
(gauge)
P95 time for executing a query.
Shown as millisecond
hive.server.queries.executing.count
(rate)
Number of executed queries.
Shown as query
hive.server.queries.executing.meanrate
(gauge)
Executing query rate.
Shown as query
hive.server.queries.succeeded.count
(rate)
Number of succeeded queries.
Shown as query
hive.server.queries.succeeded.meanrate
(gauge)
Succeeded queries rate.
Shown as query
hive.server.api.sql_operation.pending.active_call
(gauge)
Active pending sql operation.
Shown as operation
hive.server.api.sql_operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.count
(rate)
Number of sql operation in pending state.
Shown as operation
hive.server.api.sql_operation.pending.meanrate
(gauge)
Pending sql operation rate.
Shown as operation
hive.server.api.sql_operation.running.active_call
(gauge)
Active running sql operation.
Shown as operation
hive.server.api.sql_operation.running.max
(gauge)
Max time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.mean
(gauge)
Mean time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.min
(gauge)
Min time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.75percentile
(gauge)
P75 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.95percentile
(gauge)
P95 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.count
(rate)
Number of sql operation in running state.
Shown as operation
hive.server.api.sql_operation.running.meanrate
(gauge)
Running sql operation rate.
Shown as operation
hive.server.sql_operation.completed.closed
(rate)
Number of closed sql operation.
Shown as operation
hive.server.sql_operation.completed.finished
(rate)
Number of finished sql operation.
Shown as operation
hive.server.sql_operation.user.active
(gauge)
Number of active user.
Shown as user
hive.server.api.operation.initialized.active_call
(gauge)
Active initialized operation.
Shown as user
hive.server.api.operation.initialized.max
(gauge)
Max time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.mean
(gauge)
Mean time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.min
(gauge)
Min time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.75percentile
(gauge)
P75 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.95percentile
(gauge)
P95 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.count
(rate)
Number of operation initialized.
Shown as operation
hive.server.api.operation.initialized.meanrate
(gauge)
Operation initialization rate.
Shown as operation
hive.server.api.operation.pending.active_call
(gauge)
Active pending operation.
Shown as operation
hive.server.api.operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.count
(rate)
Number of operation in pending state.
Shown as operation
hive.server.api.operation.pending.meanrate
(gauge)
Operation pending rate.
Shown as operation
hive.server.api.operation.running.active_call
(gauge)
Active running operation.
Shown as operation
hive.server.api.operation.running.max
(gauge)
Max time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.mean
(gauge)
Mean time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.min
(gauge)
Min time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.75percentile
(gauge)
P75 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.95percentile
(gauge)
P95 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.count
(rate)
Number of operation in running state.
Shown as millisecond
hive.server.api.operation.running.meanrate
(gauge)
Operation running rate.
Shown as operation
hive.server.operation.completed.finished
(rate)
Number of finished operation.
Shown as operation
hive.server.operation.completed.closed
(rate)
Number of closed operation.
Shown as operation
hive.server.memory.heap.init
(gauge)
Memory used at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.heap.used
(gauge)
Memory used by the HiveServer2.
Shown as byte
hive.server.memory.heap.max
(gauge)
Maximum memory that can be used by the HiveServer2.
Shown as byte
hive.server.memory.heap.committed
(gauge)
Committed memory for the HiveServer2.
Shown as byte
hive.server.memory.non_heap.init
(gauge)
Memory used at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.used
(gauge)
Memory used by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.max
(gauge)
Maximum memory that can be used by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.committed
(gauge)
Committed memory for the HiveServer2
Shown as byte
hive.server.memory.total.init
(gauge)
Total memory at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.total.used
(gauge)
Total memory used by the HiveServer2.
Shown as byte
hive.server.memory.total.max
(gauge)
Total maximum memory that can be used for the HiveServer2.
Shown as byte
hive.server.memory.total.committed
(gauge)
Total committed memory for the HiveServer2.
Shown as byte

Checks de service

hive.can_connect :
Renvoie CRITICAL si l’Agent ne parvient pas à se connecter à l’instance HiveServer2 ou Hive Metastore qu’il surveille et d’y recueillir des métriques. Si ce n’est pas le cas, renvoie OK.

Événements

Le check Hive n’inclut aucun événement.

Dépannage

Besoin d’aide ? Contactez l’assistance Datadog.