Hive

Supported OS Linux Mac OS Windows

Intégration1.8.1

Présentation

Ce check surveille deux composants de Hive : Hive Metastore et HiveServer2.

Configuration

Installation

Le check Hive est inclus avec le package de l’Agent Datadog. Vous n’avez donc rien d’autre à installer sur votre serveur.

Configuration

Configuration de Hive

Modifiez le fichier de configuration de Hive dans HIVE_HOME/conf/hive-site.xml pour activer les métriques de Hive Metastore et de HiveServer2 en ajoutant les propriétés suivantes :

<property>
  <name>hive.metastore.metrics.enabled</name>
  <value>true</value>
</property>
<property>
  <name>hive.server2.metrics.enabled</name>
  <value>true</value>
</property>

Activez une connexion JMX à distance pour HiveServer2 et/ou Hive Metastore. Définissez par exemple la variable d’environnement HADOOP_CLIENT_OPTS :
```
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.port=8808"
```
Redémarrez ensuite HiveServer2 ou Hive Metastore. Hive Metastore et HiveServer2 ne peuvent pas partager la même connexion JMX.

Host

Pour configurer ce check lorsque l’Agent est exécuté sur un host :

Suivez les instructions ci-dessous pour configurer ce check lorsque l’Agent est exécuté sur un host. Consultez la section Environnement conteneurisé pour la configuration dans un environnement conteneurisé.

Collecte de métriques

Modifiez le fichier hive.d/conf.yaml dans le dossier conf.d/ à la racine du répertoire de configuration de votre Agent pour commencer à recueillir vos données de performance Hive. Consultez le fichier d’exemple hive.d/conf.yaml pour découvrir toutes les options de configuration disponibles.
Ce check prévoit une limite de 350 métriques par instance. Le nombre de métriques renvoyées est indiqué sur la page d’informations. Choisissez les métriques qui vous intéressent en modifiant la configuration ci-dessous. Pour découvrir comment modifier la liste des métriques à recueillir, consultez la documentation relative aux checks JMX afin d’obtenir des instructions détaillées. Si vous souhaitez surveiller davantage de métriques, contactez l’assistance Datadog.
Redémarrez l’Agent.

Collecte de logs

Disponible à partir des versions > 6.0 de l’Agent

La collecte de logs est désactivée par défaut dans l’Agent Datadog. Vous devez l’activer dans datadog.yaml :
```
logs_enabled: true
```

Ajoutez ce bloc de configuration à votre fichier hive.d/conf.yaml pour commencer à recueillir vos logs Hive :

  logs:
    - type: file
      path: /tmp/<USER>/hive.log
      source: hive
      service: '<SERVICE_NAME>'
      log_processing_rules:
        - type: multi_line
          name: new_log_start_with_date
          pattern: \d{4}\-\d{2}\-\d{2}

Modifiez les valeurs des paramètres path et service et configurez-les pour votre environnement. Consultez le fichier d’exemple hive.d/conf.yaml pour découvrir toutes les options de configuration disponibles.

Redémarrez l’Agent.

Environnement conteneurisé

Consultez la documentation relative aux modèles d’intégration Autodiscovery pour découvrir comment appliquer les paramètres ci-dessous à un environnement conteneurisé.

Collecte de métriques

Pour recueillir des métriques avec l’intégration Datadog/Hive, consultez le guide Autodiscovery avec JMX.

Collecte de logs

Disponible à partir des versions > 6.0 de l’Agent

La collecte des logs est désactivée par défaut dans l’Agent Datadog. Pour l’activer, consultez la section Collecte de logs Kubernetes.

Paramètre	Valeur
`<CONFIG_LOG>`	`{"source": "hive", "service": "<NOM_SERVICE>", "log_processing_rules":{"type":"multi_line","name":"new_log_start_with_date", "pattern":"\d{4}\-\d{2}\-\d{2}"}}`

Validation

Lancez la sous-commande status de l’Agent et cherchez Hive dans la section Checks.

Données collectées

Métriques

hive.metastore.api.alter_table (rate)	API call to alter table. Shown as task
hive.metastore.api.alter_table.active_call (gauge)	Active API call to alter table. Shown as unit
hive.metastore.api.create_table (rate)	API call to create a table. Shown as task
hive.metastore.api.create_table.active_call (gauge)	Active API call to create a table. Shown as unit
hive.metastore.api.drop_table (rate)	API call to drop a table. Shown as task
hive.metastore.api.drop_table.active_call (gauge)	Active API call to drop a table. Shown as unit
hive.metastore.api.flushcache (rate)	API flushcache. Shown as task
hive.metastore.api.flushcache.active_call (gauge)	Active API flushcache. Shown as unit
hive.metastore.api.get_all_databases (rate)	API call to get all databases. Shown as task
hive.metastore.api.get_all_databases.active_call (gauge)	Active API call to get all databases. Shown as unit
hive.metastore.api.get_all_functions (rate)	API call to get all functions. Shown as task
hive.metastore.api.get_all_functions.active_call (gauge)	Active API call to get all functions. Shown as unit
hive.metastore.api.get_all_tables (rate)	API call to get all tables. Shown as task
hive.metastore.api.get_all_tables.active_call (gauge)	Active API call to get all tables. Shown as unit
hive.metastore.api.get_database (rate)	API call to get a database. Shown as task
hive.metastore.api.get_database.active_call (gauge)	Active API call to get a database. Shown as unit
hive.metastore.api.get_table (rate)	API call to get a table. Shown as task
hive.metastore.api.get_table.active_call (gauge)	Active API call to get a table. Shown as unit
hive.metastore.api.get_table_req (rate)	Shown as task
hive.metastore.api.get_table_req.active_call (gauge)	Shown as unit
hive.metastore.api.init (rate)	API initialization. Shown as task
hive.metastore.api.init.active_call (gauge)	Active API initialization. Shown as unit
hive.metastore.api.shutdown (rate)	API shutdown. Shown as task
hive.metastore.api.shutdown.active_call (gauge)	Active API shutdown. Shown as unit
hive.metastore.db.created (rate)	Total number of created database. Shown as item
hive.metastore.db.deleted (rate)	Total number of deleted database. Shown as item
hive.metastore.db.init (rate)	Number of initialized database. Shown as item
hive.metastore.directsql_errors (gauge)	Number of SQL error. Shown as unit
hive.metastore.open_connections (gauge)	Number of connection opened. Shown as connection
hive.metastore.partition.created (rate)	Total number of created partition. Shown as item
hive.metastore.partition.deleted (rate)	Total number of deleted partition. Shown as item
hive.metastore.partition.init (rate)	Number of initialized partition. Shown as item
hive.metastore.table.created (rate)	Total number of created table. Shown as table
hive.metastore.table.deleted (rate)	Total number of deleted table. Shown as table
hive.metastore.table.init (rate)	Number of initialized table. Shown as table
hive.server.api.operation.initialized.75percentile (gauge)	P75 time to init an operation. Shown as millisecond
hive.server.api.operation.initialized.95percentile (gauge)	P95 time to init an operation. Shown as millisecond
hive.server.api.operation.initialized.active_call (gauge)	Active initialized operation. Shown as user
hive.server.api.operation.initialized.count (rate)	Number of operation initialized. Shown as operation
hive.server.api.operation.initialized.max (gauge)	Max time to init an operation. Shown as millisecond
hive.server.api.operation.initialized.mean (gauge)	Mean time to init an operation. Shown as millisecond
hive.server.api.operation.initialized.meanrate (gauge)	Operation initialization rate. Shown as operation
hive.server.api.operation.initialized.min (gauge)	Min time to init an operation. Shown as millisecond
hive.server.api.operation.pending.75percentile (gauge)	P75 time in pending state for an sql operation. Shown as millisecond
hive.server.api.operation.pending.95percentile (gauge)	P95 time in pending state for an sql operation. Shown as millisecond
hive.server.api.operation.pending.active_call (gauge)	Active pending operation. Shown as operation
hive.server.api.operation.pending.count (rate)	Number of operation in pending state. Shown as operation
hive.server.api.operation.pending.max (gauge)	Max time in pending state for an sql operation. Shown as millisecond
hive.server.api.operation.pending.mean (gauge)	Mean time in pending state for an sql operation. Shown as millisecond
hive.server.api.operation.pending.meanrate (gauge)	Operation pending rate. Shown as operation
hive.server.api.operation.pending.min (gauge)	Min time in pending state for an sql operation. Shown as millisecond
hive.server.api.operation.running.75percentile (gauge)	P75 time in running state for an sql operation. Shown as millisecond
hive.server.api.operation.running.95percentile (gauge)	P95 time in running state for an sql operation. Shown as millisecond
hive.server.api.operation.running.active_call (gauge)	Active running operation. Shown as operation
hive.server.api.operation.running.count (rate)	Number of operation in running state. Shown as millisecond
hive.server.api.operation.running.max (gauge)	Max time in running state for an sql operation. Shown as millisecond
hive.server.api.operation.running.mean (gauge)	Mean time in running state for an sql operation. Shown as millisecond
hive.server.api.operation.running.meanrate (gauge)	Operation running rate. Shown as operation
hive.server.api.operation.running.min (gauge)	Min time in running state for an sql operation. Shown as millisecond
hive.server.api.queries.executing.active_call (gauge)	Active executing queries. Shown as query
hive.server.api.sql_operation.pending.75percentile (gauge)	P75 time in pending state for an sql operation. Shown as millisecond
hive.server.api.sql_operation.pending.95percentile (gauge)	P95 time in pending state for an sql operation. Shown as millisecond
hive.server.api.sql_operation.pending.active_call (gauge)	Active pending sql operation. Shown as operation
hive.server.api.sql_operation.pending.count (rate)	Number of sql operation in pending state. Shown as operation
hive.server.api.sql_operation.pending.max (gauge)	Max time in pending state for an sql operation. Shown as millisecond
hive.server.api.sql_operation.pending.mean (gauge)	Mean time in pending state for an sql operation. Shown as millisecond
hive.server.api.sql_operation.pending.meanrate (gauge)	Pending sql operation rate. Shown as operation
hive.server.api.sql_operation.pending.min (gauge)	Min time in pending state for an sql operation. Shown as millisecond
hive.server.api.sql_operation.running.75percentile (gauge)	P75 time for running state for a sql operation. Shown as millisecond
hive.server.api.sql_operation.running.95percentile (gauge)	P95 time for running state for a sql operation. Shown as millisecond
hive.server.api.sql_operation.running.active_call (gauge)	Active running sql operation. Shown as operation
hive.server.api.sql_operation.running.count (rate)	Number of sql operation in running state. Shown as operation
hive.server.api.sql_operation.running.max (gauge)	Max time for running state for a sql operation. Shown as millisecond
hive.server.api.sql_operation.running.mean (gauge)	Mean time for running state for a sql operation. Shown as millisecond
hive.server.api.sql_operation.running.meanrate (gauge)	Running sql operation rate. Shown as operation
hive.server.api.sql_operation.running.min (gauge)	Min time for running state for a sql operation. Shown as millisecond
hive.server.memory.heap.committed (gauge)	Committed memory for the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.heap.init (gauge)	Memory used at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.heap.max (gauge)	Maximum memory that can be used by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.heap.used (gauge)	Memory used by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.non_heap.committed (gauge)	Committed memory for the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.non_heap.init (gauge)	Memory used at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.non_heap.max (gauge)	Maximum memory that can be used by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.non_heap.used (gauge)	Memory used by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.total.committed (gauge)	Total committed memory for the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.total.init (gauge)	Total memory at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.total.max (gauge)	Total maximum memory that can be used for the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.memory.total.used (gauge)	Total memory used by the HiveServer2 or Metastore. Depends on the connected instance type Shown as byte
hive.server.open_operations (gauge)	Operation opened in the HiveServer2. Shown as operation
hive.server.operation.completed.closed (rate)	Number of closed operation. Shown as operation
hive.server.operation.completed.finished (rate)	Number of finished operation. Shown as operation
hive.server.queries.compiling.75percentile (gauge)	P75 time for compiling a query. Shown as millisecond
hive.server.queries.compiling.95percentile (gauge)	P95 time for compiling a query. Shown as millisecond
hive.server.queries.compiling.active_call (gauge)	Active compiling queries. Shown as query
hive.server.queries.compiling.count (rate)	Number of compiled query. Shown as query
hive.server.queries.compiling.max (gauge)	Max time for compiling a query. Shown as millisecond
hive.server.queries.compiling.mean (gauge)	Mean time for compiling a query. Shown as millisecond
hive.server.queries.compiling.meanrate (gauge)	Compiling query rate. Shown as query
hive.server.queries.compiling.min (gauge)	Min time for compiling a query. Shown as millisecond
hive.server.queries.executing.75percentile (gauge)	P75 time for executing a query. Shown as millisecond
hive.server.queries.executing.95percentile (gauge)	P95 time for executing a query. Shown as millisecond
hive.server.queries.executing.count (rate)	Number of executed queries. Shown as query
hive.server.queries.executing.max (gauge)	Max time for executing a query. Shown as millisecond
hive.server.queries.executing.mean (gauge)	Mean time for executing a query. Shown as millisecond
hive.server.queries.executing.meanrate (gauge)	Executing query rate. Shown as query
hive.server.queries.executing.min (gauge)	Min time for executing a query. Shown as millisecond
hive.server.queries.submitted.75percentile (gauge)	P75 time for a submitted query. Shown as millisecond
hive.server.queries.submitted.95percentile (gauge)	P95 time for a submitted query. Shown as millisecond
hive.server.queries.submitted.active_call (gauge)	Active submitted queries. Shown as query
hive.server.queries.submitted.count (rate)	Number of submitted query. Shown as query
hive.server.queries.submitted.max (gauge)	Max time for a submitted query. Shown as millisecond
hive.server.queries.submitted.mean (gauge)	Mean time for a submitted query. Shown as millisecond
hive.server.queries.submitted.meanrate (gauge)	Submitted query rate. Shown as query
hive.server.queries.submitted.min (gauge)	Min time for a submitted query. Shown as millisecond
hive.server.queries.succeeded.count (rate)	Number of succeeded queries. Shown as query
hive.server.queries.succeeded.meanrate (gauge)	Succeeded queries rate. Shown as query
hive.server.session.active (gauge)	Number of active session. Shown as session
hive.server.session.active.time_mean (gauge)	Average time a session has been active. Shown as millisecond
hive.server.session.open (gauge)	Number of opened session. Shown as session
hive.server.session.open.time_mean (gauge)	Average time a session has been opened. Shown as millisecond
hive.server.sql_operation.completed.closed (rate)	Number of closed sql operation. Shown as operation
hive.server.sql_operation.completed.finished (rate)	Number of finished sql operation. Shown as operation
hive.server.sql_operation.user.active (gauge)	Number of active user. Shown as user

Événements

Le check Hive n’inclut aucun événement.

Checks de service

hive.can_connect
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored HiveServer2/Hive Metastore instance, WARNING if no metrics are collected, and OK otherwise.
Statuses: ok, critical, warning

Dépannage

Besoin d’aide ? Contactez l’assistance Datadog.