Supported OS

Versión de la integración2.1.0

Información general

Este check monitoriza dos partes de Hive: Hive Metastore y HiveServer2.

Configuración

Instalación

El check de Hive está incluido en el paquete del Datadog Agent. No es necesaria ninguna instalación adicional en tu servidor.

Configuración

Configuración de Hive

  1. Edita el archivo de configuración de Hive en HIVE_HOME/conf/hive-site.xml para habilitar las métricas de Hive Metastore y HiveServer2 añadiendo estas propiedades:

    <property>
      <name>hive.metastore.metrics.enabled</name>
      <value>true</value>
    </property>
    <property>
      <name>hive.server2.metrics.enabled</name>
      <value>true</value>
    </property>
    
  2. Habilita una conexión JMX remota para HiveServer2 o para Hive Metastore. Por ejemplo, configura la variable de entorno HADOOP_CLIENT_OPTS:

    export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.port=8808"
    

    A continuación, reinicia HiveServer2 o Hive Metastore. Hive Metastore y HiveServer2 no pueden compartir la misma conexión JMX.

Host

Para configurar este check para un Agent que se ejecuta en un host:

Sigue las instrucciones de abajo para configurar este check para un Agent que se ejecuta en un host. En el caso de entornos en contenedores, consulta la sección En contenedores.

Recopilación de métricas
  1. Edita el archivo hive.d/conf.yaml, que se encuentra en la carpeta conf.d/ en la raíz del directorio de configuración de tu Agent, para empezar a recopilar los datos de rendimiento de tu Hive. Para conocer todas las opciones de configuración disponibles, consulta el hive.d/conf.yaml de ejemplo.

    Este check tiene un límite de 350 métricas por instancia. El número de métricas devueltas se indica en la página de estado. Puedes especificar las métricas que te interesan, editando la siguiente configuración. Para saber cómo personalizar las métricas que se van a recopilar, consulta la documentación sobre checks JMX para obtener instrucciones más detalladas. Si necesitas monitorizar más métricas, ponte en contacto con el servicio de asistencia de Datadog.

  2. Reinicia el Agent.

Recopilación de logs

Disponible para la versión 6.0 o posteriores del Agent

  1. La recopilación de logs se encuentra deshabilitada de manera predeterminada en el Datadog Agent; debes habilitarla en el archivo datadog.yaml:

    logs_enabled: true
    
  2. Añade este bloque de configuración a tu archivo hive.d/conf.yaml para empezar a recopilar tus logs de Hive:

      logs:
        - type: file
          path: /tmp/<USER>/hive.log
          source: hive
          service: '<SERVICE_NAME>'
          log_processing_rules:
            - type: multi_line
              name: new_log_start_with_date
              pattern: \d{4}\-\d{2}\-\d{2}
    

    Cambia los valores de los parámetros path y service y configúralos para tu entorno. Consulta el hive.d/conf.yaml de ejemplo para conocer todas las opciones de configuración disponibles.

  3. Reinicia el Agent.

En contenedores

Para entornos en contenedores, consulta las plantillas de integración de Autodiscovery para obtener orientación sobre la aplicación de los parámetros que se indican a continuación.

Recopilación de métricas

Para recopilar métricas con la integración de Datadog y Hive, consulta la guía Autodiscovery con JMX.

Recopilación de logs

Disponible para la versión 6.0 o posteriores del Agent

La recopilación de logs se encuentra deshabilitada de manera predeterminada en el Datadog Agent. Para habilitarla, consulta la recopilación de logs de Kubernetes.

ParámetroValor
<LOG_CONFIG>{"source": "hive", "service": "<SERVICE_NAME>", "log_processing_rules":{"type":"multi_line","name":"new_log_start_with_date", "pattern":"\d{4}\-\d{2}\-\d{2}"}}

Validación

Ejecuta el subcomando de estado del Agent y busca Hive en la sección Checks.

Datos recopilados

Métricas

hive.metastore.api.alter_table
(rate)
API call to alter table.
Shown as task
hive.metastore.api.alter_table.active_call
(gauge)
Active API call to alter table.
Shown as unit
hive.metastore.api.create_table
(rate)
API call to create a table.
Shown as task
hive.metastore.api.create_table.active_call
(gauge)
Active API call to create a table.
Shown as unit
hive.metastore.api.drop_table
(rate)
API call to drop a table.
Shown as task
hive.metastore.api.drop_table.active_call
(gauge)
Active API call to drop a table.
Shown as unit
hive.metastore.api.flushcache
(rate)
API flushcache.
Shown as task
hive.metastore.api.flushcache.active_call
(gauge)
Active API flushcache.
Shown as unit
hive.metastore.api.get_all_databases
(rate)
API call to get all databases.
Shown as task
hive.metastore.api.get_all_databases.active_call
(gauge)
Active API call to get all databases.
Shown as unit
hive.metastore.api.get_all_functions
(rate)
API call to get all functions.
Shown as task
hive.metastore.api.get_all_functions.active_call
(gauge)
Active API call to get all functions.
Shown as unit
hive.metastore.api.get_all_tables
(rate)
API call to get all tables.
Shown as task
hive.metastore.api.get_all_tables.active_call
(gauge)
Active API call to get all tables.
Shown as unit
hive.metastore.api.get_database
(rate)
API call to get a database.
Shown as task
hive.metastore.api.get_database.active_call
(gauge)
Active API call to get a database.
Shown as unit
hive.metastore.api.get_table
(rate)
API call to get a table.
Shown as task
hive.metastore.api.get_table.active_call
(gauge)
Active API call to get a table.
Shown as unit
hive.metastore.api.get_table_req
(rate)

Shown as task
hive.metastore.api.get_table_req.active_call
(gauge)

Shown as unit
hive.metastore.api.init
(rate)
API initialization.
Shown as task
hive.metastore.api.init.active_call
(gauge)
Active API initialization.
Shown as unit
hive.metastore.api.shutdown
(rate)
API shutdown.
Shown as task
hive.metastore.api.shutdown.active_call
(gauge)
Active API shutdown.
Shown as unit
hive.metastore.db.created
(rate)
Total number of created database.
Shown as item
hive.metastore.db.deleted
(rate)
Total number of deleted database.
Shown as item
hive.metastore.db.init
(rate)
Number of initialized database.
Shown as item
hive.metastore.directsql_errors
(gauge)
Number of SQL error.
Shown as unit
hive.metastore.open_connections
(gauge)
Number of connection opened.
Shown as connection
hive.metastore.partition.created
(rate)
Total number of created partition.
Shown as item
hive.metastore.partition.deleted
(rate)
Total number of deleted partition.
Shown as item
hive.metastore.partition.init
(rate)
Number of initialized partition.
Shown as item
hive.metastore.table.created
(rate)
Total number of created table.
Shown as table
hive.metastore.table.deleted
(rate)
Total number of deleted table.
Shown as table
hive.metastore.table.init
(rate)
Number of initialized table.
Shown as table
hive.server.api.operation.initialized.75percentile
(gauge)
P75 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.95percentile
(gauge)
P95 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.active_call
(gauge)
Active initialized operation.
Shown as user
hive.server.api.operation.initialized.count
(rate)
Number of operation initialized.
Shown as operation
hive.server.api.operation.initialized.max
(gauge)
Max time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.mean
(gauge)
Mean time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.meanrate
(gauge)
Operation initialization rate.
Shown as operation
hive.server.api.operation.initialized.min
(gauge)
Min time to init an operation.
Shown as millisecond
hive.server.api.operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.active_call
(gauge)
Active pending operation.
Shown as operation
hive.server.api.operation.pending.count
(rate)
Number of operation in pending state.
Shown as operation
hive.server.api.operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.meanrate
(gauge)
Operation pending rate.
Shown as operation
hive.server.api.operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.75percentile
(gauge)
P75 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.95percentile
(gauge)
P95 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.active_call
(gauge)
Active running operation.
Shown as operation
hive.server.api.operation.running.count
(rate)
Number of operation in running state.
Shown as millisecond
hive.server.api.operation.running.max
(gauge)
Max time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.mean
(gauge)
Mean time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.meanrate
(gauge)
Operation running rate.
Shown as operation
hive.server.api.operation.running.min
(gauge)
Min time in running state for an sql operation.
Shown as millisecond
hive.server.api.queries.executing.active_call
(gauge)
Active executing queries.
Shown as query
hive.server.api.sql_operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.active_call
(gauge)
Active pending sql operation.
Shown as operation
hive.server.api.sql_operation.pending.count
(rate)
Number of sql operation in pending state.
Shown as operation
hive.server.api.sql_operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.meanrate
(gauge)
Pending sql operation rate.
Shown as operation
hive.server.api.sql_operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.75percentile
(gauge)
P75 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.95percentile
(gauge)
P95 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.active_call
(gauge)
Active running sql operation.
Shown as operation
hive.server.api.sql_operation.running.count
(rate)
Number of sql operation in running state.
Shown as operation
hive.server.api.sql_operation.running.max
(gauge)
Max time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.mean
(gauge)
Mean time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.meanrate
(gauge)
Running sql operation rate.
Shown as operation
hive.server.api.sql_operation.running.min
(gauge)
Min time for running state for a sql operation.
Shown as millisecond
hive.server.memory.heap.committed
(gauge)
Committed memory for the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.heap.init
(gauge)
Memory used at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.heap.max
(gauge)
Maximum memory that can be used by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.heap.used
(gauge)
Memory used by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.non_heap.committed
(gauge)
Committed memory for the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.non_heap.init
(gauge)
Memory used at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.non_heap.max
(gauge)
Maximum memory that can be used by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.non_heap.used
(gauge)
Memory used by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.total.committed
(gauge)
Total committed memory for the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.total.init
(gauge)
Total memory at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.total.max
(gauge)
Total maximum memory that can be used for the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.memory.total.used
(gauge)
Total memory used by the HiveServer2 or Metastore. Depends on the connected instance type
Shown as byte
hive.server.open_operations
(gauge)
Operation opened in the HiveServer2.
Shown as operation
hive.server.operation.completed.closed
(rate)
Number of closed operation.
Shown as operation
hive.server.operation.completed.finished
(rate)
Number of finished operation.
Shown as operation
hive.server.queries.compiling.75percentile
(gauge)
P75 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.95percentile
(gauge)
P95 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.active_call
(gauge)
Active compiling queries.
Shown as query
hive.server.queries.compiling.count
(rate)
Number of compiled query.
Shown as query
hive.server.queries.compiling.max
(gauge)
Max time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.mean
(gauge)
Mean time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.meanrate
(gauge)
Compiling query rate.
Shown as query
hive.server.queries.compiling.min
(gauge)
Min time for compiling a query.
Shown as millisecond
hive.server.queries.executing.75percentile
(gauge)
P75 time for executing a query.
Shown as millisecond
hive.server.queries.executing.95percentile
(gauge)
P95 time for executing a query.
Shown as millisecond
hive.server.queries.executing.count
(rate)
Number of executed queries.
Shown as query
hive.server.queries.executing.max
(gauge)
Max time for executing a query.
Shown as millisecond
hive.server.queries.executing.mean
(gauge)
Mean time for executing a query.
Shown as millisecond
hive.server.queries.executing.meanrate
(gauge)
Executing query rate.
Shown as query
hive.server.queries.executing.min
(gauge)
Min time for executing a query.
Shown as millisecond
hive.server.queries.submitted.75percentile
(gauge)
P75 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.95percentile
(gauge)
P95 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.active_call
(gauge)
Active submitted queries.
Shown as query
hive.server.queries.submitted.count
(rate)
Number of submitted query.
Shown as query
hive.server.queries.submitted.max
(gauge)
Max time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.mean
(gauge)
Mean time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.meanrate
(gauge)
Submitted query rate.
Shown as query
hive.server.queries.submitted.min
(gauge)
Min time for a submitted query.
Shown as millisecond
hive.server.queries.succeeded.count
(rate)
Number of succeeded queries.
Shown as query
hive.server.queries.succeeded.meanrate
(gauge)
Succeeded queries rate.
Shown as query
hive.server.session.active
(gauge)
Number of active session.
Shown as session
hive.server.session.active.time_mean
(gauge)
Average time a session has been active.
Shown as millisecond
hive.server.session.open
(gauge)
Number of opened session.
Shown as session
hive.server.session.open.time_mean
(gauge)
Average time a session has been opened.
Shown as millisecond
hive.server.sql_operation.completed.closed
(rate)
Number of closed sql operation.
Shown as operation
hive.server.sql_operation.completed.finished
(rate)
Number of finished sql operation.
Shown as operation
hive.server.sql_operation.user.active
(gauge)
Number of active user.
Shown as user

Eventos

El check de Hive no incluye eventos.

Checks de servicio

hive.can_connect
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored HiveServer2/Hive Metastore instance, WARNING if no metrics are collected, and OK otherwise.
Statuses: ok, critical, warning

Solucionar problemas

¿Necesitas ayuda? Ponte en contacto con el servicio de asistencia de Datadog.