New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Hive

Agent Check Agent Check

Supported OS: Linux Mac OS Windows

Overview

This check monitors two parts of Hive: Hive Metastore and HiveServer2.

Setup

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Installation

The Hive check is included in the Datadog Agent package. No additional installation is needed on your server.

Configuration

  1. Edit the Hive configuration file in HIVE_HOME/conf/hive-site.xml to enable the Hive Metastore and HiveServer2 metrics by adding these properties:

    <property>
      <name>hive.metastore.metrics.enabled</name>
      <value>true</value>
    </property>
    <property>
      <name>hive.server2.metrics.enabled</name>
      <value>true</value>
    </property>
    
  2. Enable a JMX remote connection for the HiveServer2 and/or for the Hive Metastore. For example, set the HADOOP_CLIENT_OPTS environment variable:

    export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.port=8808"
    

    Then restart the HiveServer2 or the Hive Metastore. Hive Metastore and HiveServer2 cannot share the same JMX connection.

  3. Edit the hive.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your hive performance data. See the sample hive.d/conf.yaml for all available configuration options.

    This check has a limit of 350 metrics per instance. The number of returned metrics is indicated in the info page. You can specify the metrics you are interested in by editing the configuration below. To learn how to customize the metrics to collect, visit the JMX Checks documentation for more detailed instructions. If you need to monitor more metrics, contact Datadog support.

  4. Restart the Agent.

Log collection

Available for Agent >6.0

  1. Collecting logs is disabled by default in the Datadog Agent, you need to enable it in datadog.yaml:

      logs_enabled: true
  2. Add this configuration block to your hive.d/conf.yaml file to start collecting your Hive logs:

      logs:
        - type: file
          path: /tmp/<USER>/hive.log
          source: hive
          service: <SERVICE_NAME>
          log_processing_rules:
            - type: multi_line
              name: new_log_start_with_date
              pattern: \d{4}\-\d{2}\-\d{2}
    

    Change the path and service parameter values and configure them for your environment. See the sample hive.d/conf.yaml for all available configuration options.

  3. Restart the Agent.

Validation

Run the Agent’s status subcommand and look for Hive under the Checks section.

Data Collected

Metrics

hive.metastore.open_connections
(gauge)
Number of connection opened.
Shown as connection
hive.metastore.db.created
(rate)
Total number of created database.
Shown as item
hive.metastore.db.deleted
(rate)
Total number of deleted database.
Shown as item
hive.metastore.db.init
(rate)
Number of initialized database.
Shown as item
hive.metastore.table.created
(rate)
Total number of created table.
Shown as table
hive.metastore.table.deleted
(rate)
Total number of deleted table.
Shown as table
hive.metastore.table.init
(rate)
Number of initialized table.
Shown as table
hive.metastore.partition.created
(rate)
Total number of created partition.
Shown as item
hive.metastore.partition.deleted
(rate)
Total number of deleted partition.
Shown as item
hive.metastore.partition.init
(rate)
Number of initialized partition.
Shown as item
hive.metastore.directsql_errors
(gauge)
Number of SQL error.
Shown as unit
hive.metastore.memory.heap.init
(gauge)
Memory used at the initialization by the metastore.
Shown as byte
hive.metastore.memory.heap.used
(gauge)
Memory used by the metastore.
Shown as byte
hive.metastore.memory.heap.max
(gauge)
Maximum memory that can be used by the metastore.
Shown as byte
hive.metastore.memory.heap.committed
(gauge)
Committed memory for the metastore.
Shown as byte
hive.metastore.memory.non_heap.init
(gauge)
Memory used at the initialization by the metastore.
Shown as byte
hive.metastore.memory.non_heap.used
(gauge)
Memory used by the metastore.
Shown as byte
hive.metastore.memory.non_heap.max
(gauge)
Maximum memory that can be used by the metastore.
Shown as byte
hive.metastore.memory.non_heap.committed
(gauge)
Committed memory for the metastore
Shown as byte
hive.metastore.memory.total.init
(gauge)
Total memory at the initialization by the metastore.
Shown as byte
hive.metastore.memory.total.used
(gauge)
Total memory used by the metastore.
Shown as byte
hive.metastore.memory.total.max
(gauge)
Total maximum memory that can be used for the metastore.
Shown as byte
hive.metastore.memory.total.committed
(gauge)
Total committed memory for the metastore.
Shown as byte
hive.metastore.api.create_table
(rate)
API call to create a table.
Shown as task
hive.metastore.api.create_table.active_call
(gauge)
Active API call to create a table.
Shown as unit
hive.metastore.api.get_table
(rate)
API call to get a table.
Shown as task
hive.metastore.api.get_table.active_call
(gauge)
Active API call to get a table.
Shown as unit
hive.metastore.api.drop_table
(rate)
API call to drop a table.
Shown as task
hive.metastore.api.drop_table.active_call
(gauge)
Active API call to drop a table.
Shown as unit
hive.metastore.api.init
(rate)
API initialization.
Shown as task
hive.metastore.api.init.active_call
(gauge)
Active API initialization.
Shown as unit
hive.metastore.api.get_all_databases
(rate)
API call to get all databases.
Shown as task
hive.metastore.api.get_all_databases.active_call
(gauge)
Active API call to get all databases.
Shown as unit
hive.metastore.api.get_database
(rate)
API call to get a database.
Shown as task
hive.metastore.api.get_database.active_call
(gauge)
Active API call to get a database.
Shown as unit
hive.metastore.api.get_all_tables
(rate)
API call to get all tables.
Shown as task
hive.metastore.api.get_all_tables.active_call
(gauge)
Active API call to get all tables.
Shown as unit
hive.metastore.api.shutdown
(rate)
API shutdown.
Shown as task
hive.metastore.api.shutdown.active_call
(gauge)
Active API shutdown.
Shown as unit
hive.metastore.api.flushcache
(rate)
API flushcache.
Shown as task
hive.metastore.api.flushcache.active_call
(gauge)
Active API flushcache.
Shown as unit
hive.metastore.api.alter_table
(rate)
API call to alter table.
Shown as task
hive.metastore.api.alter_table.active_call
(gauge)
Active API call to alter table.
Shown as unit
hive.metastore.api.get_all_functions
(rate)
API call to get all functions.
Shown as task
hive.metastore.api.get_all_functions.active_call
(gauge)
Active API call to get all functions.
Shown as unit
hive.metastore.api.get_table_req
(rate)

Shown as task
hive.metastore.api.get_table_req.active_call
(gauge)

Shown as unit
hive.server.open_operations
(gauge)
Operation opened in the HiveServer2.
Shown as operation
hive.server.session.active
(gauge)
Number of active session.
Shown as session
hive.server.session.active.time_mean
(gauge)
Average time a session has been active.
Shown as millisecond
hive.server.session.open
(gauge)
Number of opened session.
Shown as session
hive.server.session.open.time_mean
(gauge)
Average time a session has been opened.
Shown as millisecond
hive.server.queries.submitted.active_call
(gauge)
Active submitted queries.
Shown as query
hive.server.queries.submitted.max
(gauge)
Max time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.mean
(gauge)
Mean time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.min
(gauge)
Min time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.75percentile
(gauge)
P75 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.95percentile
(gauge)
P95 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.count
(rate)
Number of submitted query.
Shown as query
hive.server.queries.submitted.meanrate
(gauge)
Submitted query rate.
Shown as query
hive.server.queries.compiling.active_call
(gauge)
Active compiling queries.
Shown as query
hive.server.queries.compiling.max
(gauge)
Max time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.mean
(gauge)
Mean time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.min
(gauge)
Min time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.75percentile
(gauge)
P75 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.95percentile
(gauge)
P95 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.count
(rate)
Number of compiled query.
Shown as query
hive.server.queries.compiling.meanrate
(gauge)
Compiling query rate.
Shown as query
hive.server.api.queries.executing.active_call
(gauge)
Active executing queries.
Shown as query
hive.server.queries.executing.max
(gauge)
Max time for executing a query.
Shown as millisecond
hive.server.queries.executing.mean
(gauge)
Mean time for executing a query.
Shown as millisecond
hive.server.queries.executing.min
(gauge)
Min time for executing a query.
Shown as millisecond
hive.server.queries.executing.75percentile
(gauge)
P75 time for executing a query.
Shown as millisecond
hive.server.queries.executing.95percentile
(gauge)
P95 time for executing a query.
Shown as millisecond
hive.server.queries.executing.count
(rate)
Number of executed queries.
Shown as query
hive.server.queries.executing.meanrate
(gauge)
Executing query rate.
Shown as query
hive.server.queries.succeeded.count
(rate)
Number of succeeded queries.
Shown as query
hive.server.queries.succeeded.meanrate
(gauge)
Succeeded queries rate.
Shown as query
hive.server.api.sql_operation.pending.active_call
(gauge)
Active pending sql operation.
Shown as operation
hive.server.api.sql_operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.count
(rate)
Number of sql operation in pending state.
Shown as operation
hive.server.api.sql_operation.pending.meanrate
(gauge)
Pending sql operation rate.
Shown as operation
hive.server.api.sql_operation.running.active_call
(gauge)
Active running sql operation.
Shown as operation
hive.server.api.sql_operation.running.max
(gauge)
Max time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.mean
(gauge)
Mean time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.min
(gauge)
Min time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.75percentile
(gauge)
P75 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.95percentile
(gauge)
P95 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.count
(rate)
Number of sql operation in running state.
Shown as operation
hive.server.api.sql_operation.running.meanrate
(gauge)
Running sql operation rate.
Shown as operation
hive.server.sql_operation.completed.closed
(rate)
Number of closed sql operation.
Shown as operation
hive.server.sql_operation.completed.finished
(rate)
Number of finished sql operation.
Shown as operation
hive.server.sql_operation.user.active
(gauge)
Number of active user.
Shown as user
hive.server.api.operation.initialized.active_call
(gauge)
Active initialized operation.
Shown as user
hive.server.api.operation.initialized.max
(gauge)
Max time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.mean
(gauge)
Mean time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.min
(gauge)
Min time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.75percentile
(gauge)
P75 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.95percentile
(gauge)
P95 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.count
(rate)
Number of operation initialized.
Shown as operation
hive.server.api.operation.initialized.meanrate
(gauge)
Operation initialization rate.
Shown as operation
hive.server.api.operation.pending.active_call
(gauge)
Active pending operation.
Shown as operation
hive.server.api.operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.count
(rate)
Number of operation in pending state.
Shown as operation
hive.server.api.operation.pending.meanrate
(gauge)
Operation pending rate.
Shown as operation
hive.server.api.operation.running.active_call
(gauge)
Active running operation.
Shown as operation
hive.server.api.operation.running.max
(gauge)
Max time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.mean
(gauge)
Mean time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.min
(gauge)
Min time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.75percentile
(gauge)
P75 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.95percentile
(gauge)
P95 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.count
(rate)
Number of operation in running state.
Shown as millisecond
hive.server.api.operation.running.meanrate
(gauge)
Operation running rate.
Shown as operation
hive.server.operation.completed.finished
(rate)
Number of finished operation.
Shown as operation
hive.server.operation.completed.closed
(rate)
Number of closed operation.
Shown as operation
hive.server.memory.heap.init
(gauge)
Memory used at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.heap.used
(gauge)
Memory used by the HiveServer2.
Shown as byte
hive.server.memory.heap.max
(gauge)
Maximum memory that can be used by the HiveServer2.
Shown as byte
hive.server.memory.heap.committed
(gauge)
Committed memory for the HiveServer2.
Shown as byte
hive.server.memory.non_heap.init
(gauge)
Memory used at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.used
(gauge)
Memory used by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.max
(gauge)
Maximum memory that can be used by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.committed
(gauge)
Committed memory for the HiveServer2
Shown as byte
hive.server.memory.total.init
(gauge)
Total memory at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.total.used
(gauge)
Total memory used by the HiveServer2.
Shown as byte
hive.server.memory.total.max
(gauge)
Total maximum memory that can be used for the HiveServer2.
Shown as byte
hive.server.memory.total.committed
(gauge)
Total committed memory for the HiveServer2.
Shown as byte

Service Checks

hive.can_connect:
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored HiveServer2/Hive Metastore instance, otherwise returns OK.

Events

The Hive check does not include any events.

Troubleshooting

Need help? Contact Datadog support.


Mistake in the docs? Feel free to contribute!