Hive
セキュリティモニタリングが使用可能です セキュリティモニタリングが使用可能です

Hive

Agent Check Agentチェック

Supported OS: Linux Mac OS Windows

概要

このチェックは、Hive の Hive Metastore と HiveServer2 の 2 つを監視します。

セットアップ

インストール

Hive チェックは Datadog Agent パッケージに含まれています。サーバーに追加でインストールする必要はありません。

コンフィギュレーション

Hive のセットアップ

  1. Hive Metastore と HiveServer2 のメトリクスを有効化するには、以下のプロパティを追加して、HIVE_HOME/conf/hive-site.xml にある Hive コンフィギュレーションファイルを編集します。

    <property>
     <name>hive.metastore.metrics.enabled</name>
     <value>true</value>
    </property>
    <property>
     <name>hive.server2.metrics.enabled</name>
     <value>true</value>
    </property>
  2. HiveServer2、Hive Metastore、またはその両方の JMX リモート接続を有効にします。たとえば、以下のように HADOOP_CLIENT_OPTS 環境変数を設定します。

    export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.port=8808"

次に、HiveServer2 または Hive Metastore を再起動します。Hive Metastore と HiveServer2 で同じ JMX 接続を共有することはできません。

ホスト

ホストで実行中の Agent でこのチェックを構成する場合は、以下の手順に従います。コンテナ環境の場合は、コンテナ化セクションを参照してください。

メトリクスの収集
  1. Hive のパフォーマンスデータを収集するには、Agent のコンフィギュレーションディレクトリのルートにある conf.d/ フォルダーの hive.d/conf.yaml ファイルを編集します。使用可能なすべてのコンフィギュレーションオプションについては、サンプル hive.d/conf.yaml を参照してください。

    このチェックでは、インスタンスあたりのメトリクス数が 350 に制限されています。返されたメトリクスの数は、情報ページに表示されます。以下で説明するコンフィギュレーションを編集することで、関心があるメトリクスを指定できます。 収集するメトリクスをカスタマイズする方法については、JMX チェックのドキュメントで詳細な手順を参照してください。制限数以上のメトリクスを監視する必要がある場合は、Datadog のサポートチームまでお問い合わせください。

  2. Agent を再起動します

ログの収集

Agent バージョン 6.0 以降で利用可能

  1. Datadog Agent で、ログの収集はデフォルトで無効になっています。以下のように、datadog.yaml でこれを有効にする必要があります。

    logs_enabled: true
  2. Hive のログの収集を開始するには、次の構成ブロックを hive.d/conf.yaml ファイルに追加します。

     logs:
       - type: file
         path: /tmp/<USER>/hive.log
         source: hive
         service: '<SERVICE_NAME>'
         log_processing_rules:
           - type: multi_line
             name: new_log_start_with_date
             pattern: \d{4}\-\d{2}\-\d{2}

    path パラメーターおよび service パラメーターの値を変更し、環境に合わせて構成します。使用可能なすべてのコンフィギュレーションオプションについては、サンプル hive.d/conf.yaml を参照してください。

  3. Agent を再起動します

コンテナ化

コンテナ環境の場合は、オートディスカバリーのインテグレーションテンプレートのガイドを参照して、次のパラメーターを適用してください。

メトリクスの収集

Datadog-Hive インテグレーションを使用してメトリクスを収集するには、JMX を使用したオートディスカバリーガイドを参照してください。

ログの収集

Agent バージョン 6.0 以降で利用可能

Datadog Agent で、ログの収集はデフォルトで無効になっています。有効にする方法については、Kubernetes ログ収集のドキュメントを参照してください。

パラメーター
<LOG_CONFIG>{"source": "hive", "service": "<サービス名>", "log_processing_rules":{"type":"multi_line","name":"new_log_start_with_date", "pattern":"\d{4}\-\d{2}\-\d{2}"}}

検証

Agent の status サブコマンドを実行し、Checks セクションで Hive を探します。

収集データ

メトリクス

hive.metastore.open_connections
(gauge)
Number of connection opened.
Shown as connection
hive.metastore.db.created
(rate)
Total number of created database.
Shown as item
hive.metastore.db.deleted
(rate)
Total number of deleted database.
Shown as item
hive.metastore.db.init
(rate)
Number of initialized database.
Shown as item
hive.metastore.table.created
(rate)
Total number of created table.
Shown as table
hive.metastore.table.deleted
(rate)
Total number of deleted table.
Shown as table
hive.metastore.table.init
(rate)
Number of initialized table.
Shown as table
hive.metastore.partition.created
(rate)
Total number of created partition.
Shown as item
hive.metastore.partition.deleted
(rate)
Total number of deleted partition.
Shown as item
hive.metastore.partition.init
(rate)
Number of initialized partition.
Shown as item
hive.metastore.directsql_errors
(gauge)
Number of SQL error.
Shown as unit
hive.metastore.memory.heap.init
(gauge)
Memory used at the initialization by the metastore.
Shown as byte
hive.metastore.memory.heap.used
(gauge)
Memory used by the metastore.
Shown as byte
hive.metastore.memory.heap.max
(gauge)
Maximum memory that can be used by the metastore.
Shown as byte
hive.metastore.memory.heap.committed
(gauge)
Committed memory for the metastore.
Shown as byte
hive.metastore.memory.non_heap.init
(gauge)
Memory used at the initialization by the metastore.
Shown as byte
hive.metastore.memory.non_heap.used
(gauge)
Memory used by the metastore.
Shown as byte
hive.metastore.memory.non_heap.max
(gauge)
Maximum memory that can be used by the metastore.
Shown as byte
hive.metastore.memory.non_heap.committed
(gauge)
Committed memory for the metastore
Shown as byte
hive.metastore.memory.total.init
(gauge)
Total memory at the initialization by the metastore.
Shown as byte
hive.metastore.memory.total.used
(gauge)
Total memory used by the metastore.
Shown as byte
hive.metastore.memory.total.max
(gauge)
Total maximum memory that can be used for the metastore.
Shown as byte
hive.metastore.memory.total.committed
(gauge)
Total committed memory for the metastore.
Shown as byte
hive.metastore.api.create_table
(rate)
API call to create a table.
Shown as task
hive.metastore.api.create_table.active_call
(gauge)
Active API call to create a table.
Shown as unit
hive.metastore.api.get_table
(rate)
API call to get a table.
Shown as task
hive.metastore.api.get_table.active_call
(gauge)
Active API call to get a table.
Shown as unit
hive.metastore.api.drop_table
(rate)
API call to drop a table.
Shown as task
hive.metastore.api.drop_table.active_call
(gauge)
Active API call to drop a table.
Shown as unit
hive.metastore.api.init
(rate)
API initialization.
Shown as task
hive.metastore.api.init.active_call
(gauge)
Active API initialization.
Shown as unit
hive.metastore.api.get_all_databases
(rate)
API call to get all databases.
Shown as task
hive.metastore.api.get_all_databases.active_call
(gauge)
Active API call to get all databases.
Shown as unit
hive.metastore.api.get_database
(rate)
API call to get a database.
Shown as task
hive.metastore.api.get_database.active_call
(gauge)
Active API call to get a database.
Shown as unit
hive.metastore.api.get_all_tables
(rate)
API call to get all tables.
Shown as task
hive.metastore.api.get_all_tables.active_call
(gauge)
Active API call to get all tables.
Shown as unit
hive.metastore.api.shutdown
(rate)
API shutdown.
Shown as task
hive.metastore.api.shutdown.active_call
(gauge)
Active API shutdown.
Shown as unit
hive.metastore.api.flushcache
(rate)
API flushcache.
Shown as task
hive.metastore.api.flushcache.active_call
(gauge)
Active API flushcache.
Shown as unit
hive.metastore.api.alter_table
(rate)
API call to alter table.
Shown as task
hive.metastore.api.alter_table.active_call
(gauge)
Active API call to alter table.
Shown as unit
hive.metastore.api.get_all_functions
(rate)
API call to get all functions.
Shown as task
hive.metastore.api.get_all_functions.active_call
(gauge)
Active API call to get all functions.
Shown as unit
hive.metastore.api.get_table_req
(rate)

Shown as task
hive.metastore.api.get_table_req.active_call
(gauge)

Shown as unit
hive.server.open_operations
(gauge)
Operation opened in the HiveServer2.
Shown as operation
hive.server.session.active
(gauge)
Number of active session.
Shown as session
hive.server.session.active.time_mean
(gauge)
Average time a session has been active.
Shown as millisecond
hive.server.session.open
(gauge)
Number of opened session.
Shown as session
hive.server.session.open.time_mean
(gauge)
Average time a session has been opened.
Shown as millisecond
hive.server.queries.submitted.active_call
(gauge)
Active submitted queries.
Shown as query
hive.server.queries.submitted.max
(gauge)
Max time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.mean
(gauge)
Mean time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.min
(gauge)
Min time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.75percentile
(gauge)
P75 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.95percentile
(gauge)
P95 time for a submitted query.
Shown as millisecond
hive.server.queries.submitted.count
(rate)
Number of submitted query.
Shown as query
hive.server.queries.submitted.meanrate
(gauge)
Submitted query rate.
Shown as query
hive.server.queries.compiling.active_call
(gauge)
Active compiling queries.
Shown as query
hive.server.queries.compiling.max
(gauge)
Max time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.mean
(gauge)
Mean time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.min
(gauge)
Min time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.75percentile
(gauge)
P75 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.95percentile
(gauge)
P95 time for compiling a query.
Shown as millisecond
hive.server.queries.compiling.count
(rate)
Number of compiled query.
Shown as query
hive.server.queries.compiling.meanrate
(gauge)
Compiling query rate.
Shown as query
hive.server.api.queries.executing.active_call
(gauge)
Active executing queries.
Shown as query
hive.server.queries.executing.max
(gauge)
Max time for executing a query.
Shown as millisecond
hive.server.queries.executing.mean
(gauge)
Mean time for executing a query.
Shown as millisecond
hive.server.queries.executing.min
(gauge)
Min time for executing a query.
Shown as millisecond
hive.server.queries.executing.75percentile
(gauge)
P75 time for executing a query.
Shown as millisecond
hive.server.queries.executing.95percentile
(gauge)
P95 time for executing a query.
Shown as millisecond
hive.server.queries.executing.count
(rate)
Number of executed queries.
Shown as query
hive.server.queries.executing.meanrate
(gauge)
Executing query rate.
Shown as query
hive.server.queries.succeeded.count
(rate)
Number of succeeded queries.
Shown as query
hive.server.queries.succeeded.meanrate
(gauge)
Succeeded queries rate.
Shown as query
hive.server.api.sql_operation.pending.active_call
(gauge)
Active pending sql operation.
Shown as operation
hive.server.api.sql_operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.sql_operation.pending.count
(rate)
Number of sql operation in pending state.
Shown as operation
hive.server.api.sql_operation.pending.meanrate
(gauge)
Pending sql operation rate.
Shown as operation
hive.server.api.sql_operation.running.active_call
(gauge)
Active running sql operation.
Shown as operation
hive.server.api.sql_operation.running.max
(gauge)
Max time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.mean
(gauge)
Mean time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.min
(gauge)
Min time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.75percentile
(gauge)
P75 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.95percentile
(gauge)
P95 time for running state for a sql operation.
Shown as millisecond
hive.server.api.sql_operation.running.count
(rate)
Number of sql operation in running state.
Shown as operation
hive.server.api.sql_operation.running.meanrate
(gauge)
Running sql operation rate.
Shown as operation
hive.server.sql_operation.completed.closed
(rate)
Number of closed sql operation.
Shown as operation
hive.server.sql_operation.completed.finished
(rate)
Number of finished sql operation.
Shown as operation
hive.server.sql_operation.user.active
(gauge)
Number of active user.
Shown as user
hive.server.api.operation.initialized.active_call
(gauge)
Active initialized operation.
Shown as user
hive.server.api.operation.initialized.max
(gauge)
Max time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.mean
(gauge)
Mean time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.min
(gauge)
Min time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.75percentile
(gauge)
P75 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.95percentile
(gauge)
P95 time to init an operation.
Shown as millisecond
hive.server.api.operation.initialized.count
(rate)
Number of operation initialized.
Shown as operation
hive.server.api.operation.initialized.meanrate
(gauge)
Operation initialization rate.
Shown as operation
hive.server.api.operation.pending.active_call
(gauge)
Active pending operation.
Shown as operation
hive.server.api.operation.pending.max
(gauge)
Max time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.mean
(gauge)
Mean time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.min
(gauge)
Min time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.75percentile
(gauge)
P75 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.95percentile
(gauge)
P95 time in pending state for an sql operation.
Shown as millisecond
hive.server.api.operation.pending.count
(rate)
Number of operation in pending state.
Shown as operation
hive.server.api.operation.pending.meanrate
(gauge)
Operation pending rate.
Shown as operation
hive.server.api.operation.running.active_call
(gauge)
Active running operation.
Shown as operation
hive.server.api.operation.running.max
(gauge)
Max time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.mean
(gauge)
Mean time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.min
(gauge)
Min time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.75percentile
(gauge)
P75 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.95percentile
(gauge)
P95 time in running state for an sql operation.
Shown as millisecond
hive.server.api.operation.running.count
(rate)
Number of operation in running state.
Shown as millisecond
hive.server.api.operation.running.meanrate
(gauge)
Operation running rate.
Shown as operation
hive.server.operation.completed.finished
(rate)
Number of finished operation.
Shown as operation
hive.server.operation.completed.closed
(rate)
Number of closed operation.
Shown as operation
hive.server.memory.heap.init
(gauge)
Memory used at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.heap.used
(gauge)
Memory used by the HiveServer2.
Shown as byte
hive.server.memory.heap.max
(gauge)
Maximum memory that can be used by the HiveServer2.
Shown as byte
hive.server.memory.heap.committed
(gauge)
Committed memory for the HiveServer2.
Shown as byte
hive.server.memory.non_heap.init
(gauge)
Memory used at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.used
(gauge)
Memory used by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.max
(gauge)
Maximum memory that can be used by the HiveServer2.
Shown as byte
hive.server.memory.non_heap.committed
(gauge)
Committed memory for the HiveServer2
Shown as byte
hive.server.memory.total.init
(gauge)
Total memory at the initialization by the HiveServer2.
Shown as byte
hive.server.memory.total.used
(gauge)
Total memory used by the HiveServer2.
Shown as byte
hive.server.memory.total.max
(gauge)
Total maximum memory that can be used for the HiveServer2.
Shown as byte
hive.server.memory.total.committed
(gauge)
Total committed memory for the HiveServer2.
Shown as byte

サービスのチェック

hive.can_connect:
Agent が監視対象の HiveServer2/Hive Metastore インスタンスに接続できず、メトリクスを収集できない場合は CRITICAL を返します。それ以外の場合は OK を返します。

イベント

Hive チェックには、イベントは含まれません。

トラブルシューティング

ご不明な点は、Datadog のサポートチームまでお問合せください。