--- title: HDFS Datanode description: Track cluster disk usage, volume failures, dead DataNodes, and more. breadcrumbs: Docs > Integrations > HDFS Datanode --- # HDFS Datanode Supported OS Integration version7.4.0 ## Overview{% #overview %} Track disk utilization and failed volumes on each of your HDFS DataNodes. This Agent check collects metrics for these, as well as block- and cache-related metrics. Use this check (hdfs_datanode) and its counterpart check (hdfs_namenode), not the older two-in-one check (hdfs); that check is deprecated. **Minimum Agent version:** 6.0.0 ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/agent/kubernetes/integrations/) for guidance on applying these instructions. ### Installation{% #installation %} The HDFS DataNode check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package, so you don't need to install anything else on your DataNodes. ### Configuration{% #configuration %} #### Connect the Agent{% #connect-the-agent %} {% tab title="Host" %} #### Host{% #host %} To configure this check for an Agent running on a host: 1. Edit the `hdfs_datanode.d/conf.yaml` file, in the `conf.d/` folder at the root of your [Agent's configuration directory](https://docs.datadoghq.com/agent/guide/agent-configuration-files/#agent-configuration-directory). See the [sample hdfs_datanode.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/hdfs_datanode/datadog_checks/hdfs_datanode/data/conf.yaml.example) for all available configuration options: ```yaml init_config: instances: ## @param hdfs_datanode_jmx_uri - string - required ## The HDFS DataNode check retrieves metrics from the HDFS DataNode's JMX ## interface via HTTP(S) (not a JMX remote connection). This check must be installed on a HDFS DataNode. The HDFS ## DataNode JMX URI is composed of the DataNode's hostname and port. ## ## The hostname and port can be found in the hdfs-site.xml conf file under ## the property dfs.datanode.http.address ## https://hadoop.apache.org/docs/r3.1.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml # - hdfs_datanode_jmx_uri: http://localhost:9864 ``` 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent). {% /tab %} {% tab title="Containerized" %} #### Containerized{% #containerized %} For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/agent/kubernetes/integrations/) for guidance on applying the parameters below. | Parameter | Value | | -------------------- | --------------------------------------------------- | | `` | `hdfs_datanode` | | `` | blank or `{}` | | `` | `{"hdfs_datanode_jmx_uri": "http://%%host%%:9864"}` | #### Log collection{% #log-collection %} **Available for Agent >6.0** 1. Collecting logs is disabled by default in the Datadog Agent. Enable it in the `datadog.yaml` file with: ```yaml logs_enabled: true ``` 1. Add this configuration block to your `hdfs_datanode.d/conf.yaml` file to start collecting your DataNode logs: ```yaml logs: - type: file path: /var/log/hadoop-hdfs/*.log source: hdfs_datanode service: ``` Change the `path` and `service` parameter values and configure them for your environment. 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent). {% /tab %} ### Validation{% #validation %} [Run the Agent's status subcommand](https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information) and look for `hdfs_datanode` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **hdfs.datanode.cache\_capacity**(gauge) | Cache capacity in bytes*Shown as byte* | | **hdfs.datanode.cache\_used**(gauge) | Cache used in bytes*Shown as byte* | | **hdfs.datanode.dfs\_capacity**(gauge) | Disk capacity in bytes*Shown as byte* | | **hdfs.datanode.dfs\_remaining**(gauge) | The remaining disk space left in bytes*Shown as byte* | | **hdfs.datanode.dfs\_used**(gauge) | Disk usage in bytes*Shown as byte* | | **hdfs.datanode.estimated\_capacity\_lost\_total**(gauge) | The estimated capacity lost in bytes*Shown as byte* | | **hdfs.datanode.last\_volume\_failure\_date**(gauge) | The date/time of the last volume failure in milliseconds since epoch*Shown as millisecond* | | **hdfs.datanode.num\_blocks\_cached**(gauge) | The number of blocks cached*Shown as block* | | **hdfs.datanode.num\_blocks\_failed\_to\_cache**(gauge) | The number of blocks that failed to cache*Shown as block* | | **hdfs.datanode.num\_blocks\_failed\_to\_uncache**(gauge) | The number of failed blocks to remove from cache*Shown as block* | | **hdfs.datanode.num\_failed\_volumes**(gauge) | Number of failed volumes | ### Events{% #events %} The HDFS-datanode check does not include any events. ### Service Checks{% #service-checks %} **hdfs.datanode.jmx.can\_connect** Returns `CRITICAL` if the Agent cannot connect to the DataNode's JMX interface for any reason. Returns `OK` otherwise. *Statuses: ok, critical* ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/). ## Further Reading{% #further-reading %} - [Hadoop & HDFS Architecture: An Overview](https://www.datadoghq.com/blog/hadoop-architecture-overview) - [How to monitor Hadoop metrics](https://www.datadoghq.com/blog/monitor-hadoop-metrics) - [How to collect Hadoop metrics](https://www.datadoghq.com/blog/collecting-hadoop-metrics) - [How to monitor Hadoop with Datadog](https://www.datadoghq.com/blog/monitor-hadoop-metrics-datadog)