--- title: Map Reduce description: Monitor the status and duration of map and reduce tasks. breadcrumbs: Docs > Integrations > Map Reduce --- # Map Reduce Supported OS Integration version7.4.0 ## Overview{% #overview %} Get metrics from mapreduce service in real time to: - Visualize and monitor mapreduce states - Be notified about mapreduce failovers and events. **Minimum Agent version:** 6.0.0 ## Setup{% #setup %} ### Installation{% #installation %} The Mapreduce check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package, so you don't need to install anything else on your servers. ### Configuration{% #configuration %} {% tab title="Host" %} #### Host{% #host %} To configure this check for an Agent running on a host: 1. Edit the `mapreduce.d/conf.yaml` file, in the `conf.d/` folder at the root of your [Agent's configuration directory](https://docs.datadoghq.com/agent/guide/agent-configuration-files.md#agent-configuration-directory) to point to your server and port, set the masters to monitor. See the [sample mapreduce.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/mapreduce/datadog_checks/mapreduce/data/conf.yaml.example) for all available configuration options. 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md#restart-the-agent). ##### Log collection{% #log-collection %} 1. Collecting logs is disabled by default in the Datadog Agent, enable it in your `datadog.yaml` file: ```yaml logs_enabled: true ``` 1. Uncomment and edit the logs configuration block in your `mapreduce.d/conf.yaml` file. Change the `type`, `path`, and `service` parameter values based on your environment. See the [sample mapreduce.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/mapreduce/datadog_checks/mapreduce/data/conf.yaml.example) for all available configuration options. ```yaml logs: - type: file path: source: mapreduce service: # To handle multi line that starts with yyyy-mm-dd use the following pattern # log_processing_rules: # - type: multi_line # pattern: \d{4}\-\d{2}\-\d{2} \d{2}:\d{2}:\d{2},\d{3} # name: new_log_start_with_date ``` 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md#restart-the-agent). {% /tab %} {% tab title="Containerized" %} #### Containerized{% #containerized %} For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/agent/kubernetes/integrations.md) for guidance on applying the parameters below. | Parameter | Value | | -------------------- | --------------------------------------------------------------------------------------------- | | `` | `mapreduce` | | `` | blank or `{}` | | `` | `{"resourcemanager_uri": "https://%%host%%:8088", "cluster_name":""}` | ##### Log collection{% #log-collection %} Collecting logs is disabled by default in the Datadog Agent. To enable it, see the [Docker Log Collection](https://docs.datadoghq.com/agent/docker/log.md). Then, set [log integrations](https://docs.datadoghq.com/agent/docker/log.md?tab=containerinstallation#log-integrations) as Docker labels: ```yaml LABEL "com.datadoghq.ad.logs"='[{"source": "mapreduce", "service": ""}]' ``` {% /tab %} ### Validation{% #validation %} Run the [Agent's status subcommand](https://docs.datadoghq.com/agent/guide/agent-commands.md#agent-status-and-information) and look for `mapreduce` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **mapreduce.job.counter.map\_counter\_value**(rate) | Counter value of map tasks*Shown as task* | | **mapreduce.job.counter.reduce\_counter\_value**(rate) | Counter value of reduce tasks*Shown as task* | | **mapreduce.job.counter.total\_counter\_value**(rate) | Counter value of all tasks*Shown as task* | | **mapreduce.job.elapsed\_time.95percentile**(gauge) | 95th percentile elapsed time since the application started*Shown as millisecond* | | **mapreduce.job.elapsed\_time.avg**(gauge) | Average elapsed time since the application started*Shown as millisecond* | | **mapreduce.job.elapsed\_time.count**(rate) | Number of times the elapsed time was sampled | | **mapreduce.job.elapsed\_time.max**(gauge) | Max elapsed time since the application started*Shown as millisecond* | | **mapreduce.job.elapsed\_time.median**(gauge) | Median elapsed time since the application started*Shown as millisecond* | | **mapreduce.job.failed\_map\_attempts**(rate) | Number of failed map attempts*Shown as task* | | **mapreduce.job.failed\_reduce\_attempts**(rate) | Number of failed reduce attempts*Shown as task* | | **mapreduce.job.killed\_map\_attempts**(rate) | Number of killed map attempts*Shown as task* | | **mapreduce.job.killed\_reduce\_attempts**(rate) | Number of killed reduce attempts*Shown as task* | | **mapreduce.job.map.task.elapsed\_time.95percentile**(gauge) | 95th percentile of all map tasks elapsed time*Shown as millisecond* | | **mapreduce.job.map.task.elapsed\_time.avg**(gauge) | Average of all map tasks elapsed time*Shown as millisecond* | | **mapreduce.job.map.task.elapsed\_time.count**(rate) | Number of times the map tasks elapsed time were sampled | | **mapreduce.job.map.task.elapsed\_time.max**(gauge) | Max of all map tasks elapsed time*Shown as millisecond* | | **mapreduce.job.map.task.elapsed\_time.median**(gauge) | Median of all map tasks elapsed time*Shown as millisecond* | | **mapreduce.job.maps\_completed**(rate) | Number of completed maps*Shown as task* | | **mapreduce.job.maps\_pending**(rate) | Number of pending maps*Shown as task* | | **mapreduce.job.maps\_running**(rate) | Number of running maps*Shown as task* | | **mapreduce.job.maps\_total**(rate) | Total number of maps*Shown as task* | | **mapreduce.job.new\_map\_attempts**(rate) | Number of new map attempts*Shown as task* | | **mapreduce.job.new\_reduce\_attempts**(rate) | Number of new reduce attempts*Shown as task* | | **mapreduce.job.reduce.task.elapsed\_time.95percentile**(gauge) | 95th percentile of all reduce tasks elapsed time*Shown as millisecond* | | **mapreduce.job.reduce.task.elapsed\_time.avg**(gauge) | Average of all reduce tasks elapsed time*Shown as millisecond* | | **mapreduce.job.reduce.task.elapsed\_time.count**(rate) | Number of times the reduce tasks elapsed time were sampled | | **mapreduce.job.reduce.task.elapsed\_time.max**(gauge) | Max of all reduce tasks elapsed time*Shown as millisecond* | | **mapreduce.job.reduce.task.elapsed\_time.median**(gauge) | Median of all reduce tasks elapsed time*Shown as millisecond* | | **mapreduce.job.reduces\_completed**(rate) | Number of completed reduces*Shown as task* | | **mapreduce.job.reduces\_pending**(rate) | Number of pending reduces*Shown as task* | | **mapreduce.job.reduces\_running**(rate) | Number of running reduces*Shown as task* | | **mapreduce.job.reduces\_total**(rate) | Number of reduces*Shown as task* | | **mapreduce.job.running\_map\_attempts**(rate) | Number of running map attempts*Shown as task* | | **mapreduce.job.running\_reduce\_attempts**(rate) | Number of running reduce attempts*Shown as task* | | **mapreduce.job.successful\_map\_attempts**(rate) | Number of successful map attempts*Shown as task* | | **mapreduce.job.successful\_reduce\_attempts**(rate) | Number of successful reduce attempts*Shown as task* | ### Events{% #events %} The Mapreduce check does not include any events. ### Service Checks{% #service-checks %} **mapreduce.resource\_manager.can\_connect** Returns `CRITICAL` if the Agent is unable to connect to the Resource Manager. Returns `OK` otherwise. *Statuses: ok, critical* **mapreduce.application\_master.can\_connect** Returns `CRITICAL` if the Agent is unable to connect to the Application Master. Returns `OK` otherwise. *Statuses: ok, critical* ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/). ## Further Reading{% #further-reading %} - [Hadoop & HDFS Architecture: An Overview](https://www.datadoghq.com/blog/hadoop-architecture-overview) - [How to monitor Hadoop metrics](https://www.datadoghq.com/blog/monitor-hadoop-metrics) - [How to collect Hadoop metrics](https://www.datadoghq.com/blog/collecting-hadoop-metrics) - [How to monitor Hadoop with Datadog](https://www.datadoghq.com/blog/monitor-hadoop-metrics-datadog)