In this chapter, we will cover the following recipes:
Running a simple MapReduce program
Hadoop streaming
Configuring YARN history server
Job history web interface and metrics
Configuring ResourceManager components
YARN containers resource allocations
ResourceManager Web UI and JMX metrics
Preserving ResourceManager states
In the previous chapters, we learned about the storage layer HDFS, how to configure it, and what are its different components. We mainly talked about Namenode, Datanode, and its concepts.
In this chapter, we will take a look at the processing layer which is MapReduce and the resource management framework YARN. Prior to Hadoop 2.x, MapReduce was the only processing layer for Hadoop, but the introduction of YARN as a framework, provided a pluggable processing layer, which could be MapReduce, Spark, and so on.
In this recipe, we will look at how to make sense of the data stored on HDFS and extract useful information out of the files like the number of occurrences of a string, a pattern, or estimations, and various benchmarks. For this purpose, we can use MapReduce, which is a computation framework that helps us answer many questions we might have about the data.
With Hadoop, we can process huge amount of data. However, to get an understanding of its working, we'll start with a simple program such as pi estimation or a word count example.
ResourceManager is the master for Yet another Resource Negotiator (YARN). The Namenode stores the file metadata and the actual blocks/data reside on the slave nodes called Datanodes. All the jobs are submitted to the ResourceManager and it then assigns tasks to its slaves, called NodeManagers.
When a job is submitted to ResourceManager (RM), it will check for the job queue it is submitted to and whether the user has permissions...
In this recipe, we will look at how we can execute jobs on an Hadoop cluster using scripts written in Bash or Python. It is not mandatory to use only Java for programming MapReduce code; any language can be used by evoking the Hadoop streaming utility. Do not confuse this with real-time streaming, which is different from what we will be discussing here.
To step through the recipes in this chapter, make sure you have a running cluster with HDFS and YARN setup correctly as discussed in the previous chapters. This can be a single node cluster or a multinode cluster, as long as the cluster is configured correctly.
It is not necessary to know Java to run MapReduce programs on Hadoop. Users can carry forward their existing scripting knowledge and use Bash or Python to run the job on Hadoop.
Whenever a MapReduce job runs, it launches containers on multiple nodes and the logs for that container are only written on that particular node. If the user needs details of the job, he needs to go to all the nodes to fetch the logs, which could be very tedious in large clusters.
A better approach will be to aggregate the logs at a common location once the job finishes and then it can be accessed using a web server or other means. To address this, History Server was introduced in Hadoop, to aggregate logs and provide a Web UI, for users to see logs for all the containers of a job at one place.
You need to have a running cluster with YARN set up and should have completed the previous recipe to make sure the cluster is working fine in terms of HDFS and YARN.
The following steps will guide you through the process of setting up Job history server.
In the previous recipe, we enabled history server, and now we will use the Web UI to the explore YARN metrics and job history.
Make sure you have completed the previous recipe and have a History Server running as daemon, as shown here in the list of processes:
Using web browser, connect to the JobHistoryServer Web UI port, which in this case is port 19888 and host IP master1.cyrus.com.
Once connected to the Web UI, the user can see JobHistory and other details as shown here:
Under the Tools section on the left-most side, the user can see links to view YARN parameters currently in effect, using the link configuration as shown here:
Another section is metrics, which gives information about JvmMetrics, stats, and so on. The output format is JSON:
The preceding output can also be viewed from the command line, as shown in the following screenshot:
In YARN, ResourceManager is modular in nature, primarily limited to scheduling and not bothered about application state management, which is delegated to Application Masters. Although there are many components of RM, the core ones are: ApplicationsManager (AsM), Application Master Launcher, scheduler, and ResourceManager. AsM keeps track of which AM got assigned for which job and requests the launch of AM using AM Launcher. These are all part of the ResourceManager and are depicted in the following diagram. These components can be segregated for better control and management of resources:
In this recipe, we will see how different components can be separated out, although not necessary, and controlled independently.
In YARN, there are many configuration parameters, which control the memory available to AM, containers, or the total memory that can be allocated for MapReduce or JVM heap size to be used and the number of CPU cores to be used for a job. This is covered in more detail in Chapter 8, Performance Tuning, but a rough idea is to have one core for each container and each Mapper container should have a memory of about 1 GB and the Reducer should have a memory twice the size of the Mapper. In addition to this, each node must have about 20% spare for the operating system and Hadoop daemons.
For this recipe, you will again need a running cluster and should have completed the previous recipes to make sure the cluster is working fine in terms of HDFS and YARN.
In the previous recipe, we presented how to configure parameters for YARN and MapReduce. As stated initially, each daemon runs a Jetty web server, which can be accessed using a web browser.
Users must take note of the fact that their RPC ports are different from HTTP ports and must not be confused with the options we used in the previous recipe. There are default web ports such as Namenode 50070, ResourceManager 8088, Datanode 50075. All these can be configured to custom ports, if needed.
Make sure that the user has a running cluster with YARN and HDFS configured. The user must be able to run MapReduce jobs on it.
Point your web browser to http://master1.cyrus.com/8088, to access the ResourceManager Web UI:
The Web UI gives information on running the application and the resources it uses, as shown in the following screenshot:
The web interface also shows the scheduler used, which is by default capacity scheduler:
The ResourceManager...
It is important to preserve the state of ResourceManager during the restart of RM, so as to keep the application running with minimal interruptions. The concept is that the RM preserves the application state in a store and reloads it on restart. ApplicationMasters (AM) and NodeManagers continuously poll RM for status and re-register with it when available, thus resuming the containers from saved state.
For this recipe, you will again need a running cluster and have completed the previous recipes to make sure the cluster is working fine in terms of HDFS and YARN.
Connect to the master1.cyrus.com master node and switch to user hadoop.
Navigate to the directory /opt/cluster/hadoop/etc/hadoop.
Edit the yarn-site.xml configuration file to make the necessary changes as shown in the following steps.
Enable RM recovery by making changes as shown in the following screenshot:
Specify the state-store to be used for this, as shown in the following...
© 2017 Packt Publishing Limited All Rights Reserved
