Building Rich YARN Applications

"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live."

– Martin Golding

YARN or (Yet Another Resource Negotiator) was introduced in Hadoop version 2 to open distributed programming for all of the problems that may not necessarily be addressed using the MapReduce programming technique. Let's look at the key reasons behind introducing YARN in Hadoop:

• The older Hadoop used Job Tracker to coordinate running jobs whereas Task Tracker was used to run assigned jobs. This eventually became a bottleneck due to a single Job Tracker when working with a high number of Hadoop nodes.
• With traditional MapReduce, the nodes were assigned fixed numbers of Map and Reduce slots. Due to this nature, the utilization of the cluster resources was not optimal due to inflexibility between...

Technical requirements

You will need Eclipse development environment and Java 8 installed on your system where you can run/tweak these examples. If you prefer to use maven, then you will need maven installed to compile the code. To run the example, you also need Apache Hadoop 3.1 setup on Linux system. Finally, to use the Git repository of this book, you need to install Git.

The code files of this chapter can be found on GitHub:
https://github.com/PacktPublishing/Apache-Hadoop-3-Quick-Start-Guide/tree/master/Chapter5

Check out the following video to see the code in action:

http://bit.ly/2CRSq5P

Understanding YARN architecture

YARN separates the role of Job Tracker into two separate entities. A Resource Manager is a central authority and is responsible for allocation and management of cluster resources, and an application master to manage the life cycle of applications that are running on the cluster. The following diagram depicts YARN architecture and the flow of requests-response:

YARN provides the basic units of applications such as memory, CPU, and GPU. The units of an application are utilized by containers. All containers are managed by respective Node Managers running on the Hadoop cluster. The Application master (AM) negotiates with the Resource Manager (RM) for container availability along with the resource manager. The AM container is initialized by client through resource manager as shown in step 2. Once AM is initialized, it demands container availability...

Key features of YARN

YARN offers significant gain over traditional MapReduce programming that comes from older versions of Apache Hadoop. With YARN, you can write any custom applications that can utilize the power of commodity hardware and Apache Hadoop's HDFS filesystem to scale and perform. Let's go through some of the key features of YARN that brings major additions. We have already covered the new features of YARN 3.0, such as the intra-disk balancer in Chapter 1, Hadoop 3.0 - Background and Introduction.

Resource models in YARN

YARN supports an extensible resource model. This means that the definition of resources can be extended from its default values (such as CPU and memory) to any types of resources that...

Configuring the YARN environment in a cluster

We have seen the configuration of MapReduce and HDFS. To enable YARN, first you need to inform Hadoop that you are using YARN as your framework, so you need to add the following entries in mapred-site.xml:

<configuration>
                <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
                </property>
</configuration>

Please refer to Chapter 2, Planning and Setting Up Hadoop Clusters, for additional properties and steps for configuring YARN. Now, let's look at key configuration elements in yarn-site.xml that you would be looking at day by day:

Property Name	Default Value	Description
yarn.resourcemanager .hostname	0.0.0.0	Specify the hostname of resource manager.
yarn.resourcemanager .address		IP...

Working with YARN distributed CLI

YARN CLI provides three types of commands. The first type is for users who wish to use YARN infrastructure for developing applications. The second type are administrative commands, which provide monitoring and administrative capabilities of all components of YARN including resource manager, application master, and timeline server. The third type are daemon commands, which are used for maintenance purposes covering stopping, starting, and restarting of daemons. Now, let's look at user commands for YARN:

Command	Parameters	Description	Important Parameters
application	yarn application <command> <parameters>	All actions pertaining to applications such as print and kill.	- appID <applicationID> - kill <applicationID> - list - status <applicationID>
applicationattempt	yarn applicationattempt...

Deep dive with YARN application framework

In this section, we will do a deep dive into YARN application development. YARN offers flexibility to developers to write applications that can run on Hadoop clusters in different programming languages. In this section, we will focus on setting up a YARN project, we will write a sample client and application master, and we will see how it runs on a YARN cluster. The following block diagram shows typical interaction patterns between various components of Apache Hadoop when a YARN application is developed and deployed:

Primarily, there are three major components involved: Resource Manager, Application Master, and Node Manager. We will be creating a custom client application, a custom application master, and a YARN client app. As you can see, there are three different interactions that take place between different components:

• Client and...

Building and monitoring a YARN application on a cluster

YARN is a completely rewritten architecture of a Hadoop cluster. Once you are done with your development of the YARN application framework, the next step is to create your own custom application that you wish to run on YARN across a Hadoop cluster. Let's write a small application. In my example code, I have provided two applications:

• MyApplication.java: This prints Hello World
• MyApplication2.java: This calculates the value of PI to the 1,000^th level

These simple applications would be run on the YARN environment through the client we have created. Let's look at how you can build a YARN application.

Building a YARN application

There are different approaches...

Summary

In this chapter, we have done a deep dive into YARN. We understood YARN architecture and key features of YARN such as resource models, federation, and RESTful APIs. We then configured a YARN environment in a Hadoop distributed cluster. We also studied some of the additional properties of yarn-site.xml. We then looked at the YARN distributed command-line interface. After this, we dived deep into building a YARN application, where we first created a framework needed for the application to run, then we created a sample application. We also covered building YARN applications and monitoring them.

In the next chapter, we will look at monitoring and administration of a Hadoop cluster.