In this chapter, we will cover the following recipes:
Configuring users and groups
Fair Scheduler configuration
Fair Scheduler pools
Configuring job queues
Job queue ACLs
Configuring Capacity Scheduler
Queuing mappings in Capacity Scheduler
YARN and Mapred commands
YARN label-based scheduling
YARN SLS
In this chapter, we will configure YARN schedulers and job queues so that multiple users can use the cluster at the same time and make a legitimate use of the resources provided to them. There are two approaches: either setup a separate cluster for different business units or share the clusters.
The first approach is fine if there are a few clusters, but managing a large number of clusters is challenging. A better approach is to build multitenancy clusters, which can support different users with varied use cases.
If users are finding it difficult to write scripts or configurations, all these are available at my GitHub at https://github.com/netxillon/hadoop.
In our previous recipes, we installed or configured Hadoop clusters as user hadoop. But, in production, it is good to run jobs as different users and also the Hadoop daemons can be separated to run with different user IDs, so as to have better control and security.
The security aspects will be covered in the security chapter, but it is important to understand the user segregation and grouping users per project or business units.
In this recipe, we will see how to create users and groups for job submission. These recipes do not talk about the HDFS user permissions or file ACLs, but only about the permission to submit jobs and what percentage of cluster capacity each user or department can use within an organization.
To go through the recipe in this section, we need Hadoop Cluster setup and running. By default, Apache Hadoop 1.x distribution uses FIFO scheduler and Hadoop 2.x uses Capacity Scheduler. In a cluster with multiple jobs, it is not good to use FIFO scheduler, as it will starve the jobs for resources and only the very first job in the queue is executed; all other jobs have to wait.
To address the preceding issue, there are two commonly used Schedulers: Fair Scheduler, and Capacity Scheduler, to allocate the cluster resources in a fair manner. In this recipe, we will see how to configure Fair Scheduler. Simply put, Fair Scheduler shares resources fairly among running jobs based on queues and weights assigned.
In this recipe, we look at configuring Fair Scheduler with pools instead of queues. This is for backwards compatibility. In Hadoop 1.X, Fair Scheduler was addressed with pools and it means the same as queues.
It is recommended to use queues, as this is quite standard across the board. But, for the sake of the readers, it is good to cover the concepts of pools.
To go through the recipe, complete the previous recipe and just modify the fair-scheduler.xml file to reflect pools.
Connect to the master1.cyrus.com master node in the cluster and switch as user hadoop.
Edit the allocation file fair-scheduler.xml, as shown in the following screenshot:
Copy the fair-scheduler.xml file to all the nodes in the cluster and restart the YARN daemons.
Check the ResourceManager page to confirm whether the pools are visible or not, as shown in the following screenshot:
Submit a sample job such as wordcount as user hadoop and see the ResourceManager page, as shown in...
In this recipe, we will configure the job queue and allow users to submit jobs to the queues. In production, there might be many departments such as marketing, sales, and finance, sharing a cluster of resources and it is important to have the correct shares proportional to business and funding.
In the previous recipe, although the queues were setup, they were still not used. Queues were dynamically created for jobs submitted by specifying a queue. If no queue is specified, jobs are submitted to a queue by the name of the user who submitted the job. We will explore these a bit more in this recipe.
To complete the recipe, the user must have a running cluster with HDFS and YARN configured and must have completed the previous two recipes.
In YARN, whenever a job is submitted, it is submitted either to a specified queue or default queue. This behavior has been explored in the last recipe. In this recipe, we will configure ACLs to block users from submitting jobs to other queues.
In order to get started, you will need a running cluster with HDFS and YARN set up properly and an understanding of the last recipe.
Connect to the master1.cyrus.com master node and switch as user hadoop.
Execute the command as shown next in the picture to see the queue ACLs. By default, users can see that d1 has administrative and submit rights to all the queues:
Edit the fair-scheduler.xml allocation file as shown next. Note the users specified for the specific queues (within the...
Capacity Scheduler is mainly designed for multitenancy, where multiple organizations collectively fund the cluster based on the computing needs. There is an added benefit that an organization can access any excess capacity not being used by others. This provides elasticity for the organizations in a cost-effective manner.
For this recipe, you will again need a running cluster with YARN and HDFS configured in the cluster. Readers are recommended to read the previous recipes in this chapter to understand this recipe better.
In Hadoop 2.x, the default scheduler is Capacity Scheduler and it is enabled by default, unless modified explicitly as seen in the previous recipes where we have configured Fair Scheduler.
In this recipe, we will be configuring users who can submit jobs to the queue and can also set rule for various job submissions.
Let's look at another use case where, if user hadoop submits a job, it should go to the prod queue and if any other users submits a job, it must go to dev queue. How can we set up something like this?
Make sure that the user has a running cluster with HDFS and YARN configured. It's best to have gone through at least the previous recipe.
Connect to the master1.cyrus.com Namenode and switch as user hadoop.
Edit the capacity-scheduler.xml allocation file as shown next:
<property>
<name>yarn.scheduler.capacity.queue-mappings</name>
<value>u:d1:dev,g:group1:default,u:hadoop:prod</value>
</property>Make the preceding changes and copy the file across all nodes and restart the YARN daemons.
Whenever the d1 user submits a job, it should go to the dev queue and for user...
In this recipe, we will cover a few of the important commands which can help with the administration of YARN.
Until now, after making any change to the configuration, we have restarted the YARN daemons. But, this is not always required if the parameters already exist in the configuration files. Updating the queue list and capacity can be done dynamically at run time. In this recipe, we will cover a few of the easy ways to make changes.
For this recipe, you will again need a running cluster with at least HDFS and YARN configured and have the queue setup as discussed in this chapter. It is recommended that users go through all the previous recipes in this chapter before following this particular recipe.
In this recipe, we will configure YARN label-based scheduling. In a cluster, there can be a mixture of nodes with different configurations, some with more memory and CPU compared to other nodes in the cluster.
If we want to control which set of nodes a job executes, we need to assign labels to the nodes. A typical case could be that you want to run a Spark streaming job and want that to execute on nodes with high memory. For such a situation, we will configure the queue and assign a set of nodes for that, so that if a job is submitted to that queue, it executes on the nodes which have higher configuration in terms of memory and cores.
Make sure that the user has a running cluster with at least two Datanodes and YARN working perfectly. Users are expected to have a basic knowledge about queues in Hadoop, for which they can refer to the previous few recipes in this chapter.
In this recipe, we will take a look at YARN simulator, which is useful to test and determine the load of YARN under various test conditions.
The YARN Scheduler Load Simulator (SLS) is such a tool, which can simulate large-scale YARN clusters and application loads in a single machine within a single JVM.
For this recipe, you will need a single machine with Hadoop installed. For this, readers can refer to the first chapter, where we have covered a single node cluster setup.
Connect to the master1.cyrus.com single node and switch as user hadoop.
The SLS is located at $HADOOP_HOME/share/hadoop/tools/sls/.
The SLS runs the simulator using the sls-runner.xml configuration file under $HADOOP_HOME/etc/hadoop.
A sample file is located at $HADOOP_HOME/share/hadoop/tools/sls/sample-conf/sls-runner.xml.
The sls-runner.xml file contains parameters to tune a number of threads, container memory, NM threads, and many other parameters or the choice of scheduler you want.
The...
© 2017 Packt Publishing Limited All Rights Reserved
