Configuring MapReduce for performance

In this recipe, we will touch upon MapReduce parameters and see how we can optimize them.

Getting ready

For this recipe, you will again need a running cluster with HDFS and YARN. Users must have completed the recipe Configuring YARN for performance recipe.

How to do it...

1. Connect to the master node master1.cyrus.com and switch to the hadoop user.

2. The file where these changes will be made is mapred-site.xml.

3. The first thing to adjust is to sort the buffer according to the HDFS block size. It must always be greater than the value of dfs.blocksize. This can be configured as follows:

   <property>
   <name>mapreduce.task.io.sort.mb</name>
   <value>200</value>
   </property>

4. The next value to tune is the number of streams to merge while sorting. This many file handles will be open per mapper:

   <property>
   <name>mapreduce.task.io.sort.factor</name>
   <value>24</value>
   </property>

5. Another important thing to take...