Decommissioning nodes

There will always be failures in clusters, such as hardware issues or a need to upgrade nodes. This should be done in a graceful manner, without any data loss.

When the Datanode daemon is stopped on a Datanode, it takes approximately ten minutes for the Namenode to remove that node. This has to do with the heartbeat retry interval. At any time, we can abruptly remove the Datanode, but it can result in data loss.

It is recommended that you opt for the graceful removal of the node from the cluster, as this ensures that all the data on that node is drained.

Getting ready

For the following steps, we assume that the cluster that is up and running with Datanodes is in a healthy state and the one with the Datanode dn1.cluster1.com needs maintenance and must be removed from the cluster. We will login to the Namenode and make changes there.

How to do it...

1. ssh to Namenode and edit the file hdfs-site.xml by adding the following property to it:

   <property>
   <name>dfs.hosts.exclude</name>
   <value>/home/hadoop/excludes</value>
   <final>true</final>
   </property>

2. Make sure the file excludes is readable by the user hadoop.

3. Restart the Namenode daemon for the property to take effect:

   $ hadoop-daemons.sh stop namenode
   $ hadoop-daemons.sh start namenode
   

4. A restart of Namenode is required only when any property is changed in the file. Once the property is in place, Namenode can read the changes to the contents of the file excludes by simply refreshing nodes.

5. Add the dn1.cluster1.com node to the file excludes:

   $ cat excludes
   dn1.cluster1.com
   

6. After adding the node to the file, we just need to reload the file by doing the following:

   $ hadoop dfsadmin -refreshNodes
   

7. After sometime, the node will be decommissioned. The time will vary according to the data the particular Datanode had. We can see the decommissioned nodes using the following:

   $ hdfs dfsadmin -report
   

8. The preceding command will list the nodes in the cluster, and against the dn1.cluster1.com node we can see that its status will either be decommissioning or decommissioned.

How it works...

Let's have a look at what we did throughout this recipe.

In steps 1 through 6, we added the new property to the hdfs-site.xml file and then restarted Namenode to make it aware of the changes. Once the property is in place, the Namenode is aware of the excludes file, and it can be asked to re-read by simply refreshing the node list, as done in step 6.

With these steps, the data on the Datanode dn1.cluster1.com will be moved to other nodes in the cluster, and once the data has been drained, the Datanode daemon on the node will be shutdown. During the process, the node will change the status from normal to decommissioning and then to decommissioned.

Care must be taken while decommissioning nodes in the cluster. The user should not decommission multiple nodes at a time as this will generate lot of network traffic and cause congestion and data loss.

See also

• The Add nodes to the cluster recipe