Argentina
Australia
Austria
Belgium
Brazil
Bulgaria
Canada
Chile
Colombia
Cyprus
Czechia
Denmark
Ecuador
Egypt
Estonia
Finland
France
Germany
Great Britain
Greece
Hungary
India
Indonesia
Ireland
Italy
Japan
Latvia
Lithuania
Luxembourg
Malaysia
Malta
Mexico
Netherlands
New Zealand
Norway
Philippines
Poland
Portugal
Romania
Russia
Singapore
Slovakia
Slovenia
South Africa
South Korea
Spain
Sweden
Switzerland
Taiwan
Thailand
Turkey
Ukraine
United States
Aman Singh
In this chapter, we will cover the following recipes:
Configuring HDFS block size
Setting up Namenode metadata location
Loading data into HDFS
Configuring HDFS replication
HDFS balancer
Quota configuration
HDFS health and FSCK
Configuring rack awareness
Recycle or trash bin configuration
Distcp usage
Controlling block report storm
Configuring Datanode heartbeat
In this chapter, we will take a look at the storage layer, which is HDFS, and how it can be configured for storing data. It is important to ensure the good health of this distributed filesystem, and make sure that the data it contains is available, even in the case of failures. In this chapter, we will take a look at the replication, quota setup, and balanced distribution of data across nodes, as well as covering recipes on rack awareness and heartbeat for communication with the master.
The recipes in this chapter assume that you already have a running cluster and have completed the steps given in Chapter 1, Hadoop Architecture and Deployment.
While the recipes in this chapter will give you an overview of a typical configuration, we encourage you to adapt this proposal according to your needs. The block size plays an important role in the performance and the amount of data that is worked on by a mapper. It is good practice to set up passphrase less access between nodes, so...
To step through the recipes in this chapter, make sure you have completed the recipes in Chapter 1, Hadoop Architecture and Deployment or at least understand the basic Hadoop cluster setup.
ssh to the master node, which is Namenode, and navigate to the directory where Hadoop is installed. In the previous chapter, Hadoop was installed at /opt/cluster/hadoop:
$ ssh root@10.0.0.4
Change to the Hadoop user, or any other user that is running Hadoop, by using the following:
$ sudo su - hadoop
Edit the hdfs-site.xml file and modify the parameter to reflect the changes, as shown in the following screenshot:
dfs.blocksize is the parameter that decides on the value of the HDFS block size. The unit is bytes and the default value is 64 MB in Hadoop 1 and 128 MB in Hadoop 2. The block size can be configured according to the need.
Once the changes are made to hdfs-site.xml, copy the file across all nodes in the cluster.
Then restart the Namenode and...
The most critical component of Hadoop is Namenode, and it is important to safeguard the information it stores. It stores metadata, which is a combination of namespace and inode structure.
All the steps are to be performed as the hadoop user. It is expected that the user has gone through Chapter 1, Hadoop Architecture and Deployment and understands the uses and function of Namenode.
You are going to need a preinstalled Hadoop as discussed in Chapter 1, Hadoop Architecture and Deployment. In the following recipes, we will configure the parameters for a copy of Hadoop that is already installed.
ssh to the Namenode, which in this case is nn1.cluster1.com.
Navigate to the /opt/cluster/hadoop/etc/hadoop directory. This is the directory where we installed Hadoop in the first chapter. If the user has installed it at a different location, then navigate to this directory.
Configure the dfs.namenode.name.dir parameter, which defines the...
It is important to make sure that the cluster is working fine and the user can perform file operations on the cluster.
Log in to any of the nodes in the cluster. It's best to use the edge node, as stated in Chapter 1, Hadoop Architecture and Deployment, and switch to user hadoop.
Create a simple text file named file1.txt using any of your favorite text editors, and write some content in it.
Connect to the client1.cluster1.com edge node and switch to the hadoop user.
Copy the file1.txt file to HDFS, as shown in the following screenshot:
The user can check for the status of a file, as shown in the following screenshot:
The user can make sure that the file exists and its type is correct, as shown in the following screenshot. The user can execute the commands to see the sub options:
$ hadoop fs (and hit enter to see the options)
For redundancy, it is important to have multiple copies of data. In HDFS, this is achieved by placing copies of blocks on different nodes. By default, the replication factor is 3, which means that for each block written to HDFS, there will be three copies in total on the nodes in the cluster.
It is important to make sure that the cluster is working fine and the user can perform file operations on the cluster.
Log in to any of the nodes in the cluster. It is best to use the edge node, as stated in Chapter 1, and switch to the user hadoop.
Create a simple text file named file1.txt using any of your favorite text editors, and write some content in it.
ssh to the Namenode, which in this case is nn1.cluster1.com, and switch to user hadoop.
Navigate to the /opt/cluster/hadoop/etc/hadoop directory. This is the directory where we installed Hadoop in Chapter 1, Hadoop Architecture and Deployment. If the user has installed it at a different location...
In a long-running cluster, there might be unequal distribution of data across Datanodes. This could be due to failures of nodes or the addition of nodes to the cluster.
To make sure that the data is equally distributed across Datanodes, it is important to use Hadoop balancer to redistribute the blocks.
For this recipe, you will again use the same node on which we have already configured Namenode.
All operations will be done by user hadoop.
Log in the nn1.cluster1.com node and change to user hadoop.
Execute the balancer command as shown in the following screenshot:
By default, the balancer threshold is set to 10%, but we can change it, as shown in the following screenshot:
In a multitenancy cluster, it is important to control the utilization both in terms of HDFS space, memory, and CPU utilization. In this recipe, we will be looking at how we can restrict a user or a project from using more than the allotted HDFS space.
Make sure that there is a running cluster, and that the user is already well versed in the recipes that we have looked at so far.
Connect to Namenode and change the user to hadoop.
Create a directory named projects on HDFS, as shown in the following screenshot:
By default, there is no quota configured on any directory.
To see what options can be set on the projects directory, use the following command:
$ hadoop fs -count -q /projects
The two leftmost fields show the namespace and disk space quota, which currently is not set, as shown in the following screenshot:
To set the namespace quota, which will define how many inodes can be allocated for this projects directory, enter the following code. Inodes...
The health of the filesystem is very important for data retrieval and optimal performance. In a distributed system, it becomes more critical to maintain the good health of the HDFS filesystem so as to ensure block replication and near-parallel streaming of data blocks.
In this recipe, we will see how to check the health of the filesystem and do repairs, if any are needed.
Make sure you have a running cluster that has already been up for a few days with data. We can run the commands on a new cluster as well, but for the sake of this lab, it will give you more insights if it is run on a cluster with a large dataset.
ssh to the master1.cyrus.com Namenode and change the user to hadoop.
To check the HDFS root filesystem, execute the hdfs fsck / command, as shown in the following screenshot:
We can also check the status of just one file instead of the entire filesystem, as shown in the following screenshot:
There will always be failures in clusters, such as hardware issues with servers, racks, switches, power supplies, and so on.
To make sure that there is no single point of failure across the entire Hadoop infrastructure, and to ensure that the contention of resources is in a distributed manner, rack awareness plays an important role. Rack awareness is a concept in which Namenode is made aware of the layout of servers in a cluster, thus making intelligent decisions on block placement.
For the following steps, we assume that the cluster that is up and running with Datanodes is in a healthy state. We will log in to the Namenode and make changes there.
There will also be cases where we need to restore an accidently deleted file or directory. This may be due to a user error or some archiving policy that cleans data periodically.
For such situations, we can configure the recycle bin so that the deleted files can be restored for a specified amount of time. In this recipe, we will see that this can be configured.
This recipe shows the steps needed to edit the configuration file and add new parameters to the file to enable trash in the Hadoop cluster.
ssh to Namenode and edit the core-site.xml file to add the following property to it:
<property> <name>fs.trash.interval</name> <value>10080</value> </property>
The fs.trash.interval parameter defines the time in minutes after which the checkpoint will be deleted.
Restart the namenode daemon for the property to take effect:
$ hadoop-daemons.sh stop namenode $ hadoop-daemons.sh start namenode
Once trash...
In Hadoop, we deal with large data, so performing a simple copy operation might not be the optimal thing to do. Imagine copying a 1 TB file from one cluster to another, or within the same cluster to a different path, and after 50% of the copy operation it times out. In this situation, the copy has to be started from the beginning.
This recipe shows the steps needed to copy files within and across the cluster. Ensure that the user has a running cluster with YARN configured to run MapReduce, as discussed in Chapter 1, Hadoop Architecture and Deployment.
For this recipe, there is no configuration needed to run Distcp; just make sure HDFS and YARN is up and running.
ssh to Namenode or the edge node and execute the following command to copy the projects directory to the new directory:
$ hadoop distcp /projects /new
The preceding command will submit a MapReduce job to the cluster, and once the job finishes we can see the data copied at the destination.
We can...
When Datanodes come up in large clusters with more than 200 nodes, Namenode will be overwhelmed by the block reports and this can cause Namenode to become unresponsive.
This recipe makes more sense for large clusters, not in terms of the number of nodes, but the number of blocks in the cluster.
ssh to Namenode and edit the hdfs-site.xml file to add the following property to it:
<property> <name>dfs.blockreport.initialDelay</name> <value>20</value> </property>
Copy hdfs-site.xml across all nodes in the cluster.
Restart HDFS daemons across the nodes for the property to take effect:
$ stop-dfs.sh $ start-dfs.sh
The Datanodes periodically update the Namenode about its presence or any changes in the blocks. The default Datanode heartbeat time is three seconds. But this does not mean that if a Datanode does not send a heartbeat for, say, 10 seconds, that the node will be marked dead.
In this recipe, we will look at how a heartbeat is configured and the parameters that play a role in its function.
You have a running cluster, and the user is familiar with Datanode communication with Namenode.
ssh to Namenode and edit the hdfs-site.xml file to add the following property to it:
<property> <name>dfs.heartbeat.interval</name> <value>3</value> </property> <property> <name>dfs.namenode.heartbeat.recheck-interval</name> <value>300000</value> </property>
Copy hdfs-site.xml across all nodes in the cluster.
Restart HDFS daemons across nodes for the property to take effect:
$ stop-dfs.sh...
The rest of the chapter is locked
© 2017 Packt Publishing Limited All Rights Reserved
