HBase Administration Cookbook: Master HBase configuration and administration for optimum database performance with this book and ebook

You are going to need a Linux machine to run the stack. Running HBase on top of Windows is not recommended. We will use Debian 6.0.1 (Debian Squeeze) in this book, because we have several Hadoop/HBase clusters running on top of Debian in production at my company, Rakuten Inc., and 6.0.1 is the latest Amazon Machine Image (AMI) we have, at http://wiki.debian.org/Cloud/AmazonEC2Image.

As HBase is written in Java, you will need to have Java installed first. HBase runs on Oracle's JDK only, so do not use OpenJDK for the setup. Although Java 7 is available, we don't recommend you to use Java 7 now because it needs more time to be tested. You can download the latest Java SE 6 from the following link: http://www.oracle.com/technetwork/java/javase/downloads/index.html.

Execute the downloaded bin file to install Java SE 6. We will use /usr/local/jdk1.6 as JAVA_HOME in this book:

root# ln -s /your/java/install/directory /usr/local/jdk1.6

We will add a user with the name hadoop, as the owner of all HBase/Hadoop daemons and files. We will have all HBase files and data stored under /usr/local/hbase:

root# useradd hadoop
root# mkdir /usr/local/hbase
root# chown hadoop:hadoop /usr/local/hbase

Get the latest stable HBase release from HBase's official site, http://www.apache.org/dyn/closer.cgi/hbase/. At the time of writing this book, the current stable release was 0.92.1.

You can set up a standalone HBase instance by following these instructions:

hadoop$ /usr/local/hbase/current/bin/stop-hbase.sh
stopping hbase.......

We installed HBase 0.92.1 on a single server. We have used a symbolic link named current for it, so that version upgrading in the future is easy to do.

In order to inform HBase where Java is installed, we will set JAVA_HOME in hbase-env.sh, which is the environment setting file of HBase. You will see some Java heap and HBase daemon settings in it too. We will discuss these settings in the last two chapters of this book.

In step 1, we created a directory on the local filesystem, for HBase to store its data. For a fully distributed installation, HBase needs to be configured to use HDFS, instead of a local filesystem. The HBase master daemon (HMaster) is started on the server where start-hbase.sh is executed. As we did not configure the region server here, HBase will start a single slave daemon (HRegionServer) on the same JVM too.

As we mentioned in the Introduction section, HBase depends on ZooKeeper as its coordination service. You may have noticed that we didn't start ZooKeeper in the previous steps. This is because HBase will start and manage its own ZooKeeper ensemble, by default.

Then we connected to HBase via HBase Shell. Using HBase Shell, you can manage your cluster, access data in HBase, and do many other jobs. Here, we just created a table called test, we inserted data into HBase, scanned the test table, and then disabled and dropped it, and exited the shell.

HBase can be stopped using its stop-hbase.sh script. This script stops both HMaster and HRegionServer daemons.

You will need to sign up, or create an Amazon Web Service (AWS)  account at http://aws.amazon.com/.

We will use EC2 command-line tools to manage our instances. You can download and set up the tools by following the instructions available at the following page:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?SettingUp_CommandLine.html.

You need a public/private key to log in to your EC2 instances. You can generate your key pairs and upload your public key to EC2, using these instructions:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/generating-a-keypair.html.

Before you can log in to an instance, you must authorize access. The following link contains instructions for adding rules to the default security group:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/adding-security-group-rules.html.

After all these steps are done, review the following checklist to make sure everything is ready:

$ cat ~/.bashrc
export EC2_HOME=~/opt/ec2-api-tools-1.4.4.2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=~/.ec2/pk-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export EC2_CERT=~/.ec2/cert-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export JAVA_HOME=/Library/Java/Home
export EC2_URL=https://ec2.us-west-1.amazonaws.com

We need to import our EC2 key pairs to manage EC2 instances via EC2 command-line tools:

$ ec2-import-keypair your-key-pair-name --public-key-file ~/.ssh/id_rsa.pub

Verify the settings by typing the following command:

$ ec2-describe-instances

If everything has been set up properly, the command will show your instances similarly to how you had configured them in the previous command.

The last preparation is to find a suitable AMI. An AMI is a preconfigured operating system and software, which is used to create a virtual machine within EC2. We can find a registered Debian AMI at http://wiki.debian.org/Cloud/AmazonEC2Image.

For the purpose of practicing HBase, a 32-bit, EBS-backed AMI is the most cost effective AMI to use. Make sure you are choosing AMIs for your region. As we are using US-West (us-west-1) for this book, the AMI ID for us is ami-77287b32. This is a 32-bit, small instance of EC2. A small instance is good for practicing HBase on EC2 because it's cheap. For production, we recommend you to use at least High-Memory Extra Large Instance with EBS, or a real server.

Follow these instructions to get your EC2 instances ready for HBase. We will start two EC2 instances; one is a DNS/NTP server, and the other one is the client:

You will need four small EC2 instances, which can be obtained by using the following command:

$ec2-run-instances ami-77287b32 -t m1.small -n 4 -k your-key-pair

All these instances must be set up properly, as described in the previous recipe, Getting ready on Amazon EC2. Besides the NTP and DNS setups, Java installation is required by all servers too.

We will use the hadoop user as the owner of all Hadoop daemons and files. All Hadoop files and data will be stored under /usr/local/hadoop. Add the hadoop user and create a /usr/local/hadoop directory on all the servers, in advance.

We will set up one Hadoop client node as well. We will use client1, which we set up in the previous recipe. Therefore, the Java installation, hadoop user, and directory should be prepared on client1 too.

Here are the steps to set up a fully distributed Hadoop cluster:

To start/stop the daemon on remote slaves from the master node, a passwordless SSH login of the hadoop user is required. We did this in step 1.

HBase must run on a special HDFS that supports a durable sync implementation. If HBase runs on an HDFS that has no durable sync implementation, it may lose data if its slave servers go down. Hadoop versions later than 0.20.205, including Hadoop 1.0.2 which we have chosen, support this feature.

HDFS and MapReduce use local filesystems to store their data. We created directories required by Hadoop in step 3, and set up the path to the Hadoop's configuration file in step 5.

In steps 9 to 11, we set up Hadoop so it could find HDFS, JobTracker, and slave servers. Before starting Hadoop, all Hadoop directories and settings need to be synced with the slave servers. The first time you start Hadoop (HDFS), you need to format NameNode. Note that you should only do this at the initial installation.

At this point, you can start/stop Hadoop using its start/stop script. Here we started/stopped HDFS and MapReduce separately, in case you don't require MapReduce. You can also use $HADOOP_HOME/bin/start-all.sh and stop-all.sh to start/stop HDFS and MapReduce using one command.

First, make sure Java is installed in your ZooKeeper server.

We will use the hadoop user as the owner of all ZooKeeper daemons and files. All the ZooKeeper files and data will be stored under /usr/local/ZooKeeper; you need to create this directory in advance. Our ZooKeeper will be set up on master1 too.

We will set up one ZooKeeper client on client1. So, the Java installation, hadoop user, and directory should be prepared on client1 as well.

To set up a standalone ZooKeeper installation, follow these instructions:

In this recipe, we set up a basic standalone ZooKeeper instance. As you can see, the setting is very simple; all you need to do is to tell ZooKeeper where to find Java and where to save its data.

In step 4, we created a file named java.env and placed the Java settings in this file. You must use this filename as ZooKeeper, which by default, gets its Java settings from this file.

ZooKeeper's settings file is called zoo.cfg. You can copy the settings from the sample file shipped with ZooKeeper. The default setting is fine for basic installation. As ZooKeeper always acts as a central role in a cluster system, it should be set up properly to gain the best performance.

To connect to a running ZooKeeper ensemble, use its command-line tool, and specify the ZooKeeper server and port you want to connect to. The default client port is 2181. You don't need to specify it, if you are using the default port setting.

All ZooKeeper data is called a Znode. Znodes are constructed like a filesystem hierarchy. ZooKeeper provides commands to access or update Znode from its command-line tool; type help for more information.

As HBase relays ZooKeeper as its coordination service, the ZooKeeper service must be extremely reliable. In production, you must run a ZooKeeper cluster of at least three nodes. Also, make sure to run an odd number of nodes.

The procedure to set up a clustered ZooKeeper is basically the same as shown in this recipe. You can follow the previous steps to set up each cluster node at first. Add the following settings to each node's zoo.cfg, so that every node knows about every other node in the ensemble:

hadoop@node{1,2,3}$ vi $ZK_HOME/conf/zoo.cfg

server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

Also, you need to put a myid file under ${dataDir}. The myid file consists of a single line containing only the node ID. So myid of node1 would contain the text 1 and nothing else.

Start ZooKeeper from each node of your cluster respectively. Then, you can connect to the cluster from your client, by using the following command:

$ zkCli.sh -server node1,node2,node3

ZooKeeper will function as long as more than half of the nodes in the ZooKeeper cluster are alive. This means, in a three node cluster, only one server can die.

Make sure you have root privileges on all of your servers.

You will need to make the following kernel setting changes to all servers of the cluster:

The previous setting changes the hadoop user's open file limit to 65535. It also changes the hadoop user's max processes number to 32000. With this change of the kernel setting, HBase can keep enough files open at the same time and also run smoothly.

First, make sure Java is installed on all servers of the cluster.

We will use the hadoop user as the owner of all HBase daemons and files, too. All HBase files and data will be stored under /usr/local/hbase. Create this directory on all servers of your HBase cluster, in advance.

We will set up one HBase client on client1. Therefore, the Java installation, hadoop user, and directory should be prepared on client1 too.

Make sure HDFS is running. You can ensure it started properly by accessing HDFS, using the following command:

hadoop@client1$ $HADOOP_HOME/bin/hadoop fs -ls /

MapReduce does not need to be started, as HBase does not normally use it.

We assume that you are managing your own ZooKeeper, in which case, you can start it and confirm if it is running properly. You can ensure it is running properly by sending the ruok command to its client port:

hadoop@client1$ echo ruok | nc master1 2181

To set up our fully distributed HBase cluster, we will download and configure HBase on the master node first, and then sync to all slave nodes and clients.

Get the latest stable HBase release from HBase's official site, http://www.apache.org/dyn/closer.cgi/hbase/.

At the time of writing this book, the current stable release was 0.92.1.

Our HBase cluster is configured to use /hbase as its root directory on HDFS, by specifying the hbase.rootdir property. Because it is the first time HBase was started, it will create the directory automatically. You can see the files HBase created on HDFS from the client:

hadoop@client1$ $HADOOP_HOME/bin/hadoop fs -ls /hbase

We want our HBase to run on distributed mode, so we set hbase.cluster.distributed to true in hbase-site.xml.

We also set up the cluster to use an independent ZooKeeper ensemble by specifying HBASE_MANAGES_ZK=false in hbase-env.sh. The ZooKeeper ensemble is specified by the hbase.ZooKeeper.quorum property. You can use clustered ZooKeeper by listing all the servers of the ensemble, such as zoo1,zoo2,zoo3.

All region servers are configured in the $HBASE_HOME/conf/regionservers file. You should use one line per region server. When starting the cluster, HBase will SSH into each region server configured here, and start the HRegionServer daemon on that server.

By linking hdfs-site.xml under the $HBASE_HOME/conf directory, HBase will use all the client configurations you made for your HDFS in hdfs-site.xml, such as the dfs.replication setting.

HBase ships with its prebuilt hadoop-core and ZooKeeper JAR files. They may be out of date, compared to what you used in your Hadoop and ZooKeeper installation. Make sure HBase uses the same version of .jar files with Hadoop and ZooKeeper, to avoid any unexpected problems.

The configurations that should be considered for change are as follows:

$ grep -i "HDFS-200" hbase-hadoop-master-master1.log
...syncFs -- HDFS-200 -- not available, dfs.support.append=false

You should already have HDFS and HBase installed. We will set up a standby master node (master2), as you need another server ready to use. Make sure all the dependencies have been configured properly. Sync your Hadoop and HBase root directory from the active master (master1) to the standby master.

We will need NFS in this recipe as well. Set up your NFS server, and mount the same NFS directory from both master1 and master2. Make sure the hadoop user has write permission to the NFS directory. Create a directory on NFS to store Hadoop's metadata. We assume the directory is /mnt/nfs/hadoop/dfs/name.

We will set up VIP for the two masters, and assume you have the following IP addresses and DNS mapping:

The following instructions describe how to set up two highly available master nodes.

The previous steps finally leave us with a cluster structure like the following diagram:

At first, we installed Heartbeat and Pacemaker on the two masters and then configured Heartbeat to enable Pacemaker.

In step 2 of the Create and install a NameNode resource agent section, we created the namenode script, which is implemented as a standard OCF resource agent. The most important function of the namenode script is namenode_status, which monitors the status of the NameNode daemon. Here we use the jps command to show all running Java processes owned by the hadoop user, and the grep name of the NameNode daemon to see if it has started. The namenode resource agent is used by Pacemaker to start/stop/monitor the NameNode daemon. In the namenode script, as you can see in the namenode_start and namenode_stop methods, we actually start/stop NameNode by using hadoop-daemon.sh, which is used to start/stop the Hadoop daemon on a single server. You can find a full list of the code from the source shipped with this book.

We started Heartbeat after our namenode resource agent was tested and installed. Then, we made some changes to the default crm configurations. The default-resource-stickiness=1 setting is very important as it turns off the automatic failback of a resource.

We added a VIP resource to Pacemaker and configured Hadoop and HBase to use it in steps 3 to 5 of the Configure highly available NameNode section. By using VIP in their configuration, Hadoop and HBase can switch to communicate with the standby master if the active one is down.

In step 6 of the same section, we configured Hadoop (HDFS NameNode) to write its metadata to both the local disk and NFS. If an active master is down, NameNode will be started from the standby master. Because they were mounted on the same NFS directory, NameNode started from the standby master can apply the latest metadata from NFS, and restore HDFS to the status before the original active master is down.

In steps 7 to 10, we added the NAMENODE resource using the namenode resource agent we created in step 2 of the Create and install a NameNode resource agent section, then we set up VIP and NAMENODE resources as a group (step 8), and made sure they always run on the same server (step 9), with the right start-up order (step 10). We did this because we didn't want VIP running on master1, while NameNode was running on master2.

Because Pacemaker will start NameNode for us via the namenode resource agent, we need to start DataNode separately, which is what we did in step 1 of the Start DataNode, HBase cluster, and backup HBase master section.

After starting HBase normally, we started our standby HBase master (HMaster) on the standby master server. If you check your HBase master log, you will find output like the following, which shows itself as a standby HMaster:

2011-11-21 23:38:55,168 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the active master, ip-10-174-14-15.us-west-1.compute.internal:60000; waiting to become the next active master

Finally, we got NameNode and HMaster running on two servers with an active-standby configuration. The single point of failure of the cluster was avoided.

However, it leaves us with lots of works to do in production. You need to test your HA cluster in all rare cases, such as a server power off, unplug of a network cable, shut down of network switch, or anything else you can think of.

On the other hand, SPOF of the cluster may not be as critical as you think. Based on our experience, almost all of the downtime of the cluster is due to an operational miss or software upgrade. It's better to make your cluster simple.

It is more complex to set up a highly available HBase cluster on Amazon EC2 because EC2 does not support static IP addresses, and so we can't use VIP on EC2. An alternative way is to use an Elastic IP address. An Elastic IP address is the role of a static IP address on EC2 while it is associated with your account, not a particular instance. We can use Heartbeat to associate EIP to the standby master automatically, if the active one is down. Then, we configure Hadoop and HBase to use an instance's public DNS associated with EIP, to find an active master. Also, in the namenode resource agent, we have to start/stop not only NameNode, but also all DataNodes. This is because the IP address of an active master has changed, but DataNode cannot find the new active master unless it is restarted.

We will skip the details because it's out of the scope of this book. We created an elastic-ip resource agent to achieve this purpose. You can find it in the source shipped with this book.