HBase Administration Cookbook

By Yifeng Jiang
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Setting Up HBase Cluster

About this book

As an Open Source distributed big data store, HBase scales to billions of rows, with millions of columns and sits on top of the clusters of commodity machines. If you are looking for a way to store and access a huge amount of data in real-time, then look no further than HBase.

HBase Administration Cookbook provides practical examples and simple step-by-step instructions for you to administrate HBase with ease. The recipes cover a wide range of processes for managing a fully distributed, highly available HBase cluster on the cloud. Working with such a huge amount of data means that an organized and manageable process is key and this book will help you to achieve that.

The recipes in this practical cookbook start from setting up a fully distributed HBase cluster and moving data into it. You will learn how to use all of the tools for day-to-day administration tasks as well as for efficiently managing and monitoring the cluster to achieve the best performance possible. Understanding the relationship between Hadoop and HBase will allow you to get the best out of HBase so the book will show you how to set up Hadoop clusters, configure Hadoop to cooperate with HBase, and tune its performance.

Publication date:
August 2012
Publisher
Packt
Pages
332
ISBN
9781849517140

 

Chapter 1. Setting Up HBase Cluster

In this chapter, we will cover:

  • Quick start

  • Getting ready on Amazon EC2

  • Setting up Hadoop

  • Setting up ZooKeeper

  • Changing the kernel settings

  • Setting up HBase

  • Basic Hadoop/ZooKeeper/HBase configurations

  • Setting up multiple High Availability (HA) masters

 

Introduction


This chapter explains how to set up HBase cluster, from a basic standalone HBase instance to a fully distributed, highly available HBase cluster on Amazon EC2.

According to Apache HBase's home page:

HBase is the Hadoop database. Use HBase when you need random, real-time, read/write access to your Big Data. This project's goal is the hosting of very large tables—billions of rows X millions of columns—atop clusters of commodity hardware.

HBase can run against any filesystem. For example, you can run HBase on top of an EXT4 local filesystem, Amazon Simple Storage Service (Amazon S3), and Hadoop Distributed File System (HDFS) , which is the primary distributed filesystem for Hadoop. In most cases, a fully distributed HBase cluster runs on an instance of HDFS, so we will explain how to set up Hadoop before proceeding.

Apache ZooKeeper is an open source software providing a highly reliable, distributed coordination service. A distributed HBase depends on a running ZooKeeper cluster.

HBase, which is a database that runs on Hadoop, keeps a lot of files open at the same time. We need to change some Linux kernel settings to run HBase smoothly.

A fully distributed HBase cluster has one or more master nodes (HMaster), which coordinate the entire cluster, and many slave nodes (RegionServer), which handle the actual data storage and request. The following diagram shows a typical HBase cluster structure:

HBase can run multiple master nodes at the same time, and use ZooKeeper to monitor and failover the masters. But as HBase uses HDFS as its low-layer filesystem, if HDFS is down, HBase is down too. The master node of HDFS, which is called NameNode, is the Single Point Of Failure (SPOF) of HDFS, so it is the SPOF of an HBase cluster. However, NameNode as a software is very robust and stable. Moreover, the HDFS team is working hard on a real HA NameNode, which is expected to be included in Hadoop's next major release.

The first seven recipes in this chapter explain how we can get HBase and all its dependencies working together, as a fully distributed HBase cluster. The last recipe explains an advanced topic on how to avoid the SPOF issue of the cluster.

We will start by setting up a standalone HBase instance, and then demonstrate setting up a distributed HBase cluster on Amazon EC2.

 

Quick start


HBase has two run modes—standalone mode and distributed mode. Standalone mode is the default mode of HBase. In standalone mode, HBase uses a local filesystem instead of HDFS, and runs all HBase daemons and an HBase-managed ZooKeeper instance, all in the same JVM.

This recipe describes the setup of a standalone HBase. It leads you through installing HBase, starting it in standalone mode, creating a table via HBase Shell, inserting rows, and then cleaning up and shutting down the standalone HBase instance.

Getting ready

You are going to need a Linux machine to run the stack. Running HBase on top of Windows is not recommended. We will use Debian 6.0.1 (Debian Squeeze) in this book, because we have several Hadoop/HBase clusters running on top of Debian in production at my company, Rakuten Inc., and 6.0.1 is the latest Amazon Machine Image (AMI) we have, at http://wiki.debian.org/Cloud/AmazonEC2Image.

As HBase is written in Java, you will need to have Java installed first. HBase runs on Oracle's JDK only, so do not use OpenJDK for the setup. Although Java 7 is available, we don't recommend you to use Java 7 now because it needs more time to be tested. You can download the latest Java SE 6 from the following link: http://www.oracle.com/technetwork/java/javase/downloads/index.html.

Execute the downloaded bin file to install Java SE 6. We will use /usr/local/jdk1.6 as JAVA_HOME in this book:

root# ln -s /your/java/install/directory /usr/local/jdk1.6

We will add a user with the name hadoop, as the owner of all HBase/Hadoop daemons and files. We will have all HBase files and data stored under /usr/local/hbase:

root# useradd hadoop
root# mkdir /usr/local/hbase
root# chown hadoop:hadoop /usr/local/hbase

How to do it...

Get the latest stable HBase release from HBase's official site, http://www.apache.org/dyn/closer.cgi/hbase/. At the time of writing this book, the current stable release was 0.92.1.

You can set up a standalone HBase instance by following these instructions:

  1. 1. Download the tarball and decompress it to our root directory for HBase. We will set an HBASE_HOME environment variable to make the setup easier, by using the following commands:

    root# su - hadoop
    hadoop$ cd /usr/local/hbase
    hadoop$ tar xfvz hbase-0.92.1.tar.gz
    hadoop$ ln -s hbase-0.92.1 current
    hadoop$ export HBASE_HOME=/usr/local/hbase/current
    
  2. 2. Set JAVA_HOME in HBase's environment setting file, by using the following command:

    hadoop$ vi $HBASE_HOME/conf/hbase-env.sh
    # The java implementation to use. Java 1.6 required.
    export JAVA_HOME=/usr/local/jdk1.6
    
  3. 3. Create a directory for HBase to store its data and set the path in the HBase configuration file (hbase-site.xml), between the<configuration> tag, by using the following commands:

    hadoop$ mkdir -p /usr/local/hbase/var/hbase
    hadoop$ vi /usr/local/hbase/current/conf/hbase-site.xml
    <property>
    <name>hbase.rootdir</name>
    <value>file:///usr/local/hbase/var/hbase</value>
    </property>
    
  4. 4. Start HBase in standalone mode by using the following command:

    hadoop$ $HBASE_HOME/bin/start-hbase.sh
    starting master, logging to /usr/local/hbase/current/logs/hbase-hadoop-master-master1.out
    
  5. 5. Connect to the running HBase via HBase Shell, using the following command:

    hadoop$ $HBASE_HOME/bin/hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.92.1, r1298924, Fri Mar 9 16:58:34 UTC 2012
    
  6. 6. Verify HBase's installation by creating a table and then inserting some values. Create a table named test, with a single column family named cf1, as shown here:

    hbase(main):001:0> create 'test', 'cf1'
    0 row(s) in 0.7600 seconds
    

    i. In order to list the newly created table, use the following command:

    hbase(main):002:0> list
    TABLE
    test
    1 row(s) in 0.0440 seconds
    

    ii. In order to insert some values into the newly created table, use the following commands:

    hbase(main):003:0> put 'test', 'row1', 'cf1:a', 'value1'
    0 row(s) in 0.0840 seconds
    hbase(main):004:0> put 'test', 'row1', 'cf1:b', 'value2'
    0 row(s) in 0.0320 seconds
    
  7. 7. Verify the data we inserted into HBase by using the scan command:

    hbase(main):003:0> scan 'test'
    ROW COLUMN+CELL row1 column=cf1:a, timestamp=1320947312117, value=value1 row1 column=cf1:b, timestamp=1320947363375, value=value2
    1 row(s) in 0.2530 seconds
    
  8. 8. Now clean up all that was done, by using the disable and drop commands:

    i. In order to disable the table test, use the following command:

    hbase(main):006:0> disable 'test'
    0 row(s) in 7.0770 seconds
    

    ii. In order to drop the the table test, use the following command:

    hbase(main):007:0> drop 'test'
    0 row(s) in 11.1290 seconds
    
  9. 9. Exit from HBase Shell using the following command:

    hbase(main):010:0> exit
    
  10. 10. Stop the HBase instance by executing the stop script:

hadoop$ /usr/local/hbase/current/bin/stop-hbase.sh
stopping hbase.......

How it works...

We installed HBase 0.92.1 on a single server. We have used a symbolic link named current for it, so that version upgrading in the future is easy to do.

In order to inform HBase where Java is installed, we will set JAVA_HOME in hbase-env.sh, which is the environment setting file of HBase. You will see some Java heap and HBase daemon settings in it too. We will discuss these settings in the last two chapters of this book.

In step 1, we created a directory on the local filesystem, for HBase to store its data. For a fully distributed installation, HBase needs to be configured to use HDFS, instead of a local filesystem. The HBase master daemon (HMaster) is started on the server where start-hbase.sh is executed. As we did not configure the region server here, HBase will start a single slave daemon (HRegionServer) on the same JVM too.

As we mentioned in the Introduction section, HBase depends on ZooKeeper as its coordination service. You may have noticed that we didn't start ZooKeeper in the previous steps. This is because HBase will start and manage its own ZooKeeper ensemble, by default.

Then we connected to HBase via HBase Shell. Using HBase Shell, you can manage your cluster, access data in HBase, and do many other jobs. Here, we just created a table called test, we inserted data into HBase, scanned the test table, and then disabled and dropped it, and exited the shell.

HBase can be stopped using its stop-hbase.sh script. This script stops both HMaster and HRegionServer daemons.

 

Getting ready on Amazon EC2


Amazon Elastic Compute Cloud (EC2) is a web service that provides resizable computer capacity in the cloud. By using Amazon EC2, we can practice HBase on a fully distributed mode easily, at low cost. All the servers that we will use to demonstrate HBase in this book are running on Amazon EC2.

This recipe describes the setup of the Amazon EC2 environment, as a preparation for the installation of HBase on it. We will set up a name server and client on Amazon EC2. You can also use other hosting services such as Rackspace, or real servers to set up your HBase cluster.

Getting ready

You will need to sign up, or create an Amazon Web Service (AWS)  account at http://aws.amazon.com/.

We will use EC2 command-line tools to manage our instances. You can download and set up the tools by following the instructions available at the following page:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?SettingUp_CommandLine.html.

You need a public/private key to log in to your EC2 instances. You can generate your key pairs and upload your public key to EC2, using these instructions:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/generating-a-keypair.html.

Before you can log in to an instance, you must authorize access. The following link contains instructions for adding rules to the default security group:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/adding-security-group-rules.html.

After all these steps are done, review the following checklist to make sure everything is ready:

  • X.509 certificates: Check if the X.509 certificates are uploaded. You can check this at your account's Security Credentials page.

  • EC2 key pairs: Check if EC2 key pairs are uploaded. You can check this at AWS Management Console | Amazon EC2 | NETWORK & SECURITY | Key Pairs.

  • Access: Check if the access has been authorized. This can be checked at AWS Management Console | Amazon EC2 | NETWORK & SECURITY | Security Groups | Inbound.

  • Environment variable settings: Check if the environment variable settings are done. As an example, the following snippet shows my settings; make sure you are using the right EC2_URL for your region:

$ cat ~/.bashrc
export EC2_HOME=~/opt/ec2-api-tools-1.4.4.2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=~/.ec2/pk-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export EC2_CERT=~/.ec2/cert-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export JAVA_HOME=/Library/Java/Home
export EC2_URL=https://ec2.us-west-1.amazonaws.com

We need to import our EC2 key pairs to manage EC2 instances via EC2 command-line tools:

$ ec2-import-keypair your-key-pair-name --public-key-file ~/.ssh/id_rsa.pub

Verify the settings by typing the following command:

$ ec2-describe-instances

If everything has been set up properly, the command will show your instances similarly to how you had configured them in the previous command.

Note

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

The last preparation is to find a suitable AMI. An AMI is a preconfigured operating system and software, which is used to create a virtual machine within EC2. We can find a registered Debian AMI at http://wiki.debian.org/Cloud/AmazonEC2Image.

For the purpose of practicing HBase, a 32-bit, EBS-backed AMI is the most cost effective AMI to use. Make sure you are choosing AMIs for your region. As we are using US-West (us-west-1) for this book, the AMI ID for us is ami-77287b32. This is a 32-bit, small instance of EC2. A small instance is good for practicing HBase on EC2 because it's cheap. For production, we recommend you to use at least High-Memory Extra Large Instance with EBS, or a real server.

How to do it...

Follow these instructions to get your EC2 instances ready for HBase. We will start two EC2 instances; one is a DNS/NTP server, and the other one is the client:

  1. 1. Start a micro instance for the mentioned server. We will use ns1.hbase-admin-cookbook.com (ns1) as its Fully Qualified Domain Name (FQDN) , in a later section of this book:

    $ ec2-run-instances ami-77287b32 -t t1.micro -k your-key-pair
    
  2. 2. Start a small instance for the client. We will use client1.hbase-admin-cookbook.com (client1) as its FQDN, later in this book:

    $ ec2-run-instances ami-77287b32 -t m1.small -k your-key-pair
    
  3. 3. Verify the startup from AWS Management Console, or by typing the following command:

    $ ec2-describe-instances
    
    • You should see two instances from the output of the command. From the output of the ec2-describe-instances command, or AWS Management Console, you can find the public DNS of the instances that have already started. The DNS shows a value such as ec2-xx-xx-xxx-xx.us-west-1.compute.amazonaws.com:

  4. 4. Log in to the instances via SSH, using the following command:

  5. 5. Update the package index files before we install packages on the server, by using the following command:

    root# apt-get update
    
  6. 6. Change your instances' time zone to your local timezone, using the following command:

    root# dpkg-reconfigure tzdata
    
  7. 7. Install the NTP server daemon on the DNS server, using the following command:

    [email protected]# apt-get install ntp ntp-server ntpdate
    
  8. 8. Install the NTP client on the client/server, using the following command:

    [email protected]# apt-get install ntp ntpdate
    
  9. 9. Configure /etc/ntp.conf on ns1 to run as an NTP server, and client1 to run as an NTP client, using ns1 as its server.

    Because there is no HBase-specific configuration for the NTP setup, we will skip the details. You can find the sample ntp.conf files for both the server and client, from the sample source of this book.

  10. 10. Install BIND9 on ns1 to run as a DNS server, using the following command:

    [email protected]# apt-get install bind9
    
    • You will need to configure BIND9 to run as a primary master server for internal lookup, and run as a caching server for external lookup. You also need to configure the DNS server, to allow other EC2 instances to update their record on the DNS server.

      We will skip the details as this is out of the scope of this book. For sample BIND9 configuration, please refer to the source, shipped with this book.

  11. 11. For client1, just set it up using ns1 as its DNS server:

    [email protected]# vi /etc/resolv.conf
    nameserver 10.160.49.250 #private IP of ns
    search hbase-admin-cookbook.com #domain name
    
  12. 12. Update the DNS hostname automatically. Set up hostname to the EC2 instance's user data of the client. From the My Instances page of AWS Management Console, select client1 from the instances list, stop it, and then click Instance Actions | View | Change User Data; enter the hostname of the instance you want to use (here client1) in the pop-up page:

  13. 13. Create a script to update the client's record on the DNS server, using user data:

    [email protected]# vi ec2-hostname.sh
    #!/bin/bash
    #you will need to set up your DNS server to allow update from this key
    DNS_KEY=/root/etc/Kuser.hbase-admin-cookbook.com.+157+44141.private
    DOMAIN=hbase-admin-cookbook.com
    USER_DATA=`/usr/bin/curl -s http://169.254.169.254/latest/user-data`
    HOSTNAME=`echo $USER_DATA`
    #set also the hostname to the running instance
    hostname $HOSTNAME
    #we only need to update for local IP
    LOCIP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
    cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
    server ns.$DOMAIN
    zone $DOMAIN
    update delete $HOSTNAME.$DOMAIN A
    update add $HOSTNAME.$DOMAIN 60 A $LOCIP
    send
    EOF
    
  14. 14. Finally, to run this at boot time from rc.local, add the following script to the rc.local file:

    [email protected]# vi /etc/rc.local
    sh /root/bin/ec2-hostname.sh
    

How it works...

First we started two instances, a micro instance for DNS/NTP server, and a small one for client. To provide a name service to other instances, the DNS name server has to be kept running. Using micro instance can reduce your EC2 cost.

In step 3, we set up the NTP server and client. We will run our own NTP server on the same DNS server, and NTP clients on all other servers.

Note

Note: Make sure that the clocks on the HBase cluster members are in basic alignment.

EC2 instances can be started and stopped on demand; we don't need to pay for stopped instances. But, restarting an EC2 instance will change the IP address of the instance, which makes it difficult to run HBase. We can resolve this issue by running a DNS server to provide a name service to all EC2 instances in our HBase cluster. We can update name records on the DNS server every time other EC2 instances are restarted.

That's exactly what we have done in steps 4 and 5. Step 4 is a normal DNS setup. In step 5, we stored the instance name in its user data property at first, so that when the instance is restarted, we can get it back using EC2 API. Also, we will get the private IP address of the instance via EC2 API. With this data, we can then send a DNS update command to our DNS server every time the instance is restarted. As a result, we can always use its fixed hostname to access the instance.

We will keep only the DNS instance running constantly. You can stop all other instances whenever you do not need to run your HBase cluster.

 

Setting up Hadoop


A fully distributed HBase runs on top of HDFS. As a fully distributed HBase cluster installation, its master daemon (HMaster) typically runs on the same server as the master node of HDFS (NameNode), while its slave daemon (HRegionServer) runs on the same server as the slave node of HDFS, which is called DataNode.

Hadoop MapReduce is not required by HBase. MapReduce daemons do not need to be started. We will cover the setup of MapReduce in this recipe too, in case you like to run MapReduce on HBase. For a small Hadoop cluster, we usually have a master daemon of MapReduce (JobTracker) run on the NameNode server, and slave daemons of MapReduce (TaskTracker) run on the DataNode servers.

This recipe describes the setup of Hadoop. We will have one master node (master1) run NameNode and JobTracker on it. We will set up three slave nodes (slave1 to slave3), which will run DataNode and TaskTracker on them, respectively.

Getting ready

You will need four small EC2 instances, which can be obtained by using the following command:

$ec2-run-instances ami-77287b32 -t m1.small -n 4 -k your-key-pair

All these instances must be set up properly, as described in the previous recipe, Getting ready on Amazon EC2. Besides the NTP and DNS setups, Java installation is required by all servers too.

We will use the hadoop user as the owner of all Hadoop daemons and files. All Hadoop files and data will be stored under /usr/local/hadoop. Add the hadoop user and create a /usr/local/hadoop directory on all the servers, in advance.

We will set up one Hadoop client node as well. We will use client1, which we set up in the previous recipe. Therefore, the Java installation, hadoop user, and directory should be prepared on client1 too.

How to do it...

Here are the steps to set up a fully distributed Hadoop cluster:

  1. 1. In order to SSH log in to all nodes of the cluster, generate the hadoop user's public key on the master node:

    [email protected]$ ssh-keygen -t rsa -N ""
    
    • This command will create a public key for the hadoop user on the master node, at ~/.ssh/id_rsa.pub.

  2. 2. On all slave and client nodes, add the hadoop user's public key to allow SSH login from the master node:

    [email protected]$ mkdir ~/.ssh
    [email protected]$ chmod 700 ~/.ssh
    [email protected]$ cat >> ~/.ssh/authorized_keys
    
  3. 3. Copy the hadoop user's public key you generated in the previous step, and paste to ~/.ssh/authorized_keys. Then, change its permission as following:

    [email protected]$ chmod 600 ~/.ssh/authorized_keys
    
  4. 4. Get the latest, stable, HBase-supported Hadoop release from Hadoop's official site, http://www.apache.org/dyn/closer.cgi/hadoop/common/. While this chapter was being written, the latest HBase-supported, stable Hadoop release was 1.0.2. Download the tarball and decompress it to our root directory for Hadoop, then add a symbolic link, and an environment variable:

    [email protected]$ ln -s hadoop-1.0.2 current
    [email protected]$ export HADOOP_HOME=/usr/local/hadoop/current
    
  5. 5. Create the following directories on the master node:

    [email protected]$ mkdir -p /usr/local/hadoop/var/dfs/name
    [email protected]$ mkdir -p /usr/local/hadoop/var/dfs/data
    [email protected]$ mkdir -p /usr/local/hadoop/var/dfs/namesecondary
    
  6. 6. You can skip the following steps if you don't use MapReduce:

    [email protected]$ mkdir -p /usr/local/hadoop/var/mapred
    
  7. 7. Set up JAVA_HOME in Hadoop's environment setting file (hadoop-env.sh):

    [email protected]$ vi $HADOOP_HOME/conf/hadoop-env.sh
    export JAVA_HOME=/usr/local/jdk1.6
    
  8. 8. Add the hadoop.tmp.dir property to core-site.xml:

    [email protected]$ vi $HADOOP_HOME/conf/core-site.xml
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop/var</value>
    </property>
    
  9. 9. Add the fs.default.name property to core-site.xml:

    [email protected]$ vi $HADOOP_HOME/conf/core-site.xml
    <property>
    <name>fs.default.name</name>
    <value>hdfs://master1:8020</value>
    </property>
    
  10. 10. If you need MapReduce, add the mapred.job.tracker property to mapred-site.xml:

    [email protected]$ vi $HADOOP_HOME/conf/mapred-site.xml
    <property>
    <name>mapred.job.tracker</name>
    <value>master1:8021</value>
    </property>
    
  11. 11. Add a slave server list to the slaves file:

    [email protected]$ vi $HADOOP_HOME/conf/slaves
    slave1
    slave2
    slave3
    
  12. 12. Sync all Hadoop files from the master node, to client and slave nodes. Don't sync ${hadoop.tmp.dir} after the initial installation:

    [email protected]$ rsync -avz /usr/local/hadoop/ client1:/usr/local/hadoop/
    [email protected]$ for i in 1 2 3
    do rsync -avz /usr/local/hadoop/ slave$i:/usr/local/hadoop/
    sleep 1
    done
    
  13. 13. You need to format NameNode before starting Hadoop. Do it only for the initial installation:

    [email protected]$ $HADOOP_HOME/bin/hadoop namenode -format
    
  14. 14. Start HDFS from the master node:

    [email protected]$ $HADOOP_HOME/bin/start-dfs.sh
    
  15. 15. You can access your HDFS by typing the following command:

    [email protected]$ $HADOOP_HOME/bin/hadoop fs -ls /
    
    • You can also view your HDFS admin page from the browser. Make sure the 50070 port is opened. The HDFS admin page can be viewed at http://master1:50070/dfshealth.jsp:

  16. 16. Start MapReduce from the master node, if needed:

    [email protected]$ $HADOOP_HOME/bin/start-mapred.sh
    
    • Now you can access your MapReduce admin page from the browser. Make sure the 50030 port is opened. The MapReduce admin page can be viewed at http://master1:50030/jobtracker.jsp:

  17. 17. To stop HDFS, execute the following command from the master node:

    [email protected]$ $HADOOP_HOME/bin/stop-dfs.sh
    
  18. 18. To stop MapReduce, execute the following command from the master node:

    [email protected]$ $HADOOP_HOME/bin/stop-mapred.sh
    

How it works...

To start/stop the daemon on remote slaves from the master node, a passwordless SSH login of the hadoop user is required. We did this in step 1.

HBase must run on a special HDFS that supports a durable sync implementation. If HBase runs on an HDFS that has no durable sync implementation, it may lose data if its slave servers go down. Hadoop versions later than 0.20.205, including Hadoop 1.0.2 which we have chosen, support this feature.

HDFS and MapReduce use local filesystems to store their data. We created directories required by Hadoop in step 3, and set up the path to the Hadoop's configuration file in step 5.

In steps 9 to 11, we set up Hadoop so it could find HDFS, JobTracker, and slave servers. Before starting Hadoop, all Hadoop directories and settings need to be synced with the slave servers. The first time you start Hadoop (HDFS), you need to format NameNode. Note that you should only do this at the initial installation.

At this point, you can start/stop Hadoop using its start/stop script. Here we started/stopped HDFS and MapReduce separately, in case you don't require MapReduce. You can also use $HADOOP_HOME/bin/start-all.sh and stop-all.sh to start/stop HDFS and MapReduce using one command.

 

Setting up ZooKeeper


A distributed HBase depends on a running ZooKeeper cluster. All HBase cluster nodes and clients need to be able to access the ZooKeeper ensemble.

This recipe describes how to set up a ZooKeeper cluster. We will only set up a standalone ZooKeeper node for our HBase cluster, but in production it is recommended that you run a ZooKeeper ensemble of at least three nodes. Also, make sure to run an odd number of nodes.

We will cover the setting up of a clustered ZooKeeper in the There's more... section of this recipe.

Getting ready

First, make sure Java is installed in your ZooKeeper server.

We will use the hadoop user as the owner of all ZooKeeper daemons and files. All the ZooKeeper files and data will be stored under /usr/local/ZooKeeper; you need to create this directory in advance. Our ZooKeeper will be set up on master1 too.

We will set up one ZooKeeper client on client1. So, the Java installation, hadoop user, and directory should be prepared on client1 as well.

How to do it...

To set up a standalone ZooKeeper installation, follow these instructions:

  1. 1. Get the latest stable ZooKeeper release from ZooKeeper's official site, http://ZooKeeper.apache.org/releases.html#download.

  2. 2. Download the tarball and decompress it to our root directory for ZooKeeper. We will set a ZK_HOME environment variable to make the setup easier. As of this writing, ZooKeeper 3.4.3 is the latest stable version:

    [email protected]$ ln -s ZooKeeper-3.4.3 current
    [email protected]$ export ZK_HOME=/usr/local/ZooKeeper/current
    
  3. 3. Create directories for ZooKeeper to store its snapshot and transaction log:

    [email protected]$ mkdir -p /usr/local/ZooKeeper/data
    [email protected]$ mkdir -p /usr/local/ZooKeeper/datalog
    
  4. 4. Create the $ZK_HOME/conf/java.env file and put the Java settings there:

    [email protected]$ vi $ZK_HOME/conf/java.env
    JAVA_HOME=/usr/local/jdk1.6
    export PATH=$JAVA_HOME/bin:$PATH
    
  5. 5. Copy the sample ZooKeeper setting file, and make the following changes to set where ZooKeeper should store its data:

    [email protected]$ cp $ZK_HOME/conf/zoo_sample.cfg $ZK_HOME/conf/zoo.cfg
    [email protected]$ vi $ZK_HOME/conf/zoo.cfg
    dataDir=/usr/local/ZooKeeper/var/data
    dataLogDir=/usr/local/ZooKeeper/var/datalog
    
  6. 6. Sync all files under /usr/local/ZooKeeper from the master node to the client. Don't sync ${dataDir} and ${dataLogDir} after this initial installation.

  7. 7. Start ZooKeeper from the master node by executing this command:

    [email protected]$ $ZK_HOME/bin/zkServer.sh start
    
  8. 8. Connect to the running ZooKeeper, and execute some commands to verify the installation:

    [email protected]$ $ZK_HOME/bin/zkCli.sh -server master1:2181
    [zk: master1:2181(CONNECTED) 0] ls /
    [ZooKeeper]
    [zk: master1:2181(CONNECTED) 1] quit
    
  9. 9. Stop ZooKeeper from the master node by executing the following command:

    [email protected]$ $ZK_HOME/bin/zkServer.sh stop
    

How it works...

In this recipe, we set up a basic standalone ZooKeeper instance. As you can see, the setting is very simple; all you need to do is to tell ZooKeeper where to find Java and where to save its data.

In step 4, we created a file named java.env and placed the Java settings in this file. You must use this filename as ZooKeeper, which by default, gets its Java settings from this file.

ZooKeeper's settings file is called zoo.cfg. You can copy the settings from the sample file shipped with ZooKeeper. The default setting is fine for basic installation. As ZooKeeper always acts as a central role in a cluster system, it should be set up properly to gain the best performance.

To connect to a running ZooKeeper ensemble, use its command-line tool, and specify the ZooKeeper server and port you want to connect to. The default client port is 2181. You don't need to specify it, if you are using the default port setting.

All ZooKeeper data is called a Znode. Znodes are constructed like a filesystem hierarchy. ZooKeeper provides commands to access or update Znode from its command-line tool; type help for more information.

There's more...

As HBase relays ZooKeeper as its coordination service, the ZooKeeper service must be extremely reliable. In production, you must run a ZooKeeper cluster of at least three nodes. Also, make sure to run an odd number of nodes.

The procedure to set up a clustered ZooKeeper is basically the same as shown in this recipe. You can follow the previous steps to set up each cluster node at first. Add the following settings to each node's zoo.cfg, so that every node knows about every other node in the ensemble:

[email protected]{1,2,3}$ vi $ZK_HOME/conf/zoo.cfg
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

Also, you need to put a myid file under ${dataDir}. The myid file consists of a single line containing only the node ID. So myid of node1 would contain the text 1 and nothing else.

Note

Note that clocks on all ZooKeeper nodes must be synchronized. You can use Network Time Protocol (NTP)  to have the clocks synchronized.

Start ZooKeeper from each node of your cluster respectively. Then, you can connect to the cluster from your client, by using the following command:

$ zkCli.sh -server node1,node2,node3

ZooKeeper will function as long as more than half of the nodes in the ZooKeeper cluster are alive. This means, in a three node cluster, only one server can die.

 

Changing the kernel settings


HBase is a database running on Hadoop, and just like other databases, it keeps a lot of files open at the same time. Linux limits the number of file descriptors that any one process may open; the default limits are 1024 per process. To run HBase smoothly, you need to increase the maximum number of open file descriptors for the user, who started HBase. In our case, the user is called hadoop.

You should also increase Hadoop's nproc setting. The nproc setting specifies the maximum number of processes that can exist simultaneously for the user. If nproc is too low, an OutOfMemoryError error may happen.

We will describe how to show and change the kernel settings, in this recipe.

Getting ready

Make sure you have root privileges on all of your servers.

How to do it...

You will need to make the following kernel setting changes to all servers of the cluster:

  1. 1. To confirm the current open file limits, log in as the hadoop user and execute the following command:

    hadoop$ ulimit -n
    1024
    
  2. 2. To show the setting for maximum processes, use the -u option of the ulimit command:

    hadoop$ ulimit -u
    unlimited
    
  3. 3. Log in as the root user to increase open file and nproc limits. Add the following settings to the limits.conf file:

    root# vi /etc/security/limits.conf
    hadoop soft nofile 65535
    hadoop hard nofile 65535
    hadoop soft nproc 32000
    hadoop hard nproc 32000
    
  4. 4. To apply the changes, add the following line into the /etc/pam.d/common-session file:

    root# echo "session required pam_limits.so" >> /etc/pam.d/common-session
    
  5. 5. Log out and back in again, as the hadoop user, and confirm the setting values again; you should see the above changes have been applied:

    hadoop$ ulimit -n
    65535
    hadoop$ ulimit -u
    32000
    

How it works...

The previous setting changes the hadoop user's open file limit to 65535. It also changes the hadoop user's max processes number to 32000. With this change of the kernel setting, HBase can keep enough files open at the same time and also run smoothly.

See also

 

Setting up HBase


A fully distributed HBase instance has one or more master nodes (HMaster), and many slave nodes (RegionServer) running on HDFS. It uses a reliable ZooKeeper ensemble to coordinate all the components of the cluster, including masters, slaves, and clients.

It's not necessary to run HMaster on the same server of HDFS NameNode, but, for a small cluster, it's typical to have them run on the same server, just for ease of management. RegionServers are usually configured to run on servers of HDFS DataNode. Running RegionServer on the DataNode server has the advantage of data locality too. Eventually, DataNode running on the same server, will have a copy on it of all the data that RegionServer requires.

This recipe describes the setup of a fully distributed HBase. We will set up one HMaster on master1, and three region servers (slave1 to slave3). We will also set up an HBase client on client1.

Getting ready

First, make sure Java is installed on all servers of the cluster.

We will use the hadoop user as the owner of all HBase daemons and files, too. All HBase files and data will be stored under /usr/local/hbase. Create this directory on all servers of your HBase cluster, in advance.

We will set up one HBase client on client1. Therefore, the Java installation, hadoop user, and directory should be prepared on client1 too.

Make sure HDFS is running. You can ensure it started properly by accessing HDFS, using the following command:

[email protected]$ $HADOOP_HOME/bin/hadoop fs -ls /

MapReduce does not need to be started, as HBase does not normally use it.

We assume that you are managing your own ZooKeeper, in which case, you can start it and confirm if it is running properly. You can ensure it is running properly by sending the ruok command to its client port:

[email protected]$ echo ruok | nc master1 2181

How to do it...

To set up our fully distributed HBase cluster, we will download and configure HBase on the master node first, and then sync to all slave nodes and clients.

Get the latest stable HBase release from HBase's official site, http://www.apache.org/dyn/closer.cgi/hbase/.

At the time of writing this book, the current stable release was 0.92.1.

  1. 1. Download the tarball and decompress it to our root directory for HBase. Also, set an HBASE_HOME environment variable to make the setup easier:

    [email protected]$ ln -s hbase-0.92.1 current
    [email protected]$ export HBASE_HOME=/usr/local/hbase/current
    
  2. 2. We will use /usr/local/hbase/var as a temporary directory of HBase on the local filesystem. Remove it first if you have created it for your standalone HBase installation:

    [email protected]$ mkdir -p /usr/local/hbase/var
    
  3. 3. To tell HBase where the Java installation is, set JAVA_HOME in the HBase environment setting file (hbase-env.sh):

    [email protected]$ vi $HBASE_HOME/conf/hbase-env.sh
    # The java implementation to use. Java 1.6 required.
    export JAVA_HOME=/usr/local/jdk1.6
    
  4. 4. Set up HBase to use the independent ZooKeeper ensemble:

    [email protected]$ vi $HBASE_HOME/conf/hbase-env.sh
    # Tell HBase whether it should manage it's own instance of ZooKeeper or not.
    export HBASE_MANAGES_ZK=false
    
  5. 5. Add these settings to HBase's configuration file (hbase-site.xml):

    [email protected]$ vi $HBASE_HOME/conf/hbase-site.xml
    <configuration>
    <property>
    <name>hbase.rootdir</name>
    <value>hdfs://master1:8020/hbase</value>
    </property>
    <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    </property>
    <property>
    <name>hbase.tmp.dir</name>
    <value>/usr/local/hbase/var</value>
    </property>
    <property>
    <name>hbase.ZooKeeper.quorum</name>
    <value>master1</value>
    </property>
    </configuration>
    
  6. 6. Configure the slave nodes of the cluster:

    [email protected]$ vi $HBASE_HOME/conf/regionservers
    slave1
    slave2
    slave3
    
  7. 7. Link the HDFS configuration file (hdfs-site.xml) to HBase's configuration folder (conf), so that HBase can see the HDFS's client configuration on your Hadoop cluster:

    [email protected]$ ln -s $HADOOP_HOME/conf/hdfs-site.xml $HBASE_HOME/conf/hdfs-site.xml
    
  8. 8. Copy the hadoop-core and Zookeeper JAR file, and their dependencies, from your Hadoop and ZooKeeper installation:

    [email protected]$ rm -i $HBASE_HOME/lib/hadoop-core-*.jar
    [email protected]$ rm -i $HBASE_HOME/lib/ZooKeeper-*.jar
    [email protected]$ cp -i $HADOOP_HOME/hadoop-core-*.jar $HBASE_HOME/lib/
    [email protected]$ cp -i $HADOOP_HOME/lib/commons-configuration-1.6.jar $HBASE_HOME/lib/
    [email protected]$ cp -i $ZK_HOME/ZooKeeper-*.jar $HBASE_HOME/lib/
    
  9. 9. Sync all the HBase files under /usr/local/hbase from master, to the same directory as client and slave nodes.

  10. 10. Start the HBase cluster from the master node:

    [email protected]$ $HBASE_HOME/bin/start-hbase.sh
    
  11. 11. Connect to your HBase cluster from the client node:

    [email protected]$ $HBASE_HOME/bin/hbase shell
    
    • You can also access the HBase web UI from your browser. Make sure your master server's 60010 port is opened. The URL is http://master1:60010/master.jsp:

  12. 12. Stop the HBase cluster from the master node:

    [email protected]$ $HBASE_HOME/bin/stop-hbase.sh
    

How it works...

Our HBase cluster is configured to use /hbase as its root directory on HDFS, by specifying the hbase.rootdir property. Because it is the first time HBase was started, it will create the directory automatically. You can see the files HBase created on HDFS from the client:

[email protected]$ $HADOOP_HOME/bin/hadoop fs -ls /hbase

We want our HBase to run on distributed mode, so we set hbase.cluster.distributed to true in hbase-site.xml.

We also set up the cluster to use an independent ZooKeeper ensemble by specifying HBASE_MANAGES_ZK=false in hbase-env.sh. The ZooKeeper ensemble is specified by the hbase.ZooKeeper.quorum property. You can use clustered ZooKeeper by listing all the servers of the ensemble, such as zoo1,zoo2,zoo3.

All region servers are configured in the $HBASE_HOME/conf/regionservers file. You should use one line per region server. When starting the cluster, HBase will SSH into each region server configured here, and start the HRegionServer daemon on that server.

By linking hdfs-site.xml under the $HBASE_HOME/conf directory, HBase will use all the client configurations you made for your HDFS in hdfs-site.xml, such as the dfs.replication setting.

HBase ships with its prebuilt hadoop-core and ZooKeeper JAR files. They may be out of date, compared to what you used in your Hadoop and ZooKeeper installation. Make sure HBase uses the same version of .jar files with Hadoop and ZooKeeper, to avoid any unexpected problems.

 

Basic Hadoop/ZooKeeper/HBase configurations


There are some basic settings we should tune, before moving forward. These are very basic and important Hadoop (HDFS), ZooKeeper, and HBase settings that you should consider to change immediately after setting up your cluster.

Some of these settings take effect due to data durability or cluster availability, which must be configured, while some are recommended configurations for running HBase smoothly.

Configuration settings depend on your hardware, data, and cluster size. We will describe a guideline in this recipe. You may need to change the settings to fit your environment.

Every time you make changes, you need to sync to all clients and slave nodes, then restart the respective daemon to apply the changes.

How to do it...

The configurations that should be considered for change are as follows:

  1. 1. Turn on dfs.support.append for HDFS. The dfs.support.append property determines whether HDFS should support the append (sync) feature or not. The default value is false. It must be set to true, or you may lose data if the region server crashes:

    hadoop$ vi $HADOOP_HOME/conf/hdfs-site.xml
    
    <property>
    <name>dfs.support.append</name>
    <value>true</value>
    </property>
    
  2. 2. Increase the dfs.datanode.max.xcievers value to have DataNode keep more threads open, to handle more concurrent requests:

    hadoop$ vi $HADOOP_HOME/conf/hdfs-site.xml
    <property>
    <name>dfs.datanode.max.xcievers</name>
    <value>4096</value>
    </property>
    
  3. 3. Increase ZooKeeper's heap memory size so that it does not swap:

    hadoop$ vi $ZK_HOME/conf/java.env
    export JAVA_OPTS="-Xms1000m -Xmx1000m"
    
  4. 4. Increase ZooKeeper's maximum client connection number to handle more concurrent requests:

    hadoop$ echo "maxClientCnxns=60" >> $ZK_HOME/conf/zoo.cfg
    
  5. 5. Increase HBase's heap memory size to run HBase smoothly:

    hadoop$ vi $HBASE_HOME/conf/hbase-env.sh
    
    export HBASE_HEAPSIZE=8000
    
  6. 6. Decrease the zookeeper.session.timeout value so that HBase can find the crashed region server fast, and recover it in a short time:

    hadoop$ vi $HBASE_HOME/conf/hbase-site.xml
    <property>
    <name>zookeeper.session.timeout</name>
    <value>60000</value>
    </property>
    
  7. 7. To change Hadoop/ZooKeeper/HBase log settings, edit the log4j.properties file and the hadoop-env.sh/hbase-env.sh file under the conf directory of the Hadoop/ZooKeeper/HBase installation. It's better to change the log directory out of the installation folder. For example, the following specifies HBase to generate its logs under the /usr/local/hbase/logs directory:

    hadoop$ vi $HBASE_HOME/conf/hbase-env.sh
    export HBASE_LOG_DIR=/usr/local/hbase/logs
    

How it works...

In step 1, by turning on dfs.support.append, the HDFS flush is enabled. With this feature enabled, a writer of HDFS can guarantee that data will be persisted by invoking a flush call. So, HBase can guarantee that when a region server dies, data can be recovered and replayed on other region servers using its Write-Ahead Log (WAL) .

To verify if the HDFS append is supported or not, see your HMaster log of the HBase startup. If the append is not turned to on, you will find a log like the following:

$ grep -i "HDFS-200" hbase-hadoop-master-master1.log
...syncFs -- HDFS-200 -- not available, dfs.support.append=false

For step 2, we configured the dfs.datanode.max.xcievers setting, which specifies the upper bound on the number of files HDFS DataNode will serve at any one time.

Note

Note that the name is xcievers—it's a misspelled name. Its default value is 256, which is too low for running HBase on HDFS.

Steps 3 and 4 are about ZooKeeper settings. ZooKeeper is very sensitive to swapping, which will seriously degrade its performance. ZooKeeper's heap size is set in the java.env file. ZooKeeper has an upper bound on the number of connections it will serve at any one time. Its default is 10, which is too low for HBase, especially when running MapReduce on it. We would suggest setting it to 60.

In step 5, we configured HBase's heap memory size. HBase ships with a heap size of 1 GB, which is too low for modern machines. A reasonable value for large machines is 8 GB or larger, but under 16 GB.

In step 6, we changed the ZooKeeper's session timeout to a lower value. Lower timeout means HBase can find crashed region servers faster, and thus, recover the crashed regions on other servers in a short time. On the other hand, with a very short session timeout, there is a risk that the HRegionServer daemon may kill itself when the cluster is in heavy load, because it may not be able to send a heartbeat to the ZooKeeper before getting a timeout.

See also

  • Chapter 8, Basic Performance Tuning

  • Chapter 9, Advanced Confi gurations and Performance Tuning

 

Setting up multiple High Availability (HA) masters


Hadoop and HBase are designed to handle the failover of their slave nodes automatically. Because there may be many nodes in a large cluster, a hardware failure of a server or shut down of a slave node are considered as normal in the cluster.

For the master nodes, HBase itself has no SPOF. HBase uses ZooKeeper as its central coordination service. A ZooKeeper ensemble is typically clustered with three or more servers; as long as more than half of the servers in the cluster are online, ZooKeeper can provide its service normally.

HBase saves its active master node, root region server location, and other important running data in ZooKeeper. Therefore, we can just start two or more HMaster daemons on separate servers and the one started first will be the active master server of the HBase cluster.

But, NameNode of HDFS is the SPOF of the cluster. NameNode keeps the entire HDFS's filesystem image in its local memory. HDFS cannot function anymore if NameNode is down, as HBase is down too. As you may notice, there is a Secondary NameNode of HDFS. Note that Secondary NameNode is not a standby of NameNode, it just provides a checkpoint function to NameNode. So, the challenge of a highly available cluster is to make NameNode highly available.

In this recipe, we will describe the setup of two highly available master nodes, which will use Heartbeat to monitor each other. Heartbeat is a widely used HA solution to provide communication and membership for a Linux cluster. Heartbeat needs to be combined with a Cluster Resource Manager (CRM)  to start/stop services for that cluster. Pacemaker is the preferred cluster resource manager for Heartbeat. We will set up a Virtual IP (VIP) address using Heartbeat and Pacemaker, and then associate it with the active master node. Because EC2 does not support static IP addresses, we cannot demonstrate it on EC2, but we will discuss an alternative way of using Elastic IP (EIP) to achieve our purpose.

We will focus on setting up NameNode and HBase; you can simply use a similar method to set up two JobTracker nodes as well.

Getting ready

You should already have HDFS and HBase installed. We will set up a standby master node (master2), as you need another server ready to use. Make sure all the dependencies have been configured properly. Sync your Hadoop and HBase root directory from the active master (master1) to the standby master.

We will need NFS in this recipe as well. Set up your NFS server, and mount the same NFS directory from both master1 and master2. Make sure the hadoop user has write permission to the NFS directory. Create a directory on NFS to store Hadoop's metadata. We assume the directory is /mnt/nfs/hadoop/dfs/name.

We will set up VIP for the two masters, and assume you have the following IP addresses and DNS mapping:

  • master1: This has its IP address as 10.174.14.11.

  • master2: This has its IP address as 10.174.14.12.

  • master: This has its IP address as 10.174.14.10. It is the VIP that will be set up later.

How to do it...

The following instructions describe how to set up two highly available master nodes.

Install and configure Heartbeat and Pacemaker

First, we will install Heartbeat and Pacemaker, and make some basic configurations:

  1. 1. Install Heartbeat and Pacemaker on master1 and master2:

    root# apt-get install heartbeat cluster-glue cluster-agents pacemaker
    
  2. 2. To configure Heartbeat, make the following changes to both master1 and master2:

    root# vi /etc/ha.d/ha.cf
    # enable pacemaker, without stonith
    crm yes
    # log where ?
    logfacility local0
    # warning of soon be dead
    warntime 10
    # declare a host (the other node) dead after:
    deadtime 20
    # dead time on boot (could take some time until net is up)
    initdead 120
    # time between heartbeats
    keepalive 2
    # the nodes
    node master1
    node master2
    # heartbeats, over dedicated replication interface!
    ucast eth0 master1 # ignored by master1 (owner of ip)
    ucast eth0 master2 # ignored by master2 (owner of ip)
    # ping the name server to assure we are online
    ping ns
    
  3. 3. Create an authkeys file. Execute the following script as a root user on master1 and master2:

    root# ( echo -ne "auth 1\n1 sha1 "; \
    dd if=/dev/urandom bs=512 count=1 | openssl md5 ) \
    > /etc/ha.d/authkeys
    root# chmod 0600 /etc/ha.d/authkeys
    

Create and install a NameNode resource agent

Pacemaker depends on a resource agent to manager the cluster. A resource agent is an executable that manages a cluster resource. In our case, the VIP address and the HDFS NameNode service is the cluster resource we want to manage, using Pacemaker. Pacemaker ships with an IPaddr resource agent to manage VIP, so we only need to create our own namenode resource agent:

  1. 1. Add environment variables to the .bashrc file of the root user on master1 and master2. Don't forget to apply the changes:

    root# vi /root/.bashrc
    export JAVA_HOME=/usr/local/jdk1.6
    export HADOOP_HOME=/usr/local/hadoop/current
    export OCF_ROOT=/usr/lib/ocf
    

    Invoke the following command to apply the previous changes:

    root# source /root/.bashrc
    
  2. 2. Create a standard Open Clustering Framework (OCF) resource agent file called namenode, with the following content.

    The namenode resource agent starts with including standard OCF functions such as the following:

    root# vi namenode
    #!/bin/sh
    : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/resource.d/heartbeat}
    . ${OCF_FUNCTIONS_DIR}/.ocf-shellfuncs
    usage() {
    echo "Usage: $0 {start|stop|status|monitor|meta-data|validate-all}"
    }
    
  3. 3. Add a meta_data() function as shown in the following code. The meta_data() function dumps the resource agent metadata to standard output. Every resource agent must have a set of XML metadata describing its own purpose and supported parameters:

    root# vi namenode
    meta_data() {cat <<END
    <?xml version="1.0"?>
    <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
    <resource-agent name="namenode">
    <version>0.1</version>
    <longdesc lang="en">
    This is a resource agent for NameNode. It manages HDFS namenode daemon.
    </longdesc>
    <shortdesc lang="en">Manage namenode daemon.</shortdesc>
    <parameters></parameters>
    <actions>
    <action name="start" timeout="120" />
    <action name="stop" timeout="120" />
    <action name="status" depth="0" timeout="120" interval="120" />
    <action name="monitor" depth="0" timeout="120" interval="120" />
    <action name="meta-data" timeout="10" />
    <action name="validate-all" timeout="5" />
    </actions>
    </resource-agent>
    END
    }
    
  4. 4. Add a namenode_start() function. This function is used by Pacemaker to actually start the NameNode daemon on the server. In the namenode_start() function, we firstly check whether NameNode is already started on the server; if it is not started, we invoke hadoop-daemon.sh from the hadoop user to start it:

    root# vi namenode
    namenode_start() {
    # if namenode is already started on this server, bail out early
    namenode_status
    if [ $? -eq 0 ]; then
    ocf_log info "namenode is already running on this server, skip"
    return $OCF_SUCCESS
    fi
    # start namenode on this server
    ocf_log info "Starting namenode daemon..."
    su - hadoop -c "${HADOOP_HOME}/bin/hadoop-daemon.sh start name node"
    if [ $? -ne 0 ]; then
    ocf_log err "Can not start namenode daemon."
    return $OCF_ERR_GENERIC;
    fi
    sleep 1
    return $OCF_SUCCESS
    }
    
  5. 5. Add a namenode_stop() function. This function is used by Pacemaker to actually stop the NameNode daemon on the server. In the namenode_stop() function, we first check whether NameNode is already stopped on the server; if it is running, we invoke hadoop-daemon.sh from the hadoop user to stop it:

    root# vi namenode
    
    namenode_stop () {
    # if namenode is not started on this server, bail out early
    namenode_status
    if [ $? -ne 0 ]; then
    ocf_log info "namenode is not running on this server, skip"
    return $OCF_SUCCESS
    fi
    # stop namenode on this server
    ocf_log info "Stopping namenode daemon..."
    su - hadoop -c "${HADOOP_HOME}/bin/hadoop-daemon.sh stop name node"
    if [ $? -ne 0 ]; then
    ocf_log err "Can not stop namenode daemon."
    return $OCF_ERR_GENERIC;
    fi
    sleep 1
    return $OCF_SUCCESS
    }
    
  6. 6. Add a namenode_status() function. This function is used by Pacemaker to monitor the status of the NameNode daemon on the server. In the namenode_status() function, we use the jps command to show all running Java processes owned by the hadoop user, and the grep name of the NameNode daemon to see whether it has started:

    root# vi namenode
    namenode_status () {
    ocf_log info "monitor namenode"
    su - hadoop -c "${JAVA_HOME}/bin/jps" | egrep -q "NameNode"
    rc=$?
    # grep will return true if namenode is running on this machine
    if [ $rc -eq 0 ]; then
    ocf_log info "Namenode is running"
    return $OCF_SUCCESS else
    ocf_log info "Namenode is not running" return $OCF_NOT_RUNNING
    fi
    }
    
  7. 7. Add a namenode_validateAll() function to make sure the environment variables are set properly before we run other functions:

    root# vi namenode
    namenode_validateAll () {
    if [ -z "$JAVA_HOME" ]; then
    ocf_log err "JAVA_HOME not set."
    exit $OCF_ERR_INSTALLED
    fi
    if [ -z "$HADOOP_HOME" ]; then
    ocf_log err "HADOOP_HOME not set."
    exit $OCF_ERR_INSTALLED
    fi
    # Any subject is OK
    return $OCF_SUCCESS
    }
    
  8. 8. Add the following main routine. Here, we will simply call the previous functions to implement the required standard OCF resource agent actions:

    root# vi namenode
    # See how we were called.
    if [ $# -ne 1 ]; then
    usage
    exit $OCF_ERR_GENERIC
    fi
    namenode_validateAll
    case $1 in
    meta-data) meta_data
    exit $OCF_SUCCESS;;
    usage) usage
    exit $OCF_SUCCESS;;
    *);;
    esac
    case $1 in
    status|monitor) namenode_status;;
    start) namenode_start;;
    stop) namenode_stop;;
    validate-all);;
    *)usage
    exit $OCF_ERR_UNIMPLEMENTED;;
    esac
    exit $?
    
  9. 9. Change the namenode file permission and test it on master1 and master2:

    root# chmod 0755 namenode
    root# ocf-tester -v -n namenode-test /full/path/of/namenode
    
  10. 10. Make sure all the tests are passed before proceeding to the next step, or the HA cluster will behave unexpectedly.

  11. 11. Install the namenode resource agent under the hac provider on master1 and master2:

    root# mkdir ${OCF_ROOT}/resource.d/hac
    root# cp namenode ${OCF_ROOT}/resource.d/hac
    root# chmod 0755 ${OCF_ROOT}/resource.d/hac/namenode
    

Configure highly available NameNode

We are ready to configure highly available NameNode using Heartbeat and Pacemaker. We will set up a VIP address and configure Hadoop and HBase to use this VIP address as their master node. NameNode will be started on the active master where VIP is assigned. If active master has crashed, Heartbeat and Pacemaker will detect it and assign the VIP address to the standby master node, and then start NameNode there.

  1. 1. Start Heartbeat on master1 and master2:

    root# /etc/init.d/heartbeat start
    
  2. 2. Change the default crm configuration. All resource-related commands are only executed once, from master1 or master2:

    root# crm configure property stonith-enabled=false
    root# crm configure property default-resource-stickiness=1
    
  3. 3. Add a VIP resource using our VIP address:

    root# crm configure primitive VIP ocf:heartbeat:IPaddr params ip="10.174.14.10" op monitor interval="10s"
    
  4. 4. Make the following changes to configure Hadoop to use our VIP address. Sync to all masters, clients, and slaves after you've made the changes:

    hadoop$ vi $HADOOP_HOME/conf/core-site.xml
    <property>
    <name>fs.default.name</name>
    <value>hdfs://master:8020</value>
    </property>
    
  5. 5. Make the following changes to configure HBase to use our VIP address. Sync to all masters, clients, and slaves after you've made the changes:

    hadoop$ vi $HBASE_HOME/conf/hbase-site.xml
    <property>
    <name>hbase.rootdir</name>
    <value>hdfs://master:8020/hbase</value>
    </property>
    
  6. 6. To configure Hadoop to write its metadata to a local disk and NFS, make the following changes and sync to all masters, clients, and slaves:

    hadoop$ vi $HADOOP_HOME/conf/hdfs-site.xml
    <property>
    <name>dfs.name.dir</name>
    <value>/usr/local/hadoop/var/dfs/name,/mnt/nfs/hadoop /dfs/name</value>
    </property>
    
  7. 7. Add the namenode resource agent we created in step 5 to Pacemaker. We will use NAMENODE as its resource name:

    root# crm configure primitive NAMENODE ocf:hac:namenode op monitor interval="120s" timeout="120s" op start timeout="120s" op stop timeout="120s" meta resource-stickiness="1"
    
  8. 8. Configure the VIP resource and the NAMENODE resource as a resource group:

    root# crm configure group VIP-AND-NAMENODE VIP NAMENODE
    
  9. 9. Configure colocation of a VIP resource and the NAMENODE resource:

    root# crm configure colocation VIP-WITH-NAMENODE inf: VIP NAMENODE
    
  10. 10. Configure the resource order of the VIP resource and the NAMENODE resource:

    root# crm configure order IP-BEFORE-NAMENODE inf: VIP NAMENODE
    
  11. 11. Verify the previous Heartbeat and resource configurations by using the crm_mon command. If everything is configured properly, you should see an output like the following :

    [email protected] hac$ crm_mon -1r
    ============
    Last updated: Tue Nov 22 22:39:11 2011
    Stack: Heartbeat
    Current DC: master2 (7fd92a93-e071-4fcb-993f-9a84e6c7846f) - partition with quorum
    Version: 1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b
    2 Nodes configured, 1 expected votes
    1 Resources configured.
    ============
    Online: [ master1 master2 ]
    Full list of resources:
    Resource Group: VIP-AND-NAMENODE
    VIP (ocf::heartbeat:IPaddr): Started master1
    NAMENODE (ocf::hac:namenode): Started master1
    
  12. 12. Make sure that the VIP and NAMENODE resources are started on the same server.

  13. 13. Now stop Heartbeat from master1; VIP-AND-NAMENODE should be started at master2 after several seconds.

  14. 14. Restart Heartbeat from master1; VIP-AND-NAMENODE should remain started at master2. Resources should NOT failback to master1.

Start DataNode, HBase cluster, and backup HBase master

We have confirmed that our HA configuration works as expected, so we can start HDFS and HBase now. Note that NameNode has already been started by Pacemaker, so we need only start DataNode here:

  1. 1. If everything works well, we can start DataNode now:

    [email protected]$ for i in 1 2 3
    do
    ssh slave$i "$HADOOP_HOME/bin/hadoop-daemon.sh start datanode"
    sleep 1
    done
    
  2. 2. Start your HBase cluster from master, which is the active master server where the VIP address is associated:

    [email protected]$ $HBASE_HOME/bin/start-hbase.sh
    
  3. 3. Start standby HMaster from the standby master server, master2 in this case:

    [email protected]$ $HBASE_HOME/bin/hbase-daemon.sh start master
    

How it works...

The previous steps finally leave us with a cluster structure like the following diagram:

At first, we installed Heartbeat and Pacemaker on the two masters and then configured Heartbeat to enable Pacemaker.

In step 2 of the Create and install a NameNode resource agent section, we created the namenode script, which is implemented as a standard OCF resource agent. The most important function of the namenode script is namenode_status, which monitors the status of the NameNode daemon. Here we use the jps command to show all running Java processes owned by the hadoop user, and the grep name of the NameNode daemon to see if it has started. The namenode resource agent is used by Pacemaker to start/stop/monitor the NameNode daemon. In the namenode script, as you can see in the namenode_start and namenode_stop methods, we actually start/stop NameNode by using hadoop-daemon.sh, which is used to start/stop the Hadoop daemon on a single server. You can find a full list of the code from the source shipped with this book.

We started Heartbeat after our namenode resource agent was tested and installed. Then, we made some changes to the default crm configurations. The default-resource-stickiness=1 setting is very important as it turns off the automatic failback of a resource.

We added a VIP resource to Pacemaker and configured Hadoop and HBase to use it in steps 3 to 5 of the Configure highly available NameNode section. By using VIP in their configuration, Hadoop and HBase can switch to communicate with the standby master if the active one is down.

In step 6 of the same section, we configured Hadoop (HDFS NameNode) to write its metadata to both the local disk and NFS. If an active master is down, NameNode will be started from the standby master. Because they were mounted on the same NFS directory, NameNode started from the standby master can apply the latest metadata from NFS, and restore HDFS to the status before the original active master is down.

In steps 7 to 10, we added the NAMENODE resource using the namenode resource agent we created in step 2 of the Create and install a NameNode resource agent section, then we set up VIP and NAMENODE resources as a group (step 8), and made sure they always run on the same server (step 9), with the right start-up order (step 10). We did this because we didn't want VIP running on master1, while NameNode was running on master2.

Because Pacemaker will start NameNode for us via the namenode resource agent, we need to start DataNode separately, which is what we did in step 1 of the Start DataNode, HBase cluster, and backup HBase master section.

After starting HBase normally, we started our standby HBase master (HMaster) on the standby master server. If you check your HBase master log, you will find output like the following, which shows itself as a standby HMaster:

2011-11-21 23:38:55,168 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the active master, ip-10-174-14-15.us-west-1.compute.internal:60000; waiting to become the next active master

Finally, we got NameNode and HMaster running on two servers with an active-standby configuration. The single point of failure of the cluster was avoided.

However, it leaves us with lots of works to do in production. You need to test your HA cluster in all rare cases, such as a server power off, unplug of a network cable, shut down of network switch, or anything else you can think of.

On the other hand, SPOF of the cluster may not be as critical as you think. Based on our experience, almost all of the downtime of the cluster is due to an operational miss or software upgrade. It's better to make your cluster simple.

There's more...

It is more complex to set up a highly available HBase cluster on Amazon EC2 because EC2 does not support static IP addresses, and so we can't use VIP on EC2. An alternative way is to use an Elastic IP address. An Elastic IP address is the role of a static IP address on EC2 while it is associated with your account, not a particular instance. We can use Heartbeat to associate EIP to the standby master automatically, if the active one is down. Then, we configure Hadoop and HBase to use an instance's public DNS associated with EIP, to find an active master. Also, in the namenode resource agent, we have to start/stop not only NameNode, but also all DataNodes. This is because the IP address of an active master has changed, but DataNode cannot find the new active master unless it is restarted.

We will skip the details because it's out of the scope of this book. We created an elastic-ip resource agent to achieve this purpose. You can find it in the source shipped with this book.

About the Author

  • Yifeng Jiang

    Yifeng Jiang is a Hadoop and HBase Administrator and Developer at Rakuten—the largest e-commerce company in Japan. After graduating from the University of Science and Technology of China with a B.S. in Information Management Systems, he started his career as a professional software engineer, focusing on Java development. In 2008, he started looking over the Hadoop project. In 2009, he led the development of his previous company's display advertisement data infrastructure using Hadoop and Hive. In 2010, he joined his current employer, where he designed and implemented the Hadoop- and HBase-based, large-scale item ranking system. He is also one of the members of the Hadoop team in the company, which operates several Hadoop/HBase clusters

    Browse publications by this author
Book Title
Unlock this full book FREE 10 day trial
Start Free Trial