HBase High Performance Cookbook

The first step will be to create a directory at user/u/HBase B and download the TAR file from the location given later. The location can be local, mount points or in cloud environments; it can be block storage:

wget wget –b http://apache.mirrors.pair.com/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz

Untar it using the following commands:

tar -xzvf hadoop-2.2.0.tar.gz

This is used to untar the file in a folder hadoop-2.2.0 in your current directory location.

Once the untar process is done, for clarity it's recommended use two different folders one for NameNode and other for DataNode.

To keep the NameNodeData and the DataNodeData for clarity let's create two folders by using the following command, inside /u/HBase B:

Mkdir NameNodeData DataNodeData

NameNodeData will have the data which is used by the name nodes and DataNodeData will have the data which will be used by the data nodes:

ls –ltr will show the below results.
drwxrwxr-x 2 app app  4096 Jun 19 22:22 NameNodeData
drwxrwxr-x 2 app app  4096 Jun 19 22:22 DataNodeData

-bash-4.1$ pwd
/u/HBase B/hadoop-2.2.0
-bash-4.1$ ls -ltr
total 60K
drwxr-xr-x 2 app app 4.0K Mar 31 08:49 bin
drwxrwxr-x 2 app app 4.0K Jun 19 22:22 DataNodeData
drwxr-xr-x 3 app app 4.0K Mar 31 08:49 etc

The steps in choosing Hadoop cluster are:

HDFS core architecture is based on master/slave, where an HDFS cluster comprises of solo NameNode, which is essentially used as a master node, and owns the accountability for that orchestrating, handling the file system, namespace, and controlling access to files by client. It performs this task by storing all the modifications to the underlying file system and propagates these changes as logs, appends to the native file system files, and edits. SecondaryNameNode is designed to merge the fsimage and the edits log files regularly and controls the size of edit logs to an acceptable limit.

In a true cluster/distributed environment, it runs on a different machine. It works as a checkpoint in HDFS.

We will require the following for the NameNode:

RAID is nothing but a random access inexpensive drive or independent disk. There are many levels of RAID drives, but for master or a NameNode, RAID 1 will be enough.

JBOD stands for Just a bunch of Disk. The design is to have multiple hard drives stacked over each other with no redundancy. The calling software needs to take care of the failure and redundancy. In essence, it works as a single logical volume:

Before we start for the cluster setup, a quick recap of the Hadoop setup is essential with brief descriptions.

Let's create a directory where you will have all the software components to be downloaded:

Let's assume that there is a /u directory and you have downloaded the entire stack of software from: /u/HBase B/hadoop-2.2.0/etc/hadoop/ and look for the file core-site.xml.

Place the following lines in this configuration file:

<configuration>
<property>
    <name>fs.default.name</name>
    <value>hdfs://addressofbsdnsofmynamenode-hadoop:9001</value>
 </property>
</configuration>

Save the file. This helps us create a master /NameNode.

Now, let's move to set up SecondryNodes, let's edit /u/HBase B/hadoop-2.2.0/etc/hadoop/ and look for the file core-site.xml:

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://custome location of your hdfs</value>
</property>
<configuration>
<property>       
    <name>fs.checkpoint.dir</name>       
    <value>/u/HBase B/dn001/hadoop/hdf/secdn
        /u/HBase B/dn002/hadoop/hdfs/secdn 
</value>   
</property>
</configuration>

Now, let's move towards changing the setup for hdfs; the file location will be /u/HBase B/hadoop-2.2.0/etc/hadoop/hdfs-site.xml.

Add these properties in hdfs-site.xml:

For NameNode:

<property>         
<name>dfs.name.dir</name>         
<value>
/u/HBase B/nn01/hadoop/hdfs/nn,/u/HBase B/nn02/hadoop/hdfs/nn
</value>     
</property>

For DataNode:

<property>         
<name>dfs.data.dir</name>         
<value>
/u/HBase B/dnn01/hadoop/hdfs/dn,/HBase B/u/dnn02/hadoop/hdfs/dn
</value> 
</property>

Now, let's go for NameNode for http address or to access using http protocol:

<property> 
<name>dfs.http.address</name> 
<value>yournamenode.full.hostname:50070</value>
</property>
<property> 
<name>dfs.secondary.http.address</name>
<value>
secondary.yournamenode.full.hostname:50090
</value>     
</property>

We can go for the https setup for the NameNode too, but let's keep it optional for now:

Let's set up the yarn resource manager:

Now, let's make the changes for Map reduce:

ZooKeeper: Now, let's focus on the setup of ZooKeeper. In distributed env, let's go to this location and rename the zoo_sample.cfg to zoo.cfg:

/u/HBase B/zookeeper-3.4.6/conf

Open zoo.cfg by vi zoo.cfg and place the details as follows; this will create two instances of zookeeper on different ports:

yourzooKeeperserver.1=zoo1:2888:3888 
yourZooKeeperserver.2=zoo2:2888:3888

If you want to test this setup locally, please use different port combinations. In a production-like setup as mentioned earlier, yourzooKeeperserver.1=zoo1:2888:3888 is server.id=host:port:port:

yourzooKeeperserver.1= server.id
zoo1=host
2888=port
3888=port

Atomic broadcasting is an atomic messaging system that keeps all the servers in sync and provides reliable delivery, total order, casual order, and so on.

Region servers: Before concluding it, let's go through the region server setup process.

Go to this folder /u/HBase B/HBase -0.98.3-hadoop2/conf and edit the regionserver file.

Specify the region servers accordingly:

RegionServer1 
RegionServer2 
RegionServer3 
RegionServer4

Copy all the configuration files of HBase and ZooKeeper to the relative host dedicated for HBase and ZooKeeper. As the setup is in a fully distributed cluster mode, we will be using a different host for HBase and its components and a dedicated host for ZooKeeper.

Next, we validate the setup we've worked on by adding the following to the bashrc, this will make sure later we are able to configure the NameNode as expected:

Now let's format NameNode:

Sudo su $HDFS_USER 
/u/HBase B/hadoop-2.2.0/bin/hadoop namenode -format 
HDFS is implemented on the existing local file system of your cluster. When you want to start the Hadoop setup first time you need to start with a clean slate and hence any existing data needs to be formatted and erased.

Before formatting we need to take care of the following.

Check whether there is a Hadoop cluster running and using the same HDFS; if it's done accidentally all the data will be lost.

/u/HBase B/hadoop-2.2.0/sbin/hadoop-daemon.sh --config
$HADOOP_CONF_DIR start namenode

Now let's go to the SecondryNodes:

Sudo su $HDFS_USER
/u/HBase B/hadoop-2.2.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start secondarynamenode

Repeating the same procedure in DataNode:

Sudo su $HDFS_USER
/u/HBase B/hadoop-2.2.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode
Test 01>

See if you can reach from your browser http://namenode.full.hostname:50070:

Test 02> sudo su $HDFS_USER touch /tmp/hello.txt

Now, hello.txt file will be created in tmp location:

/u/HBase B/hadoop-2.2.0/bin/hadoop dfs  -mkdir -p /app
/u/HBase B/hadoop-2.2.0/bin/hadoop dfs  -mkdir -p /app/apphduser

This will create a specific directory for this application user in the HDFS FileSystem location(/app/apphduser)
/u/HBase B/hadoop-2.2.0/bin/hadoop dfs -copyFromLocal /tmp/hello.txt /app/apphduser 
/u/HBase B/hadoop-2.2.0/bin/hadoop dfs –ls /app/apphduser

So that the data is separated based on the users, in a true production env many users will be using it.

You must see hello.txt once the command executes:

Test 03> Browse http://datanode.full.hostname:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&nnaddr=$datanode.full.hostname:8020

You should see the details on the DataNode. Once you hit the preceding URL you will get the following screenshot:

On the command line it will be as follows:

Validate Yarn/MapReduce setup and execute this command from the resource manager:

<login as $YARN_USER> /u/HBase B/hadoop-2.2.0/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

Execute the following command from NodeManager:

<login as $YARN_USER >
/u/HBase B/hadoop-2.2.0/sbin /yarn-daemon.sh --config 
$HADOOP_CONF_DIR start nodemanager

Executing the following commands will create the directories in the hdfs and apply the respective access rights:

Cd u/HBase B/hadoop-2.2.0/bin
hadoop fs -mkdir /app-logs // creates the dir in HDFS
hadoop fs -chown $YARN_USER /app-logs //changes the ownership
hadoop fs -chmod 1777 /app-logs // explained in the note section
Execute MapReduce

Start jobhistory servers:

<login as $MAPRED_USER>
/u/HBase B/hadoop-2.2.0/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR

Let's have a few tests to be sure we have configured properly:

Test 01: From the browser or from curl use the link to browse: http://yourresourcemanager.full.hostname:8088/.

Test 02:

Sudo su $HDFS_USER
/u/HBase B/hadoop-2.2.0/bin/hadoop jar /u/HBase B/hadoop-2.2.0/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.2.1-alpha.jar teragen 100 /test/10gsort/input 
/u/HBase B/hadoop-2.2.0/bin/hadoop jar /u/HBase B/hadoop-2.2.0/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.2.1-alpha.jar 

Validate the HBase setup:

Login as $HDFS_USER
/u/HBase B/hadoop-2.2.0/bin/hadoop fs –mkdir -p /apps/HBase 
/u/HBase B/hadoop-2.2.0/bin/hadoop fs –chown app:app –R  /apps/HBase 

Now login as $HBase _USER:

/u/HBase B/HBase -0.98.3-hadoop2/bin/HBase -daemon.sh –-config $HBase _CONF_DIR start master

This command will start the master node. Now let's move to HBase Region server nodes:

/u/HBase B/HBase -0.98.3-hadoop2/bin/HBase -daemon.sh –-config $HBase _CONF_DIR start regionserver

This command will start the regionservers:

You can check the log files and check for any errors:

Now let's login using:

Sudo su- $HBase _USER
/u/HBase B/HBase -0.98.3-hadoop2/bin/HBase shell

We will connect HBase to the master.

Validate the ZooKeeper setup. If you want to use an external zookeeper, make sure there is no internal HBase based zookeeper running while working with the external zookeeper or existing zookeeper and is not managed by HBase :

For this you have to edit /opt/HBase B/HBase -0.98.5-hadoop2/conf/ HBase -env.sh.

Change the following statement (HBase _MANAGES_ZK=false):

# Tell HBase whether it should manage its own instance of Zookeeper or not.

export HBase _MANAGES_ZK=true.

Once this is done we can add zoo.cfg to HBase 's CLASSPATH.

HBase looks into zoo.cfg as a default lookup for configurations

dataDir=/opt/HBase B/zookeeper-3.4.6/zooData 

# this is the place where the zooData will be present

server.1=172.28.182.45:2888:3888

# IP and port for server 01

server.2=172.29.75.37:4888:5888

# IP and port for server 02

You can edit the log4j.properties file which is located at /opt/HBase B/zookeeper-3.4.6/conf and point the location where you want to keep the logs.

# Define some default values that can be overridden by system properties:

zookeeper.root.logger=INFO, CONSOLE
zookeeper.console.threshold=INFO
zookeeper.log.dir=.
zookeeper.log.file=zookeeper.log
zookeeper.log.threshold=DEBUG
zookeeper.tracelog.dir=. # you can specify the location here 
zookeeper.tracelog.file=zookeeper_trace.log

Once this is done you start zookeeper with the following command:

-bash-4.1$ sudo /u/HBase B/zookeeper-3.4.6/bin/zkServer.sh start
Starting zookeeper ... STARTED

You can also pipe the log to the ZooKeeper logs:

/u/logs//u/HBase B/zookeeper-3.4.6/zoo.out 2>&1

2 : refers to the second file descriptor for the process, that is stderr.

> : means re-direct
&1:  means the target of the redirection should be the same location as the first file descriptor i.e stdout

We dissect it into two parts, master and slave setup. We can divide it in the following parts:

Master-NameNode
Master-Secondary NameNode
Master-Jobtracker
Master-Yarn Resource Manager
Master-HBase Master 
Slave-DataNode
Slave-Map Reduce Tasktracker
Slave-Yarn Node Manager
Slave-HBase Region server

We will discuss this in more details in Chapter 3, Working with Large Distributed Systems Part 1.

Apache Yarn is a robust, distributed, application management framework that surpasses the traditional Apache Hadoop MapReduce framework to process data in a large Hadoop clusters.

This change was needed because during the map phase of the mapreduce process, the data is chunked into small discrete packets that can be processed, followed by a second phase reduce, which allows this split data to be aggregated and thus produces the desired results. This works well with small, mid-sized and to some extent large clusters, but for the very large cluster (more than 4000 nodes), the unpredictable behavior starts to surface. The core issue was replication of data during the cascading failure.

Thus, it helps us in reliability, scalability, and sharing. Hadoop Yarn essentially works with JobTracker and splits the multiple accountabilities into resource management, job monitoring and scheduling into more granular and distributed by resource manager and application Master.

It works in synchronicity with per-node NodeManager and the per-application ApplicationMaster.

NodeManager takes a remote invocation from resource manager and manage resources available on a single node.

ApplicationMaster is responsible for negotiating resource with the resourceManager and works with the NodeManager to start the containers.

HBase provides low-latency random read and writes on top of HDFS, being a large-scale key value store, the main differentiating factor for HBase is that it can scan petabyte of data at a very high speed. It also comes with an inbuilt capability of autosharding by splitting the tables dynamically when the table becomes too large.

This enables HBase to horizontally scale. This is quantified as regions. Regions are a portion of table data, which are stored together and of prime efficiency. This does not make sense. The slave servers in HBase are the region server. It does a fair bit of work and provides true distribution across different regions. It can serve one or more regions based on the needs, each reason is assigned to a region server or start-up.

HBase 0.96 removed the concept of ROOT containing the META table location, rather it moved it to ZooKeeper as the META table cannot split and can be in only single region:

Refer to the following chapter:

The following are the benefits of using HDFS:

A lot of work is continuously happening on the core implementations of HDFS; some are as follows:

Data in HDFS is not placed homogeneously in the distributed DataNodes. The most obvious reason is addition of new DataNodes is the preexisting cluster. Internally the system (NameNode) performs various checks before is starts sending the data/new blocks to the DataNode, which are listed as below:

You would need to set up Hadoop 2.2.0 in a fully distributed mode, as discussed in the previous section. Web interface is also used for browsing the file system.

To use the File system we go as per the following steps:

The HBase setup

Configuring HBase in a fully distributed environment:

• Prerequisites: The hadoop/hdfs cluster is healthy

• It has namenode,data node, secondary namenode setup done as discussed earlier

• Passwordless access is there between the namenode, datanode, secondary namenocde

• The directory structure is having appropriate access levels

• Hope paths are set as described earlier

Just for recap you can run this command, and it must show the following details:

Tip

Please check the compatibility of Hadoop and HBase .

In this book, we used hadoop-2.2.0 and HBase 0.98.5-hadoop2.

1. Let's go to the NameNode of Hadoop/HDFS by typing this command:

   Vi /u/HBase B/hadoop-2.2.0/etc/hadoop/ hdfs-site.xml
   

   The setup should be like this:

   

   These are the data nodes that we will use for regional servers later on. We will use NameNode as an HBase master node.

   vi  /u/HBase B/hadoop-2.2.0/etc/hadoop/slave
   
     it should have the nodes which will be used as a data node
     your-datanode01
     your-datenode02

The following steps will help you to implement the same:

1. Copy the hdfs-stie.xml which is in Hadoop setup to:

   cd $HBase _HOME/conf
   

2. Also, copy it to all the Region servers. Edit the regionserver file by:

   Vi $HBase _HOME/conf/ regionservers on the HMASTER server
   

3. Place the IP or the fully qualified name of the region servers.

   Vi HBase -env.sh and change the export HBase _MANAGES_ZK=true
   

4. This will allow HBase to manage the zookeeper internally on port 2181.

  cd $HBase _HOME/bin start-HBase .sh 

cd $HBase _LOGS/
ls -ltr
-rw-rw-r--. 1 hadoop hadoop      0 Aug 29 19:22 SecurityAuth.audit
-rw-rw-r--. 1 hadoop hadoop  92590 Aug 30 15:04 HBase -hadoop-zookeeper-your-HBase -master.log
-rw-rw-r--. 1 hadoop hadoop 484092 Aug 30 16:31 HBase -hadoop-master-rchoudhry-your-HBase -master.log

tail -200 HBase -hadoop-zookeeper-your-HBase -master.log

tail -200 hadoop hadoop 484092 Aug 30 16:31 HBase -hadoop-master-rchoudhry-your-HBase -master.log

[hadoop@rchoudhry-linux64 logs]$ jps
960 SecondaryNameNode  // secondary name node is up
8467 NameNode // Name node is up
11892 HQuorumPeer // zookeeper is running in Quorum mode
25318 Jps // pls neglect this 
12008 HMaster // HBase Master is running successfully
8699 ResourceManager // Resource manager is running 
12171 HRegionServer  // HBase Region server is running
8974 JobHistoryServer // JobHistory Server is running

13026 NodeManager
12425 Jps
12778 DataNode
13567 HRegionServer

cd $HBase _HOME/bin
[hadoop@rchoudhry-linux64 bin]$ HBase shell -d
HBase (main):001:0>

HBase (main):001:0> list
City_Table                                                        
MyClickStream                                                      
t1                                                                 
3 row(s) in 1.1270 seconds

["City_Table", "MyClickStream", "t1"]
HBase (main):002:0>  statusHBase (main):002:0> status 'simple'HBase (main):002:0> status 'summary'HBase (main):002:0> status 'detailed'
HBase (main):002:0> describe 'MyClickStream'
HBase (main):002:0> scan 'yourtablename'
HBase (main):002:0> create 'yourtablename','cf01',cf'02'

hdfs dfs -mkdir /snapshot 
using this command we can make it enable for snapshots.
hdfs dfsadmin -allowSnapshot /snapshot
hdfs dfs -createSnapshot /snapshot [<snapshotName>]

  hdfs dfs -deleteSnapshot <path> <snapshotName>

<property>
    <name>HBase .snapshot.enabled</name>
    <value>true</value>
</property>

./bin/HBase shell
HBase > create 'MyClickStream' ,'cf01', 'cf2'

cf01-> is represented as a column family  01
cf02-> is represented as a column family 02 

./bin/HBase shell –d 
HBase > disable 'MyClickStream'
HBase > snapshot 'MyClickStream' ,'MyClickStreamSnapshot-08302014'

./bin/HBase shell 
HBase > list_snapshots 

To better understand the concepts, I have broken down the parameter into:

DFS Administration commands:

Let's discuss other important components:

The entire process helps us set up the Hadoop/HDFS file system, and later on HBase can sit on get the benefits of the HDFS distributed architecture.

Refer to the following chapter:

We must have a HDFS/Hadoop setup in a cluster or fully distributed mode as the first step, which we did it in the earlier section. The second step is to have an HBase setup in a fully distributed mode.

HBase provides multiple ways to do administration work on the fully distributed clustered environment:

The Master run on 60010 ports as default, the web interface is looked up on this port only. This needs to be started on the HMaster node.

The master UI gives a dashboard of what's going on in the HBase cluster.

The UI contains the following details:

We will go through it in the following sections.

A Detailed discussion is out of scope at this point. For more details, see later sections. For our purpose, we will avoid to go for backup master:

Let's list all the user-created tables using Tables. It provides the details about User Tables (tables that are created by the users/actors), Catalog Tables (this contains HBase :meta and HBase :namespaces), which is seen in the following figure:

Clicking any table, as listed earlier, other important details are shown such as table attributes, table regions, and region-by-region server details. Actions as compaction and split can be taken using admin UI.

Task provides the details of talk, which is happening. We can see Monitored task, RPC Tasks, RPC Handler task, Active RPC Calls, Client Operations, and a JSON response:

The following Software Attributes page provides the details of the software used in the HBase cluster:

After clicking on zk_dump, it provides further details about the zookeeper Quorum stats as follows:

HBase provides various command-line tools for administrating, debugging, and doing an analysis on the HBase cluster.

The first tool is the HBase shell, and the details are as follows:

bin/HBase 
Usage: HBase [<options>] <command> [<args>] 

Identify inconsistencies with hbck; this tool provides consistencies, checks for corruption of data, and it runs against the cluster.

bin/HBase  hbck  

This runs against the cluster and provides the details of inconsistencies between the regions and masters.

bin/HBase  hbck –details  

The -details provides the insight of splits which happens in all the tables.

  bin/HBase hbck MyClickStream 

The preceding line enables us to vView the HFile content in a text format.

   bin/HBase org.apache.hadoop.HBase .io.hfile.HFile

Use FSFLogs for manual splitting and dumping:

HBase org.apache.hadoop.HBase .regionserver.wal.FSHLog --dump 
HBase org.apache.hadoop.HBase .regionserver.wal.FSHLog –split

Enable compressor tool:

HBase org.apache.hadoop.HBase .util.CompressionTest hdfs://host/path/to/HBase snappy

Enable compressor tool on the Column Family or while creating a tables:

HBase > disable 'MyClickStream'
HBase > alter 'MyClickStream', {NAME => 'cf', COMPRESSION => 'GZ'} 
HBase > enable 'MyClickStream'
HBase > create MyClickStream', {NAME =>'cf2', COMPRESSION => 'SNAPPY'}

Load Test too Usage below are some of the commands which can be used to do a quick load test on your compression performance:

bin/HBase org.apache.hadoop.HBase .util.LoadTestTool -h

usage: bin/HBase org.apache.hadoop.HBase .util.LoadTestTool <options>.

Options: includes –batchupdate

-compression: <arg> Compression type , arguments can be LZq,GZ,NONE and SNAPPY

We will limit ourselves with the above commands.

A good example will be as follows:

HBase org.apache.hadoop.HBase .util.LoadTestTool -write 2:20:20 -num_keys 500000 -read 60:30 -num_tables 1 -data_block_encoding NONE -tn load_test_tool_NONE
-write (here we are passing 2 as ->avg_cols_per_key)
20 is the ->avg_data_size>:
20 is number of parallel threads to be used.
-num_key has an integer arguments as 500000, this is the number of keys to read and write.

Now let's look at a read:

-read 60 is the verify percent

30 is the number of threads

-num_tables a positve interger is passed which is the number of tables to be loaded in parallel
-data_block_encoding there are various encoding algorithms which can be passed as an argument, this allow the data block to be encoded based on the need. Some of them  are [NONE,PREFIX,DIFF,FAST_DIFF,PREFIX_TREE].

-tn is a table name prefix which exports the content and data to HDFS in a sequence file using this:

HBase org.apache.hadoop.HBase .mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]] 

Importing: This tool will load data that has been exported back into HBase :

This can be done by the following command:

HBase org.apache.hadoop.HBase .mapreduce.Import <tablename> <inputdir> 

-tablename: Is the name of the table to be imported by the tool.

-inputdir: Is the input dir which will be used.

Utility to Replay WAL files into HBase :

HBase org.apache.hadoop.HBase .mapreduce.WALPlayer [options] <wal inputdir> <tables> [<tableMappings>]> 
HBase org.apache.hadoop.HBase .mapreduce.WALPlayer /backuplogdir MyClickStream newMyClickStream
walinputdi:  /backuplogdir
tables:  MyClickStream
tableMappings→newMyClickStream
 newMyClickStream

HBase clean: is dangerous and should be avoided in production setup.

HBase clean

Options: as parameters

  --cleanZk   cleans HBase related data from zookeeper.
  --cleanHdfs cleans HBase related data from hdfs.
  --cleanAll  cleans HBase related data from both zookeeper and hdfs.

HBase pe: This is a shortcut to run the performance evaluations tools

HBase ltt: This command is a shortcut provided to run the rg.apache.hadoop.HBase .util.LoadTestTool utility. It was introduced in 0.98 version.

View the details of the table as shown here:

Go to the log tab on the HBase Admin UI home page, and you will see the following details. Alternatively, you can log in to the directory using the Linux shell to tail the logs.

Directory: /logs/:

SecurityAuth.audit 0 bytes  Aug 29, 2014 7:22:00 PM
HBase -hadoop-master-rchoudhry-linux64.com.log 691391 bytes   Sep 2, 2014 11:01:34 AM
HBase -hadoop-master-rchoudhry-linux64.com.out 419 bytes 	Aug 29, 2014 7:31:21 PM
HBase -hadoop-regionserver-rchoudhry-linux64.com.log 1048281 bytes   Sep 2, 2014 11:01:23 AM
HBase -hadoop-regionserver-rchoudhry-linux64com.out 419 bytes   Aug 29, 2014 7:31:23 PM
HBase -hadoop-zookeeper-rchoudhry-linux64.log 149832 bytes   Aug 31, 2014 12:51:42 AM
HBase -hadoop-zookeeper-rchoudhry-linux64.com.out 419 bytes   Aug 29, 2014 7:31:19 PM
HBase -hadoop-zookeeper-rchoudhry-linux64.com.out.1 1146 bytes   Aug 29, 2014 7:26:29 PM

Get and set the log levels as required at runtime:

Master status for HBase -hadoop-master-rchoudhry-linux64.com,60000,1409365881345 as of Tue Sep 02 11:17:33 PDT 2014
Version Info:
===========================================================
HBase 0.98.5-hadoop2
Subversion file:///var/tmp/0.98.5RC0/HBase -0.98.5 -r Unknown
Compiled by apurtell on Mon Aug  4 23:58:06 PDT 2014
Hadoop 2.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z


Tasks:
===========================================================
Task: RpcServer.reader=1,port=60000
Status: WAITING:Waiting for a call
Running for 315970s

Task: RpcServer.reader=2,port=60000
Status: WAITING:Waiting for a call
Running for 315969s

When the browser points to the http address http://your-HBase-master:60010/master-status, the web interface of HBase admin is loaded.

Internally, it connects to the Zookeeper and can collect the details from the zookeeper interface, where in zookeeper tracks the Region server as per the region server configuration in the region server file. These are the values set in the HBase -site.xml. The data for hadoo/hdfs, region servers, zookeeper quorum details are continuously looked by the RPC calls, which the master makes via zookeeper.

In the above HBase -site.xml, the user/ sets the Master, Backup master, various other software attributes, the refresh time, memory allocations, storefiles, compactions,request, zk dumps etc

Node or Cluster view: In this, the user chooses either monitoring Hmaster or Region server data view. The HMaster view contains data and graphics about the node status. The Region server view is the main and the most important one because it allows monitoring of all the region server aspects.

You can point the http address to http://your-HBase-master:60030/rs-status. It loads the admin UI for the region servers.

The matrix that is captured here is as follows:

Backup master is a design/architecture choice, which needs careful considerations before enabling it. HBase by design is a fault-tolerant distributed system with assumes hardware failure in the network topology.

However, HBase does provide various options to for it such as:

Refer to the following chapter:

It is important to know some of the details of Ganglia and its distributed components before we get into the details of managing clusters

We will divide our how to do it into two sections. In the first section, we will talk about installing Ganglia on all the nodes.

Once it's done, we will do the integration with HBase so that the relevant metrics are available.

wget –o http://downloads.sourceforge.net/project/ganglia/\ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz

As the system grows from a few nodes to the tens or hundreds or becomes a very large cluster having more than hundreds of nodes it's pivotal to have a holistic view, drill down view, historical view of the logs at any given point of time in a graphical representation. In a large or very large installation, administrators are more concerned about redundancy, which avoids single point of failure. HBase and underlying HDFS are designed to handle the node failures gracefully, but it's equally important to monitor these failure as this can lead to pull down the cluster if a corrective action is not taken in time. HBase exposes various matrix to JMX and Ganglia like HMaster, region servers statistics, JMV (Java virtual machines), RPC (Remote procedure calls), Hadoop/HDFS, MapReduce details. Taking into consideration all these points and various other salient and powerful features, we considered Ganglia.

Ganglia provides the following advantages:

HBase works with only 3.0.X and higher version of Ganglia, hence we used 3.0.7 version.

In step 4, we installed the dependencies of libraries, which will be required for the ganglia to compile.

In step 5, we compiled ganglia and installed it by running the configure command, then we used make and then make install command.

In step 6, we created a file gmond.conf, and later on in step 7, we changed the setting to point to HBase master node. We also configured the port to 8649 with a user ganglia who can read from the cluster. By commenting the multicast address and the TTL (time to live), we also changed the UDP-based multicasting to which is a default one to unicasting, which enables us to expand the cluster to above 120 nodes. We also added a master Gmond node in this config file.

In step 8 we started the gmond and got some core monitoring such as CPU, disk, network, memory, and load average of the nodes.

In step 9, we went back to the /u/HBase B/ganglia-3.0.7/ and reran the configuration, but this time, we added configure –with-gmetad, so that it complies with gmetad.

In step 11, we copied the gmetad.conf from.

sudo /u/HBase B/gangli-3.0.7/gmetad/gmetad.conf to /etc/gmetad.conf.

In step 12, we added ganglia user and Master details in the data_source HBase B HBase bMaster.ganglia-monitor.com.

In step 13/14, we create the rrds directory that will hold the data in round-robin databases; later on, we stated the gmetad daemon on the master nodes.

In step 15, we installed all the dependency, which is required to run the web interface.

In step 16, we copied the web .php file from the existing location.

In step 17, we restarted the apache instance and saw all the basic graphs, which provides the details of the nodes and the host but not HBase details. We also copied it to all the nodes so that we have a similar configuration and the Ganglia master is getting the data from the child nodes.

In step 18, we changed the setting in hadoop-metrics2-HBase .properties so that it starts collecting the metrics and starts sending it to the ganglia servers on port 8649. The main class that is responsible for providing these details is org.apache.hadoop.metrics2.sink.FileSink and it properties.

Now we point at the URL of master, and once the page is rendered, it starts showing the graphs as described by the image HBase -Ganglia-MasterAndRegion01-01.png. It starts showing the following graphs:

Ganglia is used for monitoring very large cluster, and in the word of Hadoop/HBase , it can be very useful as it provides the following:

You could refer to the following sites for more information: