Getting ready on Amazon EC2

Amazon Elastic Compute Cloud (EC2) is a web service that provides resizable computer capacity in the cloud. By using Amazon EC2, we can practice HBase on a fully distributed mode easily, at low cost. All the servers that we will use to demonstrate HBase in this book are running on Amazon EC2.

This recipe describes the setup of the Amazon EC2 environment, as a preparation for the installation of HBase on it. We will set up a name server and client on Amazon EC2. You can also use other hosting services such as Rackspace, or real servers to set up your HBase cluster.

Getting ready

You will need to sign up, or create an Amazon Web Service (AWS)  account at http://aws.amazon.com/.

We will use EC2 command-line tools to manage our instances. You can download and set up the tools by following the instructions available at the following page:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?SettingUp_CommandLine.html.

You need a public/private key to log in to your EC2 instances. You can generate your key pairs and upload your public key to EC2, using these instructions:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/generating-a-keypair.html.

Before you can log in to an instance, you must authorize access. The following link contains instructions for adding rules to the default security group:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/adding-security-group-rules.html.

After all these steps are done, review the following checklist to make sure everything is ready:

• X.509 certificates: Check if the X.509 certificates are uploaded. You can check this at your account's Security Credentials page.

• EC2 key pairs: Check if EC2 key pairs are uploaded. You can check this at AWS Management Console | Amazon EC2 | NETWORK & SECURITY | Key Pairs.

• Access: Check if the access has been authorized. This can be checked at AWS Management Console | Amazon EC2 | NETWORK & SECURITY | Security Groups | Inbound.

• Environment variable settings: Check if the environment variable settings are done. As an example, the following snippet shows my settings; make sure you are using the right EC2_URL for your region:

$ cat ~/.bashrc
export EC2_HOME=~/opt/ec2-api-tools-1.4.4.2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=~/.ec2/pk-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export EC2_CERT=~/.ec2/cert-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export JAVA_HOME=/Library/Java/Home
export EC2_URL=https://ec2.us-west-1.amazonaws.com

We need to import our EC2 key pairs to manage EC2 instances via EC2 command-line tools:

$ ec2-import-keypair your-key-pair-name --public-key-file ~/.ssh/id_rsa.pub

Verify the settings by typing the following command:

$ ec2-describe-instances

If everything has been set up properly, the command will show your instances similarly to how you had configured them in the previous command.

Note

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

The last preparation is to find a suitable AMI. An AMI is a preconfigured operating system and software, which is used to create a virtual machine within EC2. We can find a registered Debian AMI at http://wiki.debian.org/Cloud/AmazonEC2Image.

For the purpose of practicing HBase, a 32-bit, EBS-backed AMI is the most cost effective AMI to use. Make sure you are choosing AMIs for your region. As we are using US-West (us-west-1) for this book, the AMI ID for us is ami-77287b32. This is a 32-bit, small instance of EC2. A small instance is good for practicing HBase on EC2 because it's cheap. For production, we recommend you to use at least High-Memory Extra Large Instance with EBS, or a real server.

How to do it...

Follow these instructions to get your EC2 instances ready for HBase. We will start two EC2 instances; one is a DNS/NTP server, and the other one is the client:

1. 1. Start a micro instance for the mentioned server. We will use ns1.hbase-admin-cookbook.com (ns1) as its Fully Qualified Domain Name (FQDN) , in a later section of this book:

   $ ec2-run-instances ami-77287b32 -t t1.micro -k your-key-pair
   

2. 2. Start a small instance for the client. We will use client1.hbase-admin-cookbook.com (client1) as its FQDN, later in this book:

   $ ec2-run-instances ami-77287b32 -t m1.small -k your-key-pair
   

3. 3. Verify the startup from AWS Management Console, or by typing the following command:

   $ ec2-describe-instances
   

   • You should see two instances from the output of the command. From the output of the ec2-describe-instances command, or AWS Management Console, you can find the public DNS of the instances that have already started. The DNS shows a value such as ec2-xx-xx-xxx-xx.us-west-1.compute.amazonaws.com:

   

4. 4. Log in to the instances via SSH, using the following command:

   $ ssh [e-mail]
   

5. 5. Update the package index files before we install packages on the server, by using the following command:

   root# apt-get update
   

6. 6. Change your instances' time zone to your local timezone, using the following command:

   root# dpkg-reconfigure tzdata
   

7. 7. Install the NTP server daemon on the DNS server, using the following command:

   root@ns# apt-get install ntp ntp-server ntpdate
   

8. 8. Install the NTP client on the client/server, using the following command:

   root@client1# apt-get install ntp ntpdate
   

9. 9. Configure /etc/ntp.conf on ns1 to run as an NTP server, and client1 to run as an NTP client, using ns1 as its server.

   Because there is no HBase-specific configuration for the NTP setup, we will skip the details. You can find the sample ntp.conf files for both the server and client, from the sample source of this book.

10. 10. Install BIND9 on ns1 to run as a DNS server, using the following command:

    root@ns# apt-get install bind9
    

    • You will need to configure BIND9 to run as a primary master server for internal lookup, and run as a caching server for external lookup. You also need to configure the DNS server, to allow other EC2 instances to update their record on the DNS server.

      We will skip the details as this is out of the scope of this book. For sample BIND9 configuration, please refer to the source, shipped with this book.

11. 11. For client1, just set it up using ns1 as its DNS server:

    root@client1# vi /etc/resolv.conf
    nameserver 10.160.49.250 #private IP of ns
    search hbase-admin-cookbook.com #domain name
    

12. 12. Update the DNS hostname automatically. Set up hostname to the EC2 instance's user data of the client. From the My Instances page of AWS Management Console, select client1 from the instances list, stop it, and then click Instance Actions | View | Change User Data; enter the hostname of the instance you want to use (here client1) in the pop-up page:

    

13. 13. Create a script to update the client's record on the DNS server, using user data:

    root@client1# vi ec2-hostname.sh
    #!/bin/bash
    #you will need to set up your DNS server to allow update from this key
    DNS_KEY=/root/etc/Kuser.hbase-admin-cookbook.com.+157+44141.private
    DOMAIN=hbase-admin-cookbook.com
    USER_DATA=`/usr/bin/curl -s http://169.254.169.254/latest/user-data`
    HOSTNAME=`echo $USER_DATA`
    #set also the hostname to the running instance
    hostname $HOSTNAME
    #we only need to update for local IP
    LOCIP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
    cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
    server ns.$DOMAIN
    zone $DOMAIN
    update delete $HOSTNAME.$DOMAIN A
    update add $HOSTNAME.$DOMAIN 60 A $LOCIP
    send
    EOF
    

14. 14. Finally, to run this at boot time from rc.local, add the following script to the rc.local file:

    root@client1# vi /etc/rc.local
    sh /root/bin/ec2-hostname.sh
    

How it works...

First we started two instances, a micro instance for DNS/NTP server, and a small one for client. To provide a name service to other instances, the DNS name server has to be kept running. Using micro instance can reduce your EC2 cost.

In step 3, we set up the NTP server and client. We will run our own NTP server on the same DNS server, and NTP clients on all other servers.

Note

Note: Make sure that the clocks on the HBase cluster members are in basic alignment.

EC2 instances can be started and stopped on demand; we don't need to pay for stopped instances. But, restarting an EC2 instance will change the IP address of the instance, which makes it difficult to run HBase. We can resolve this issue by running a DNS server to provide a name service to all EC2 instances in our HBase cluster. We can update name records on the DNS server every time other EC2 instances are restarted.

That's exactly what we have done in steps 4 and 5. Step 4 is a normal DNS setup. In step 5, we stored the instance name in its user data property at first, so that when the instance is restarted, we can get it back using EC2 API. Also, we will get the private IP address of the instance via EC2 API. With this data, we can then send a DNS update command to our DNS server every time the instance is restarted. As a result, we can always use its fixed hostname to access the instance.

We will keep only the DNS instance running constantly. You can stop all other instances whenever you do not need to run your HBase cluster.