Preface

This book is a quick-start guide for learning Apache Hadoop version 3. It is targeted at readers with no prior knowledge of Apache Hadoop, and covers key big data concepts, such as data manipulation using MapReduce, flexible model utilization with YARN, and storing different datasets with Hadoop Distributed File System (HDFS). This book will teach you about different configurations of Hadoop version 3 clusters, from a lightweight developer edition to an enterprise-ready deployment. Throughout your journey, this guide will demonstrate how parallel programming paradigms such as MapReduce can be used to solve many complex data processing problems, using case studies and code to do so. Along with development, the book will also cover the important aspects of the big data software development life cycle, such as quality assurance and control, performance, administration, and monitoring. This book serves as a starting point for those who wish to master the Apache Hadoop ecosystem.

Who this book is for

Hadoop 3 Quick Start Guide is intended for those who wish to learn about Apache Hadoop version 3 in the quickest manner, including the most important areas of it, such as MapReduce, YARN, and HDFS. This book serves as a starting point for programmers who are looking to analyze datasets of any kind with the help of big data, quality teams who are interested in evaluating MapReduce programs with respect to their functionality and performance, administrators who are setting up enterprise-ready Hadoop clusters with horizontal scaling, and individuals who wish to enhance their expertise on Apache Hadoop version 3 to solve complex problems.

What this book covers

Chapter 1, Hadoop 3.0 – Background and Introduction, gives you an overview of big data and Apache Hadoop. You will go through the history of Apache Hadoop's evolution, learn about what Hadoop offers today, and explore how it works. Also, you'll learn about the architecture of Apache Hadoop, as well as its new features and releases. Finally, you'll cover the commercial implementations of Hadoop.

Chapter 2, Planning and Setting Up Hadoop Clusters, covers the installation and setup of Apache Hadoop. We will start with learning about the prerequisites for setting up a Hadoop cluster. You will go through the different Hadoop configurations available for users, covering development mode, pseudo-distributed single nodes, and cluster setup. You'll learn how each of these configurations can be set up, and also run an example application of the configuration. Toward the end of the chapter, we will cover how you can diagnose Hadoop clusters by understanding log files and the different debugging tools available.

Chapter 3, Deep Diving into the Hadoop Distributed File System, goes into how HDFS works and its key features. We will look at the different data flowing patterns of HDFS, examining HDFS in different roles. Also, we'll take a look at various command-line interface commands for HDFS and the Hadoop shell. Finally, we'll look at the data structures that are used by HDFS with some examples.

Chapter 4, Developing MapReduce Applications, looks in depth at various topics pertaining to MapReduce. We will start by understanding the concept of MapReduce. We will take a look at the Hadoop application URL ports. Also, we'll study the different data formats needed for MapReduce. Then, we'll take a look at job compilation, remote job runs, and using utilities such as Tool. Finally, we'll learn about unit testing and failure handling.

Chapter 5, Building Rich YARN Applications, teaches you about the YARN architecture and the key features of YARN, such as resource models, federation, and RESTful APIs. Then, you'll configure a YARN environment in a Hadoop distributed cluster. Also, you'll study some of the additional properties of yarn-site.xml. You'll learn about the YARN distributed command-line interface. After this, we will delve into building YARN applications and monitoring them.

Chapter 6, Monitoring and Administration of a Hadoop Cluster, explores the different activities performed by Hadoop administrators for the monitoring and optimization of a Hadoop cluster. You'll learn about the roles and responsibilities of an administrator, followed by cluster planning. You'll dive deep into key management aspects of Hadoop clusters, such as resource management through job scheduling with algorithms such as Fair Scheduler and Capacity Scheduler. Also, you'll discover how to ensure high availability and security for an Apache Hadoop cluster.

Chapter 7, Demystifying Hadoop Ecosystem Components, covers the different components that constitute Hadoop's overall ecosystem offerings to solve complex industrial problems. We will take a brief overview of the tools and software that run on Hadoop. Also, we'll take a look at some components, such as Apache Kafka, Apache PIG, Apache Sqoop, and Apache Flume. After that, we'll cover the SQL and NoSQL Hadoop-based databases: Hive and HBase, respectively.

Chapter 8, Advanced Topics in Apache Hadoop, gets into advanced topics, such as the use of Hadoop for analytics using Apache Spark and processing streaming data using an Apache Storm pipeline. It will provide an overview of real-world use cases for different industries, with some sample code for you to try out independently.

To get the most out of this book

You won't need too much hardware to set up Hadoop. The minimum setup is a single machine / virtual machine, and the recommended setup is three machines.

It is better to have some hands-on experience of writing and running basic programs in Java, as well as some experience of using developer tools such as Eclipse.

Some understanding of the standard software development life cycle would be a plus.

As this is a quick-start guide, it does not provide complete coverage of all topics. Therefore, you will find links provided throughout the book o take you to the deep-dive of the given topic.

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

1. Log in or register at www.packt.com.
2. Select the SUPPORT tab.
3. Click on Code Downloads & Errata.
4. Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

• WinRAR/7-Zip for Windows
• Zipeg/iZip/UnRarX for Mac
• 7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Apache-Hadoop-3-Quick-Start-Guide. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Code in action

Visit the following link to check out videos of the code being run:

http://bit.ly/2AznxS3

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "You will need the hadoop-client-<version>.jar file to be added".

A block of code is set as follows:

    <dependencies>
         <dependency>
             <groupId>org.apache.hadoop</groupId>
             <artifactId>hadoop-client</artifactId>
             <version>3.1.0</version>
         </dependency>
     </dependencies>

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

<!-- Put site-specific property overrides in this file. --><configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://<master-host>:9000</value>
        </property>
</configuration>

Any command-line input or output is written as follows:

hrishikesh@base0:/$ df -m

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Right-click on the project and run Maven install, as shown in the following screenshot".

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [e-mail].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.