Hadoop 2.x Administration Cookbook

Over 100 practical recipes to help you become an expert Hadoop administrator
Preview in Mapt

Hadoop 2.x Administration Cookbook

Gurmukh Singh

Over 100 practical recipes to help you become an expert Hadoop administrator
Mapt Subscription
FREE
$29.99/m after trial
eBook
$28.00
RRP $39.99
Save 29%
Print + eBook
$49.99
RRP $49.99
What do I get with a Mapt Pro subscription?
  • Unlimited access to all Packt’s 5,000+ eBooks and Videos
  • Early Access content, Progress Tracking, and Assessments
  • 1 Free eBook or Video to download and keep every month after trial
What do I get with an eBook?
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with Print & eBook?
  • Get a paperback copy of the book delivered to you
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with a Video?
  • Download this Video course in MP4 format
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
$0.00
$28.00
$49.99
$29.99p/m after trial
RRP $39.99
RRP $49.99
Subscription
eBook
Print + eBook
Start 30 Day Trial

Frequently bought together


Hadoop 2.x Administration Cookbook Book Cover
Hadoop 2.x Administration Cookbook
$ 39.99
$ 28.00
Getting Started with Hadoop 2.x [Video] Book Cover
Getting Started with Hadoop 2.x [Video]
$ 124.99
$ 106.25
Buy 2 for $35.00
Save $129.98
Add to Cart
Subscribe and access every Packt eBook & Video.
 
  • 5,000+ eBooks & Videos
  • 50+ New titles a month
  • 1 Free eBook/Video to keep every month
Start Free Trial
 

Book Details

ISBN 139781787126732
Paperback348 pages

Book Description

Hadoop enables the distributed storage and processing of large datasets across clusters of computers. Learning how to administer Hadoop is crucial to exploit its unique features. With this book, you will be able to overcome common problems encountered in Hadoop administration.

The book begins with laying the foundation by showing you the steps needed to set up a Hadoop cluster and its various nodes. You will get a better understanding of how to maintain Hadoop cluster, especially on the HDFS layer and using YARN and MapReduce. Further on, you will explore durability and high availability of a Hadoop cluster.

You’ll get a better understanding of the schedulers in Hadoop and how to configure and use them for your tasks. You will also get hands-on experience with the backup and recovery options and the performance tuning aspects of Hadoop. Finally, you will get a better understanding of troubleshooting, diagnostics, and best practices in Hadoop administration.

By the end of this book, you will have a proper understanding of working with Hadoop clusters and will also be able to secure, encrypt it, and configure auditing for your Hadoop clusters.

Table of Contents

Chapter 1: Hadoop Architecture and Deployment
Introduction
Building and compiling Hadoop
Installation methods
Setting up host resolution
Installing a single-node cluster - HDFS components
Installing a single-node cluster - YARN components
Installing a multi-node cluster
Configuring the Hadoop Gateway node
Decommissioning nodes
Adding nodes to the cluster
Chapter 2: Maintaining Hadoop Cluster HDFS
Introduction
Configuring HDFS block size
Setting up Namenode metadata location
Loading data in HDFS
Configuring HDFS replication
HDFS balancer
Quota configuration
HDFS health and FSCK
Configuring rack awareness
Recycle or trash bin configuration
Distcp usage
Control block report storm
Configuring Datanode heartbeat
Chapter 3: Maintaining Hadoop Cluster – YARN and MapReduce
Introduction
Running a simple MapReduce program
Hadoop streaming
Configuring YARN history server
Job history web interface and metrics
Configuring ResourceManager components
YARN containers and resource allocations
ResourceManager Web UI and JMX metrics
Preserving ResourceManager states
Chapter 4: High Availability
Introduction
Namenode HA using shared storage
ZooKeeper configuration
Namenode HA using Journal node
Resourcemanager HA using ZooKeeper
Rolling upgrade with HA
Configure shared cache manager
Configure HDFS cache
HDFS snapshots
Configuring storage based policies
Configuring HA for Edge nodes
Chapter 5: Schedulers
Introduction
Configuring users and groups
Fair Scheduler configuration
Fair Scheduler pools
Configuring job queues
Job queue ACLs
Configuring Capacity Scheduler
Queuing mappings in Capacity Scheduler
YARN and Mapred commands
YARN label-based scheduling
YARN SLS
Chapter 6: Backup and Recovery
Introduction
Initiating Namenode saveNamespace
Using HDFS Image Viewer
Fetching parameters which are in-effect
Configuring HDFS and YARN logs
Backing up and recovering Namenode
Configuring Secondary Namenode
Promoting Secondary Namenode to Primary
Namenode recovery
Namenode roll edits – online mode
Namenode roll edits – offline mode
Datanode recovery – disk full
Configuring NFS gateway to serve HDFS
Recovering deleted files
Chapter 7: Data Ingestion and Workflow
Introduction
Hive server modes and setup
Using MySQL for Hive metastore
Operating Hive with ZooKeeper
Loading data into Hive
Partitioning and Bucketing in Hive
Hive metastore database
Designing Hive with credential store
Configuring Flume
Configure Oozie and workflows
Chapter 8: Performance Tuning
Tuning the operating system
Tuning the disk
Tuning the network
Tuning HDFS
Tuning Namenode
Tuning Datanode
Configuring YARN for performance
Configuring MapReduce for performance
Hive performance tuning
Benchmarking Hadoop cluster
Chapter 9: HBase Administration
Introduction
Setting up single node HBase cluster
Setting up multi-node HBase cluster
Inserting data into HBase
Integration with Hive
HBase administration commands
HBase backup and restore
Tuning HBase
HBase upgrade
Migrating data from MySQL to HBase using Sqoop
Chapter 10: Cluster Planning
Introduction
Disk space calculations
Nodes needed in the cluster
Memory requirements
Sizing the cluster as per SLA
Network design
Estimating the cost of the Hadoop cluster
Hardware and software options
Chapter 11: Troubleshooting, Diagnostics, and Best Practices
Introduction
Namenode troubleshooting
Datanode troubleshooting
Resourcemanager troubleshooting
Diagnose communication issues
Parse logs for errors
Hive troubleshooting
HBase troubleshooting
Hadoop best practices
Chapter 12: Security
Introduction
Encrypting disk using LUKS
Configuring Hadoop users
HDFS encryption at Rest
Configuring SSL in Hadoop
In-transit encryption
Enabling service level authorization
Securing ZooKeeper
Configuring auditing
Configuring Kerberos server
Configuring and enabling Kerberos for Hadoop

What You Will Learn

  • Set up the Hadoop architecture to run a Hadoop cluster smoothly
  • Maintain a Hadoop cluster on HDFS, YARN, and MapReduce
  • Understand High Availability with Zookeeper and Journal Node
  • Configure Flume for data ingestion and Oozie to run various workflows
  • Tune the Hadoop cluster for optimal performance
  • Schedule jobs on a Hadoop cluster using the Fair and Capacity scheduler
  • Secure your cluster and troubleshoot it for various common pain points

Authors

Table of Contents

Chapter 1: Hadoop Architecture and Deployment
Introduction
Building and compiling Hadoop
Installation methods
Setting up host resolution
Installing a single-node cluster - HDFS components
Installing a single-node cluster - YARN components
Installing a multi-node cluster
Configuring the Hadoop Gateway node
Decommissioning nodes
Adding nodes to the cluster
Chapter 2: Maintaining Hadoop Cluster HDFS
Introduction
Configuring HDFS block size
Setting up Namenode metadata location
Loading data in HDFS
Configuring HDFS replication
HDFS balancer
Quota configuration
HDFS health and FSCK
Configuring rack awareness
Recycle or trash bin configuration
Distcp usage
Control block report storm
Configuring Datanode heartbeat
Chapter 3: Maintaining Hadoop Cluster – YARN and MapReduce
Introduction
Running a simple MapReduce program
Hadoop streaming
Configuring YARN history server
Job history web interface and metrics
Configuring ResourceManager components
YARN containers and resource allocations
ResourceManager Web UI and JMX metrics
Preserving ResourceManager states
Chapter 4: High Availability
Introduction
Namenode HA using shared storage
ZooKeeper configuration
Namenode HA using Journal node
Resourcemanager HA using ZooKeeper
Rolling upgrade with HA
Configure shared cache manager
Configure HDFS cache
HDFS snapshots
Configuring storage based policies
Configuring HA for Edge nodes
Chapter 5: Schedulers
Introduction
Configuring users and groups
Fair Scheduler configuration
Fair Scheduler pools
Configuring job queues
Job queue ACLs
Configuring Capacity Scheduler
Queuing mappings in Capacity Scheduler
YARN and Mapred commands
YARN label-based scheduling
YARN SLS
Chapter 6: Backup and Recovery
Introduction
Initiating Namenode saveNamespace
Using HDFS Image Viewer
Fetching parameters which are in-effect
Configuring HDFS and YARN logs
Backing up and recovering Namenode
Configuring Secondary Namenode
Promoting Secondary Namenode to Primary
Namenode recovery
Namenode roll edits – online mode
Namenode roll edits – offline mode
Datanode recovery – disk full
Configuring NFS gateway to serve HDFS
Recovering deleted files
Chapter 7: Data Ingestion and Workflow
Introduction
Hive server modes and setup
Using MySQL for Hive metastore
Operating Hive with ZooKeeper
Loading data into Hive
Partitioning and Bucketing in Hive
Hive metastore database
Designing Hive with credential store
Configuring Flume
Configure Oozie and workflows
Chapter 8: Performance Tuning
Tuning the operating system
Tuning the disk
Tuning the network
Tuning HDFS
Tuning Namenode
Tuning Datanode
Configuring YARN for performance
Configuring MapReduce for performance
Hive performance tuning
Benchmarking Hadoop cluster
Chapter 9: HBase Administration
Introduction
Setting up single node HBase cluster
Setting up multi-node HBase cluster
Inserting data into HBase
Integration with Hive
HBase administration commands
HBase backup and restore
Tuning HBase
HBase upgrade
Migrating data from MySQL to HBase using Sqoop
Chapter 10: Cluster Planning
Introduction
Disk space calculations
Nodes needed in the cluster
Memory requirements
Sizing the cluster as per SLA
Network design
Estimating the cost of the Hadoop cluster
Hardware and software options
Chapter 11: Troubleshooting, Diagnostics, and Best Practices
Introduction
Namenode troubleshooting
Datanode troubleshooting
Resourcemanager troubleshooting
Diagnose communication issues
Parse logs for errors
Hive troubleshooting
HBase troubleshooting
Hadoop best practices
Chapter 12: Security
Introduction
Encrypting disk using LUKS
Configuring Hadoop users
HDFS encryption at Rest
Configuring SSL in Hadoop
In-transit encryption
Enabling service level authorization
Securing ZooKeeper
Configuring auditing
Configuring Kerberos server
Configuring and enabling Kerberos for Hadoop

Book Details

ISBN 139781787126732
Paperback348 pages
Read More

Read More Reviews

Recommended for You

Learning Hadoop 2 Book Cover
Learning Hadoop 2
$ 29.99
$ 21.00
Hadoop MapReduce Cookbook Book Cover
Hadoop MapReduce Cookbook
$ 29.99
$ 21.00
Mastering Hadoop Book Cover
Mastering Hadoop
$ 29.99
$ 6.00
Web Security: Common Vulnerabilities And Their Mitigation [Video] Book Cover
Web Security: Common Vulnerabilities And Their Mitigation [Video]
$ 48.99
$ 41.65
Advanced Continuous Delivery Pipeline [Video] Book Cover
Advanced Continuous Delivery Pipeline [Video]
$ 124.99
$ 106.25
The Complete Google Go Programming Course For Beginners [Video] Book Cover
The Complete Google Go Programming Course For Beginners [Video]
$ 123.99
$ 105.40