Packt+ | Advance your knowledge in tech

All Products

Best Sellers

New Releases

Books

Videos

Audiobooks

Free Learning

Hadoop 2.x Administration Cookbook

You're reading from Hadoop 2.x Administration Cookbook

Product type Book

Published in May 2017

Publisher Packt

ISBN-13 9781787126732

Pages 348 pages

Edition 1st Edition

Languages

Concepts

System Administration

Author (1):

Aman Singh

Table of Contents (20) Chapters

Hadoop 2.x Administration Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Customer Feedback

Preface

Hadoop Architecture and Deployment

Maintaining Hadoop Cluster HDFS

Maintaining Hadoop Cluster – YARN and MapReduce

High Availability

Schedulers

Backup and Recovery

Data Ingestion and Workflow

Performance Tuning

HBase Administration

Cluster Planning

Troubleshooting, Diagnostics, and Best Practices

Security

Index

Network design

In this recipe, we will be looking at the network design for the Hadoop cluster and what things to consider for planning a Hadoop cluster.

Getting ready

Make sure that the user has a running cluster with HDFS and YARN and has at least two nodes in the cluster.

How to do it...

Connect to the master1.cyrus.com Namenode and switch to the user hadoop.
Execute the commands as follows to check for the link speed and other network option modes:
```
$ ethtool eth0
$ iftop
$ netstat -s
```
Always have a separate network for Hadoop traffic by using VLANs.
Ensure the DNS resolution works for both forward and reverse lookup.
Run a caching-only DNS within the Hadoop network, which caches records for faster resolution.
Consider NIC teaming or binding for better performance.
Use dedicated core switches and rack top switches.
Consider having static IPs per node in the cluster.
Disable IPv6 for all nodes and just use IPv4.
Increasing the size of the cluster will mean more connections and more data across nodes...

The rest of the chapter is locked

Register for a free Packt account to unlock a world of extra content!

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime}

Authors (1)

Aman Singh

Gurmukh Singh is a seasoned technology professional with 14+ years of industry experience in infrastructure design, distributed systems, performance optimization, and networks. He has worked in big data domain for the last 5 years and provides consultancy and training on various technologies. He has worked with companies such as HP, JP Morgan, and Yahoo. He has authored Monitoring Hadoop by Packt Publishing

See other products by Aman Singh

Other recommended products

Related to this chapter

Apache Hadoop 3 Quick Start Guide

Apache Hadoop 3 Quick Start Guide

Apache Hadoop is a widely used distributed data platform. It enables large datasets to be efficiently processed instead of using one large computer to store and process the data. This book will get you started with the Hadoop ecosystem, and introduce you to the main technical topics such as MapReduce, YARN and HDFS.

Oct 2018 7 hours 20 minutes

HBase High Performance Cookbook

HBase High Performance Cookbook

Jan 2017 11 hours 40 minutes

Mastering Hadoop 3

Mastering Hadoop 3

This is a comprehensive guide to understand advanced concepts of Hadoop ecosystem. You will learn how Hadoop works internally, and build solutions to some of real world use cases. Finally, you will have a solid understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable Big Data pipeline

Feb 2019 18 hours 8 minutes

Apache Hive Essentials

Apache Hive Essentials

Apache Hive helps you deal with data summarization, queries, and analysis for huge amounts of data. This book will give you a background in big data, and familiarize you with your Hive working environment. Next you will cover advanced topics like performance and security in Hive and how to work efficiently to find solutions to big data problems.

Jun 2018 7 hours 0 minutes

Modern Big Data Processing with Hadoop

Modern Big Data Processing with Hadoop

This book presents unique techniques to conquer different Big Data processing and analytics challenges using Hadoop. Practical examples are provided to boost your understanding of Big Data concepts and their implementation. By the end of the book, you will have all the knowledge and skills you need to become a true Big Data expert.

Mar 2018 13 hours 8 minutes

Mastering Apache Storm

Mastering Apache Storm

With real-world examples and clear explanations, this book will ensure you will have a thorough mastery Apache Storm.You’ll get an understanding of deploying Storm on clusters. Introduce yourself to topics such as trident topology, monitoring, Storm Parallelism, scheduler and log processing. Learn how to integrate Storm with other well-known Big Data technologies such as HBase, Redis, Kafka, and Hadoop to realize the full potential of Storm.You will be able to use the knowledge to develop efficient, distributed real-time applications to cater to your business needs.

Aug 2017 9 hours 28 minutes

Data Lake for Enterprises

Data Lake for Enterprises

The term 'Data Lake' has recently emerged as a prominent term in the big data industry. Data scientists can make use of it in deriving meaningful insights which can be used by businesses to redefine or transform the way they operate. Lambda architecture is also emerging as one of the very eminent patterns in the big data landscape, as it helps to derive useful information from not only the historical data but also correlates real-time data to enable business for taking critical decisions. This book tries to bring these two important aspects into one, namely data lake and lambda architecture.

May 2017 19 hours 52 minutes

Personalised recommendations for you

Based on your interests and search pattern

Designing and Implementing Microsoft Azure Networking Solutions

Designing and Implementing Microsoft Azure Networking Solutions

Designing and Implementing Microsoft Azure Networking Solutions Exam Ref AZ-700 is an all-encompassing guide to the AZ-700 exam and contains all the information you need to succeed in the world of virtual networking with Azure. With this book, you will be fully prepared for the exam and the world of cloud networking.

Aug 2023 17 hours 28 minutes

Microsoft 365 Security, Compliance, and Identity Administration

Microsoft 365 Security, Compliance, and Identity Administration

The Microsoft 365 Security, Compliance, and Identity Administration is a comprehensive guide that helps you employ Microsoft 365's robust suite of features and empowers you to optimize your administrative tasks.

Aug 2023 21 hours 0 minutes

Zero Trust Overview and Playbook Introduction

Zero Trust Overview and Playbook Introduction

Get started on Zero Trust with this step-by-step playbook and learn everything you need to know for a successful Zero Trust journey with tailored guidance for every role, covering strategy, operations, architecture, implementation, and measuring success. This book will become an indispensable reference for everyone in your organization.

Oct 2023 8 hours 0 minutes

The Self-Taught Cloud Computing Engineer

The Self-Taught Cloud Computing Engineer

This self-study book helps you master multiple clouds, including AWS, Azure, and GCP, and serves as a roadmap to becoming a certified cloud computing expert. The book will guide you to develop a professional cloud career by helping you build a broad cloud knowledge base, developing hands-on cloud computing skills, and getting cloud certified.

Sep 2023 15 hours 44 minutes

Technology Operating Models for Cloud and Edge

Technology Operating Models for Cloud and Edge

This book will help you build and create ownership of a technology operating model, as well as connect your leadership with engineering and operations, keeping your internal and external customers in mind. It provides practical tips on why, where, and how to make the cloud and edge platform paradigm sing for you, your team, and your organization.

Aug 2023 7 hours 36 minutes

Azure Architecture Explained

Azure Architecture Explained

Azure is the preferred platform to build mission-critical and secure apps. This book provides comprehensive coverage of essential Azure products, services, and solutions vital for every solution architect's success. Elevate your knowledge and master the critical components of Azure to excel in your role with Azure Architecture Explained.

Sep 2023 14 hours 52 minutes

Pentesting Active Directory and Windows-based Infrastructure

Pentesting Active Directory and Windows-based Infrastructure

This practical guide helps you explore the pentesting of Microsoft infrastructure in detail, and enhances your offensive skillset by showing you the different ways to perform security assessment. This book will help blue teamers and IT engineers get up to speed with possible security issues they may encounter in their Windows environments.

Nov 2023 12 hours 0 minutes

Practical Ansible

Practical Ansible

In Practical Ansible, you'll work with the latest release of Ansible and learn to solve complex issues quickly with the help of task-oriented scenarios. You'll start by installing and configuring Ansible to automate monotonous and repetitive IT tasks and get to grips with concepts such as playbooks, inventories, plugins, collections, and network modules.

Sep 2023 14 hours 0 minutes

Windows 11 for Enterprise Administrators

Windows 11 for Enterprise Administrators

Microsoft’s launch of Windows 11 is a step toward satisfying the enterprise administrator’s needs for better management and enhanced user experience customization. This book provides the enterprise administrator with the knowledge needed to fully utilize the advanced feature set of Windows 11 Enterprise.

Oct 2023 9 hours 32 minutes

The Linux DevOps Handbook

The Linux DevOps Handbook

This book is for software and IT professionals seeking knowledge on Linux systems and DevOps practices. This book will provide you with guidance and tools to learn and gain proficiency in managing Linux-based infrastructures and knowledge of DevOps.

Nov 2023 14 hours 16 minutes