Chapter 13. Who Can Do What in Hadoop

This chapter introduces you to security in the Hadoop ecosystem. When you are adopting Hadoop in your enterprise, then security becomes very important. You do not want people with unauthorized access to be able to reach the data stored in the HDFS File System. Security not only concerns a single aspect; you have to think about multiple aspects while securing your Hadoop enterprise application. Let's look at these aspects and understand what roles these play in securing Hadoop-based enterprise applications.

In this chapter, we will cover the following topics:

• Different aspects of Hadoop security pillars
• Security systems
• Kerberos authentication
• User authorization

Hadoop security pillars

Before designing security for your Hadoop cluster, you should be very clear about the different aspects of security that have to be incorporated. This section talks about the different pillars of security that are required to make your Hadoop cluster secure and threat-proof. We will discuss each of those pillars in brief in this section. Later, they will be elaborately discussed in different sections of this chapter. 

The following diagram gives you a glimpse of the Hadoop security pillars:

Before discussing the rings in the preceding diagram, we should understand one important aspect of securing the Hadoop cluster, and that is Security Administration. As a Hadoop security administrator, you have to perform the following activities at a high level:

• Get or develop a means of centralized Hadoop security enablement in terms of automated scripts or security tools.
• Get or develop a means of a centralized Hadoop monitoring and alerting system. This system should comply with...

System security

System security is mostly related to the Operating System (OS) security and remote Secure Shell (SSH) access to nodes. OS security consists of regular checking and resolution of OS security vulnerability by applying patches or workarounds. As an administrator, you need to be aware of OS vulnerabilities and new malware that's released by hackers. You should also be aware of the different security patches and workarounds for those vulnerabilities.

Note

The following link has details of the latest OS vulnerabilities and malware:https://www.cvedetails.com/

 

A Hadoop cluster consists of a variety of nodes with different profiles. Some are master nodes consisting of NameNodes and journal nodes. Some are worker nodes consisting of HDFS DataNodes and HBase region servers. Especially in the case of remote SSH access, your firewall rules may also vary as per profile types of nodes. However, the following table is specific to SSH access and what kind of roles should have SSH access to...

Kerberos authentication

There are various challenges involved in Big Data like storing, processing, and analysis, and managing and securing large data assets. When enterprises started implementing Hadoop, securing Hadoop from an enterprise context became challenging due to the distributed nature of the ecosystem and a wide range of applications that are placed on top of Hadoop. One of the key security considerations in securing Hadoop is authentication.

Kerberos is chosen by the Hadoop team as the component for implementing authentication in Hadoop. Kerberos is a secured network authentication protocol that brings in major authentication for client-server applications without transferring the password through the network. Kerberos implements time-sensitive tickets that are created using symmetric key cryptography, which was chosen over the most widely used SSL-based authentication.

Kerberos advantages

There are different advantages of Kerberos, which are as follows:

• Better performance: Kerberos...

User authorization

Once the identity of the end user is established via Kerberos authentication, the next step in Hadoop security is to ensure what actions or services those established identities can perform. Authorization deals with that. In the following sections, we will look into how authorization rules can be established for different users across different services and how data is stored in HDFS. We will look into two different types of tools that facilitate centralized security policy management for authorization. Let's look into these in brief.

Ranger

The following diagram represents the architecture of the Ranger tool, which lets you centrally manage security policies for different Hadoop services:

As shown in the preceding diagram, all policies are centrally managed through an administrative web portal. The portal has three distinct parts, namely auditing, KMS, and the policy server. The policy server has several different functionalities. The following are the prime functions of...

List of security features that have been worked upon in Hadoop 3.0

The following is the a of the JIRAs that have been solved or worked upon with the Hadoop 3.0 release:

AliyunOSS: update oss-sdk version to 3.0.0 MajorResolvedFixed:

Summary

In this chapter, we have covered the Hadoop security pillars, and it should now be very clear about the different aspects of security that have to be incorporated along with system security and Kerberos authentication. We studied the different advantages of Kerberos, along with how Kerberos authentication flows. This chapter also talked about user authorization, which includes two different tools, namely Ranger and Sentry. Lastly, we were also introduced to the list of JIRAs that have been solved or worked upon with the Hadoop 3.0 release.

In the next chapter, we will study network and data security, which includes aspects like Hadoop networks, perimeter security, data encryption, data masking, row, and column level security.

The rest of the chapter is locked

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Renews at $15.99/month. Cancel anytime

Start free trial

Previous Chapter

Chapter 1 of 23

Next Chapter

You have been reading a chapter from

Mastering Hadoop 3

Published in: Feb 2019 Publisher: Packt ISBN-13: 9781788620444

© 2019 Packt Publishing Limited All Rights Reserved

Register for a free Packt account to unlock a world of extra content!

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

Sign up now

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime}

Authors (2)

Chanchal Singh

Chanchal Singh has over half decades experience in Product Development and Architect Design. He has been working very closely with leadership team of various companies including directors ,CTO's and Founding members to define technical road-map for company.He is the Founder and Speaker at meetup group Big Data and AI Pune MeetupExperience Speaks. He is Co-Author of Book Building Data Streaming Application with Apache Kafka. He has a Bachelor's degree in Information Technology from the University of Mumbai and a Master's degree in Computer Application from Amity University. He was also part of the Entrepreneur Cell in IIT Mumbai. His Linkedin Profile can be found at with the username Chanchal Singh.

Read more

See other products by Chanchal Singh

Manish Kumar

Manish Kumar works as Director of Technology and Architecture at VSquare. He has over 13 years' experience in providing technology solutions to complex business problems. He has worked extensively on web application development, IoT, big data, cloud technologies, and blockchain. Aside from this book, Manish has co-authored three books (Mastering Hadoop 3, Artificial Intelligence for Big Data, and Building Streaming Applications with Apache Kafka).

See other products by Manish Kumar

Other recommended products

Related to this chapter

Building Data Streaming Applications with Apache Kafka

Apache Kafka is a popular distributed streaming platform which acts as a messaging queue or an enterprise messaging system. This book is a comprehensive guide on designing and architecting enterprise-grade streaming applications using Apache Kafka and other Big Data tools. Once you grasp the basics, we will take you through the more advance concepts in Apache Kafka such as capacity planning and security.

Aug 2017 9 hours 16 minutes

Hadoop 2.x Administration Cookbook

A practical and use case driven approach to Hadoop administration with coverage on a vast array of topics including Hadoop cluster installation, performance tuning, cluster planning, security, and much more. This book covers Hadoop from the perspective of running clusters in critical and large environments with complex data and at scale.

May 2017 11 hours 36 minutes

Apache Hadoop 3 Quick Start Guide

Apache Hadoop is a widely used distributed data platform. It enables large datasets to be efficiently processed instead of using one large computer to store and process the data. This book will get you started with the Hadoop ecosystem, and introduce you to the main technical topics such as MapReduce, YARN and HDFS.

Oct 2018 7 hours 20 minutes

Mastering Apache Storm

With real-world examples and clear explanations, this book will ensure you will have a thorough mastery Apache Storm.You’ll get an understanding of deploying Storm on clusters. Introduce yourself to topics such as trident topology, monitoring, Storm Parallelism, scheduler and log processing. Learn how to integrate Storm with other well-known Big Data technologies such as HBase, Redis, Kafka, and Hadoop to realize the full potential of Storm.You will be able to use the knowledge to develop efficient, distributed real-time applications to cater to your business needs.

Aug 2017 9 hours 28 minutes

Learning Apache Flink

Feb 2017 9 hours 20 minutes

Big Data Analytics with Hadoop 3

Apache Hadoop is the most popular platform for big data processing to build powerful analytics solutions. This book shows you how to do just that, with the help of practical examples. You will be well-versed with the analytical capabilities of Hadoop ecosystem with Apache Spark and Apache Flink to perform big data analytics by the end of this book.

May 2018 16 hours 4 minutes

Modern Big Data Processing with Hadoop

This book presents unique techniques to conquer different Big Data processing and analytics challenges using Hadoop. Practical examples are provided to boost your understanding of Big Data concepts and their implementation. By the end of the book, you will have all the knowledge and skills you need to become a true Big Data expert.

Mar 2018 13 hours 8 minutes

Apache Hive Essentials

Apache Hive helps you deal with data summarization, queries, and analysis for huge amounts of data. This book will give you a background in big data, and familiarize you with your Hive working environment. Next you will cover advanced topics like performance and security in Hive and how to work efficiently to find solutions to big data problems.

Jun 2018 7 hours 0 minutes

HBase High Performance Cookbook

Jan 2017 11 hours 40 minutes

Data Lake for Enterprises

The term 'Data Lake' has recently emerged as a prominent term in the big data industry. Data scientists can make use of it in deriving meaningful insights which can be used by businesses to redefine or transform the way they operate. Lambda architecture is also emerging as one of the very eminent patterns in the big data landscape, as it helps to derive useful information from not only the historical data but also correlates real-time data to enable business for taking critical decisions. This book tries to bring these two important aspects into one, namely data lake and lambda architecture.

May 2017 19 hours 52 minutes

Artificial Intelligence for Big Data

Create smart systems to extract intelligent insights for decision making. You will learn about widely used Artificial Intelligence techniques for carrying out solutions in a production-ready environment. You'll explore advanced topics such as clustering, symbolic and sub-symbolic information representation, and many more.

May 2018 12 hours 48 minutes

Apache Kafka 1.0 Cookbook

Apache Kafka is an open source stream processing platform to handle real-time data feeds. This book is a highly practical guide to help you understand the fundamentals as well as the advanced applications of Apache Kafka as an enterprise messaging service. It begins with configuring the basic Kafka APIs, and then shows you how to set up Kafka clusters and basic Kafka operations. It covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. By the end of this book, you will have all the knowledge you need to take your understanding of Apache Kafka to the next level and tackle any problem you might encounter while working with it.

Dec 2017 8 hours 20 minutes

Personalised recommendations for you

Based on your interests and search pattern

Et al.

Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.

Aug 2023 7 hours 40 minutes

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

Dec 2023 12 hours 0 minutes

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

Dec 2023 12 hours 0 minutes

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

Dec 2023 12 hours 0 minutes

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

Dec 2023 12 hours 0 minutes

Mastering Tableau 2023

This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.

Aug 2023 22 hours 48 minutes

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

Sep 2023 8 hours 36 minutes

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

Sep 2023 8 hours 36 minutes

Data Engineering with AWS

Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.

Oct 2023 21 hours 12 minutes

Modern Data Architecture on AWS

Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.

Aug 2023 14 hours 0 minutes

Practical Guide to Applied Conformal Prediction in Python

Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.

Dec 2023 8 hours 0 minutes

TinyML Cookbook

With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.

Nov 2023 22 hours 8 minutes

Machine Learning Kubernetes React Python Cybersecurity Unity Game Development Power BI Azure Web Development

Legal

Terms and Conditions Privacy Policy Cookie Policy Shipping Policy Cancellation Policy Return Policy

Support

Help Contact Us

Business

Partnerships

Sponsored eBooks Custom eBooks

Careers Become an author

Packt+ Membership

Subscription DataPro SecPro TechLeaders

United States

Company Address: Packt Publishing Ltd, Grosvenor House, 11 St Paul's Square, Birmingham, B3 1RB © 2024 Packt Publishing Limited All Rights Reserved

Issue Type	Issue key	Issue ID	Parent ID	Summary
Bug	HADOOP-15866	13192984		Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility
Task	HADOOP-15816	13189089		Upgrade Apache ZooKeeper version due to security concerns
Bug	HADOOP-15861	13192122		Move DelegationTokenIssuer to the right path
Bug	HADOOP-15523	13164833		Shell command timeout given is in seconds whereas it is taken as millisec while scheduling
Improvement	HADOOP-15609	13172372		Retry KMS calls when SSLHandshakeException occurs
Improvement	HADOOP-15804	13188359		Upgrade to commons-compress 1.18
Bug	HADOOP-15698	13181280		KMS log4j is not initialized properly at startup
Bug	HADOOP-15864	13192767		Job submitter/executor fail when SBN domain name can not resolved
Subtask	HADOOP-15607	13172317	12989378	AliyunOSS: fix duplicated...

Summary

In this chapter, we have covered the Hadoop security pillars, and it should now be very clear about the different aspects of security that have to be incorporated along with system security and Kerberos authentication. We studied the different advantages of Kerberos, along with how Kerberos authentication flows. This chapter also talked about user authorization, which includes two different tools, namely Ranger and Sentry. Lastly, we were also introduced to the list of JIRAs that have been solved or worked upon with the Hadoop 3.0 release.

In the next chapter, we will study network and data security, which includes aspects like Hadoop networks, perimeter security, data encryption, data masking, row, and column level security.