Chapter 2. Hadoop Security Design

In Chapter 1, Hadoop Security Overview, we discussed the security considerations for an end-to-end Hadoop-based Big Data ecosystem. In this chapter, we will narrow our focus and take a deep dive into the security design of the Hadoop platform. Hadoop security was implemented as part of the HADOOP-4487 Jira issue, starting in late 2009 (https://issues.apache.org/jira/browse/HADOOP-4487). Currently, there are efforts to implement SSO Authentication in Hadoop. This is currently not production-ready, and hence will be out of scope of this book.

Hadoop security implementation is based on Kerberos. So in this chapter, first we will be provided with a high-level overview of key Kerberos terminologies and concepts, and then we will look into the details of the Hadoop security implementation.

The following are the topics we'll be covering in this chapter:

• What is Kerberos?

• The Hadoop default security model

• The Hadoop Kerberos security implementation

What is Kerberos?

In any distributed system, when two parties (the client and server) have to communicate over the network, the first step in this communication is to establish trust between these parties. This is usually done through the authentication process, where the client presents its password to the server and the server verifies this password. If the client sends passwords over an unsecured network, there is a risk of passwords getting compromised as they travel through the network.

Kerberos is a secured network authentication protocol that provides strong authentication for client/server applications without transferring the password over the network. Kerberos works by using time-sensitive tickets that are generated using the symmetric key cryptography. Kerberos is derived from the Greek mythology where Kerberos was the three-headed dog that guarded the gates of Hades. The three heads of Kerberos in the security paradigm are:

• The user who is trying to authenticate.

• The service to...

The Hadoop default security model without Kerberos

Now that we understand how the Kerberos security protocol works, let us look at the details of the Hadoop default security model and its limitations.

Hadoop implements a security model similar to the POSIX filesystem, which gives the ability to apply file permissions and restrict read-write access to files and directories in HDFS. The user and admin can use the chmod and chown commands to change the permissions and ownership of the file/directories, similar to the POSIX filesystem. Hadoop does not provide any user management functionality. It uses the operating system user within Hadoop.

By default, Hadoop doesn't support any authentication of users or Hadoop services. A user only authenticates with the operating system during the logon process. After that, when the user invokes the Hadoop command, the user ID and group is set by executing whoami and bash -c groups respectively. So if a user writes their own whoami script and adds it to the...

Hadoop Kerberos security implementation

Enforcing security within a distributed system such as Hadoop is complex. The detailed requirements for securing Hadoop were identified by Owen O'Malley and others as part of the Hadoop security design. The detailed document is attached with the ticket HADOOP-4487 at https://issues.apache.org/jira/browse/HADOOP-4487. A summary of these requirements is explained in this section.

User-level access controls

A brief on the user-level access controls is:

• Users of Hadoop should only be able to access data that is authorized for them

• Only authenticated users should be able to submit jobs to the Hadoop cluster

• Users should be able to view, modify, and kill only their own jobs

• Only authenticated services should be able to register themselves as DataNodes or TaskTracker

• Data block access within DataNode needs to be secured, and only authenticated users should be able to access the data stored in the Hadoop cluster

Service-level access controls

Here's a gist of the service...

Summary

In this chapter, we looked at the Kerberos authentication protocol and understood the key concepts involved in implementing Kerberos. We understood the default security implementation in Hadoop and how a Hadoop process gets the logged in user and group details. The default security implementation has many gaps and can't be used in production.

In a production scenario, securing Hadoop with Kerberos is essential. So we looked at the requirements that Hadoop supports at the user and Hadoop service level to secure the Hadoop cluster. We looked at the various internal secret keys (Delegation Token, Block Access Token, and Job Token) that are exchanged by the various Hadoop processes to ensure a secured ecosystem. Understanding the need and use of these tokens is vital to debug and troubleshoot any configuration issues in a secured Hadoop cluster. In the next chapter we will detail the procedure for securing a Hadoop cluster.