Chapter 6. Securing Sensitive Data in Hadoop

In Chapter 5, Integrating Hadoop with Enterprise Security Systems, we looked at integrating a secured Hadoop cluster with an Enterprise Identity Management system and enforce user authorization within Hadoop. User privileges are managed centrally and then synchronized with the secured Hadoop cluster. This enables enterprise users to access secured Hadoop services seamlessly. As an organization matures with their Big Data implementations, there is an increasing need to move sensitive information into the Hadoop ecosystem to generate valuable insights. Sensitive data in the cluster needs special protection and should be secured both at rest and in motion.

In this chapter, we look at how to secure sensitive data within a Hadoop ecosystem.

These are the topics we'll be covering in this chapter:

• Securing sensitive data in Hadoop

• Encrypting sensitive data in Hadoop

• Implementing data encryption in Hadoop

Securing sensitive data in Hadoop

Sensitive data inside Hadoop can be classified into two high-level categories:

• Sensitive data related to customers' personal information, customers' financial information, and so on that exists in enterprise systems and that needs to be brought to Hadoop for analysis.

• The Hadoop analytical process generates sensitive insights after processing the data stored inside Hadoop. These insights are more valuable and sensitive compared to the raw source data that is used to generate them. For example, a retail e-commerce enterprise has detailed transactions of customer purchases. These transaction details might not be very sensitive. This data is brought to Hadoop for generating various insights. Using the customer historical purchases and correlating the same with customer's household purchases, insights related to customer purchase patterns, behavior patterns, customer sentiment, and customer life events could be inferred. This information is highly sensitive compared...

Summary

In this chapter, we looked at how to secure sensitive data in the Hadoop cluster. We looked at the approaches for encryption of data in motion while block-level encryption for data is at rest. We also looked at the MapReduce processing and ways to enforce data encryption on the input side, intermediate data, and the final results created by the MapReduce program. Encryption causes performance degradation and this has to be carefully evaluated so that only sensitive data is encrypted and secured.

In the next chapter, we will look at how to identify security incidents and events in a secured Hadoop cluster. And we will also look at how to implement auditing and logging of user activities in the Hadoop cluster.