Chapter 15. Monitoring Hadoop

In the previous chapter, we focused on the different aspects of Hadoop security. So far, we have walked you through the different components in Hadoop, some advanced concepts of the Hadoop ecosystem, and a few of the best design practices that have to be taken into consideration while designing and implementing Hadoop applications. Above all, it is also important to monitor and alert any application system to avoid unnecessary losses or system failure by taking appropriate actions. In this chapter, we will cover the following topics:

• General monitoring
• Security monitoring

General monitoring

The of applications goes through various stages, such as the development environment, load test environment, stage environment, and then, finally, it goes to the production environment. Most of the application architecture has more than one component, such as the UI server, backend server, and database server. All components interacts with each other to fulfill the overall application objectives in any enterprise, no matter how big or small, so it is very important to monitor the health of the services. These server are intended to run 24/7 and if any of the servers go down it may lead to the failure of the business objective and may cause huge losses to the company. It is very important to have the proper mechanisms in place to continuously observe the health, failure, network issues, database performance, application performance, and so on, of the applications. Monitoring is the process of capturing important application matrices that can help us obtain a server report...

Security monitoring

Production-level Hadoop ecosystems are implemented with sufficient authentication, authorization, and data security in place. We have already covered various ways to implement security in Hadoop in previous chapters. But, as discussed, while implementation is one thing, monitoring and alerting are another. What if someone log into your system using some other mechanism or someone tries to enter your system by some other route? What if a user performs operations they are not allowed to?

Security information and event management

Security information and event management (SIEM) is an audit that logs entries and from the security system and converts them into an actionable item. This actionable information can be used to detect any potential threat, take action, and add a new investigation into the compliance process. Depending on how the SIEM system is designed, this entire process can be either in batch or real time, based on how critical the security threat is. The following...

Summary

In this chapter, we covered how we monitor the Hadoop ecosystem. We started by covering general security parameters such as a few metrics that can be helpful in identifying system bottlenecks. We also looked at the basic rules for configuring monitoring and alerting systems are. In the security monitoring section, we talked about the SIEM system and how this system works. There are many tools on the market that have SIEM implementation and can be used as plug and play modules. Later in the chapter, we covered intrusion detection systems and intrusion prevention systems.