Introduction to benchmarking and profiling

The Hadoop cluster are used by the organizations in different ways. One of the primary ways is to build data lakes on top of the Hadoop cluster. A data lake is built on top of different types of data sources. Each of these data sources varies in nature, such as the type of data or frequency of data. Every type of data processing for those sources in data lakes varies. Some are real-time processing and some are batch-time processing. Your Hadoop cluster on top of which the data lake is built has to take care of such different types of workloads. These workloads are memory intensive, and some are memory as well as CPU intensive. As an organization, it becomes imperative that you benchmark and profile your cluster for these different types of workloads. Another reason for benchmarking and profiling your cluster is that your cluster nodes may have different hardware configurations. For varying workloads, it is important that organizations ensure how...