Large-Scale Data Processing Frameworks

As the volume and complexity of data sources are increasing, deriving value out of data is also becoming increasingly difficult. Ever since Hadoop was made, it has built a massively scalable filesystem, HDFS. It has adopted the MapReduce concepts from functional programming to approach the large-scale data processing challenges. As technology is constantly evolving to overcome the challenges posed by data mining, enterprises are also finding ways to embrace these changes to stay ahead.

In this chapter, we will focus on these data processing solutions:

• MapReduce
• Apache Spark
• Spark SQL
• Spark Streaming

MapReduce

MapReduce is a concept that is borrowed from functional programming. The data processing is broken down into a map phase, where data preparation occurs, and a reduce phase, where the actual results are computed. The reason MapReduce has played an important role is the massive parallelism we can achieve as the data is sharded into multiple distributed servers. Without this advantage, MapReduce cannot really perform well.

Let's take up a simple example to understand how MapReduce works in functional programming:

• The input data is processed using a mapper function of our choice
• The output from the mapper function should be in a state that is consumable by the reduce function
• The output from the mapper function is fed to the reduce function to generate the necessary results

Let's understand these steps using a simple program. This program uses the following text...

Hadoop MapReduce

Apache MapReduce is a framework that makes it easier for us to run MapReduce operations on very large, distributed datasets. One of the advantages of Hadoop is a distributed file system that is rack-aware and scalable. The Hadoop job scheduler is intelligent enough to make sure that the computation happens on the nodes where the data is located. This is also a very important aspect as it reduces the amount of network IO.

Let's see how the framework makes it easier to run massively parallel computations with the help of this diagram:

This diagram looks a bit more complicated than the previous diagram, but most of the things are done by the Hadoop MapReduce framework itself for us. We still write the code for mapping and reducing our input data.

Let's see in detail what happens when we process our data with the Hadoop MapReduce framework from the preceding...

Apache Spark 2

Apache Spark is a general-purpose cluster computing system. It's very well suited for large-scale data processing. It performs 100 times better than Hadoop when run completely in-memory and 10 times better when run entirely from disk. It has a sophisticated directed acyclic graph execution engine that supports an acyclic data flow model.

Apache Spark has first-class support for writing programs in Java, Scala, Python, and R programming languages to cater to a wider audience. It offers more than 80 different operators to build parallel applications without worrying about the underlying infrastructure.

Apache Spark has libraries catering to Structured Query Language, known as Spark SQL; this supports writing queries in programs using ANSI SQL. It also has support for computing streaming data, which is very much needed in today's real-time data processing...

Summary

In this chapter, you looked at the basic concepts of large-scale data processing frameworks and also learned that one of the powerful features of spark is building applications that process real-time streaming data and produce real-time results.

In the next few chapters, we will discuss how to build real-time data search pipelines with Elasticsearch stack.