Start Working with Spark – REPL and RDDs

"All this modern technology just makes people try to do everything at once."

- Bill Watterson

In this chapter, you will learn how Spark works; then, you will be introduced to RDDs, the basic abstractions behind Apache Spark, and you'll learn that they are simply distributed collections exposing Scala-like APIs. You will then see how to download Spark and how to make it run locally via the Spark shell.

In a nutshell, the following topics will be covered in this chapter:

• Dig deeper into Apache Spark
• Apache Spark installation
• Introduction to RDDs
• Using the Spark shell
• Actions and Transformations
• Caching
• Loading and Saving data

Dig deeper into Apache Spark

Apache Spark is a fast in-memory data processing engine with elegant and expressive development APIs to allow data workers to efficiently execute streaming machine learning or SQL workloads that require fast interactive access to datasets. Apache Spark consists of Spark core and a set of libraries. The core is the distributed execution engine and the Java, Scala, and Python APIs offer a platform for distributed application development.

Additional libraries built on top of the core allow the workloads for streaming, SQL, Graph processing, and machine learning. SparkML, for instance, is designed for Data science and its abstraction makes Data science easier.

In order to plan and carry out the distributed computations, Spark uses the concept of a job, which is executed across the worker nodes using Stages and Tasks. Spark consists of a driver, which orchestrates...

Apache Spark installation

Apache Spark is a cross-platform framework, which can be deployed on Linux, Windows, and a Mac Machine as long as we have Java installed on the machine. In this section, we will look at how to install Apache Spark.

Apache Spark can be downloaded from http://spark.apache.org/downloads.html

First, let's look at the pre-requisites that must be available on the machine:

• Java 8+ (mandatory as all Spark software runs as JVM processes)
• Python 3.4+ (optional and used only when you want to use PySpark)
• R 3.1+ (optional and used only when you want to use SparkR)
• Scala 2.11+ (optional and used only to write programs for Spark)

Spark can be deployed in three primary deployment modes, which we will look at:

• Spark standalone
• Spark on YARN
• Spark on Mesos

Spark standalone...

Introduction to RDDs

A Resilient Distributed Dataset (RDD) is an immutable, distributed collection of objects. Spark RDDs are resilient or fault tolerant, which enables Spark to recover the RDD in the face of failures. Immutability makes the RDDs read-only once created. Transformations allow operations on the RDD to create a new RDD but the original RDD is never modified once created. This makes RDDs immune to race conditions and other synchronization problems.

The distributed nature of the RDDs works because an RDD only contains a reference to the data, whereas the actual data is contained within partitions across the nodes in the cluster.

Conceptually, a RDD is a distributed collection of elements spread out across multiple nodes in the cluster. We can simplify a RDD to better understand by thinking of a RDD as a large array of integers distributed across machines.

A RDD is...

Using the Spark shell

Spark shell provides a simple way to perform interactive analysis of data. It also enables you to learn the Spark APIs by quickly trying out various APIs. In addition, the similarity to Scala shell and support for Scala APIs also lets you also adapt quickly to Scala language constructs and make better use of Spark APIs.

Spark shell implements the concept of read-evaluate-print-loop (REPL), which allows you to interact with the shell by typing in code which is evaluated. The result is then printed on the console, without needing to be compiled, so building executable code.

Start it by running the following in the directory where you installed Spark:

./bin/spark-shell

Spark shell launches and the Spark shell automatically creates the SparkSession and SparkContext objects. The SparkSession is available as a Spark and the SparkContext is available as sc.

spark...

Actions and Transformations

RDDs are immutable and every operation creates a new RDD. Now, the two main operations that you can perform on an RDD are Transformations and Actions.

Transformations change the elements in the RDD such as splitting the input element, filtering out elements, and performing calculations of some sort. Several transformations can be performed in a sequence; however no execution takes place during the planning.

For transformations, Spark adds them to a DAG of computation and, only when driver requests some data, does this DAG actually gets executed. This is called lazy evaluation.

The reasoning behind the lazy evaluation is that Spark can look at all the transformations and plan the execution, making use of the understanding the Driver has of all the operations. For instance, if a filter transformation is applied immediately after some other transformation...

Caching

Caching enables Spark to persist data across computations and operations. In fact, this is one of the most important technique in Spark to speed up computations, particularly when dealing with iterative computations.

Caching works by storing the RDD as much as possible in the memory. If there is not enough memory then the current data in storage is evicted, as per LRU policy. If the data being asked to cache is larger than the memory available, the performance will come down because Disk will be used instead of memory.

You can mark an RDD as cached using either persist() or cache()

cache() is simply a synonym for persist(MEMORY_ONLY)

persist can use memory or disk or both:

persist(newLevel: StorageLevel) 

The following are the possible values for Storage level:

Storage Level	Meaning
MEMORY_ONLY	Stores RDD as deserialized Java objects in the JVM. If the RDD does not...

Loading and saving data

Loading data into an RDD and Saving an RDD onto an output system both support several different methods. We will cover the most common ones in this section.

Loading data

Loading data into an RDD can be done by using SparkContext. Some of the most common methods are:.

• textFile
• wholeTextFiles
• load from a JDBC datasource

textFile

textFile() can be used to load textFiles into an RDD and each line becomes an element in the RDD.

sc.textFile(name, minPartitions=None, use_unicode=True)

The following is an example of loading a textfile into an RDD using...

Summary

In this chapter, we discussed the internals of Apache Spark, what RDDs are, DAGs and lineages of RDDs, Transformations, and Actions. We also looked at various deployment modes of Apache Spark using standalone, YARN, and Mesos deployments. We also did a local install on our local machine and then looked at Spark shell and how it can be used to interact with Spark.

In addition, we also looked at loading data into RDDs and saving RDDs to external systems as well as the secret sauce of Spark's phenomenal performance, the caching functionality, and how we can use memory and/or disk to optimize the performance.

In the next chapter, we will dig deeper into RDD API and how it all works in Chapter 7, Special RDD Operations.