Immutable Design

In this chapter, we will look at the immutable design of Apache Spark. We will delve into the Spark RDD's parent/child chain and use RDD in an immutable way. We will then use DataFrame operations for transformations to discuss immutability in a highly concurrent environment. By the end of this chapter, we will use the Dataset API in an immutable way.

In this chapter, we will cover the following topics:

• Delving into the Spark RDD's parent/child chain
• Using RDD in an immutable way
• Using DataFrame operations to transform
• Immutability in the highly concurrent environment
• Using the Dataset API in an immutable way

Delving into the Spark RDD's parent/child chain

In this section, we will try to implement our own RDD that inherits the parent properties of RDD.

We will go through the following topics:

• Extending an RDD
• Chaining a new RDD with the parent
• Testing our custom RDD

Extending an RDD

This is a simple test that has a lot of hidden complexity. Let's start by creating a list of the record, as shown in the following code block:

class InheritanceRdd extends FunSuite {
  val spark: SparkContext = SparkSession
    .builder().master("local[2]").getOrCreate().sparkContext

  test("use extended RDD") {
    //given
    val rdd = spark.makeRDD(List(Record(1, "d1")))

The Record is just a case class that...

Using RDD in an immutable way

Now that we know how to create a chain of execution using RDD inheritance, let's learn how to use RDD in an immutable way.

In this section, we will go through the following topics:

• Understating DAG immutability
• Creating two leaves from the one root RDD
• Examining results from both leaves

Let's first understand directed acyclic graph immutability and what it gives us. We will then be creating two leaves from one node RDD, and checking if both leaves are behaving totally independently if we create a transformation on one of the leaf RDD's. We will then examine results from both leaves of our current RDD and check if any transformation on any leaf does not change or impact the root RDD. It is imperative to work like this because we have found that we will not be able to create yet another leaf from the root RDD, because the root RDD will...

Using DataFrame operations to transform

The data from the API has an RDD underneath it, and so there is no way that the DataFrame could be mutable. In DataFrame, the immutability is even better because we can add and subtract columns from it dynamically, without changing the source dataset.

In this section, we will cover the following topics:

• Understanding DataFrame immutability
• Creating two leaves from the one root DataFrame
• Adding a new column by issuing transformation

We will start by using data from operations to transform our DataFrame. First, we need to understand DataFrame immutability and then we will create two leaves, but this time from the one root DataFrame. We will then issue a transformation that is a bit different than the RDD. This will add a new column to our resulting DataFrame because we are manipulating it this way in a DataFrame. If we want to map data,...

Immutability in the highly concurrent environment

We saw how immutability affects the creation and design of programs, so now we will understand how it is useful.

In this section, we will cover the following topics:

• The cons of mutable collections
• Creating two threads that simultaneously modify a mutable collection
• Reasoning about a concurrent program

Let's first understand the cause of mutable collections. To do this, we will be creating two threads that simultaneously modify the mutable collection. We will be using this code for our test. First, we will create a ListBuffer that is a mutable list. Then, we can add and delete links without creating another list for any modification. We can then create an Executors service with two threads. We need two threads to start simultaneously to modify the state. Later, we will use a CountDownLatch construct from Java.util.concurrent...

Using the Dataset API in an immutable way

In this section, we will use the Dataset API in an immutable way. We will cover the following topics:

• Dataset immutability
• Creating two leaves from the one root dataset
• Adding a new column by issuing transformation

The test case for the dataset is quite similar, but we need to do a toDS() for our data to be type safe. The type of dataset is userData, as shown in the following example:

import com.tomekl007.UserData
import org.apache.spark.sql.SparkSession
import org.scalatest.FunSuite

class ImmutableDataSet extends FunSuite {
 val spark: SparkSession = SparkSession
 .builder().master("local[2]").getOrCreate()

test("Should use immutable DF API") {
 import spark.sqlContext.implicits._
 //given
 val userData =
 spark.sparkContext.makeRDD(List(
 UserData("a", "1"),
 UserData("b", "2"),
 UserData...

Summary

In this chapter, we delved into the Spark RDD parent-child chain and created a multiplier RDD that was able to calculate everything based on the parent RDD, and also based on the partitioning scheme on the parent. We used RDD in an immutable way. We saw that the modification of the leaf that was created from the parent didn't modify the part. We also learned a better abstraction, that is, a DataFrame, so we learned that we can employ transformation there. However, every transformation is just adding to another column—it is not modifying anything in place. Next, we just set immutability in a highly concurrent environment. We saw how the mutable state is bad when accessing multiple threads. Finally, we saw that the Dataset API is also created in an immutable type of way and that we can leverage those things here.

In the next chapter, we'll look at how to...