Understanding resilient distributed dataset - RDD

Though RDD is getting replaced with DataFrame/DataSet-based APIs, there are still a lot of APIs that have not been migrated yet. In this recipe, we will look at how the concept of lineage works in RDD.

Externally, RDD is a distributed, immutable collection of objects. Internally, it consists of the following five parts:

• Set of partitions (rdd.getPartitions)
• List of dependencies on parent RDDs (rdd.dependencies)
• Function to compute a partition, given its parents
• Partitioner, which is optional (rdd.partitioner)
• Preferred location of each partition, which is optional (rdd.preferredLocations)

The first three are needed for an RDD to be recomputed in case data is lost. When combined, it is called lineage. The last two parts are optimizations.

A set of partitions is how data is divided into nodes. In the case of HDFS, it means InputSplits, which are mostly the same as the block (except when a record crosses block boundaries; in that case, it will be slightly bigger than a block).

How to do it...

Let's revisit our word count example to understand these five parts. This is how an RDD graph looks for wordCount at the dataset level view:

Basically, this is how the flow goes:

1. Load the words folder as an RDD:

scala> val words = sc.textFile("hdfs://localhost:9000/user/hduser/words")

The following are the five parts of the words RDD:

Part	Description
Partitions	One partition per HDFS inputsplit/block (org.apache.spark.rdd.HadoopPartition)
Dependencies	None
Compute function	To read the block
Preferred location	The HDFS block's location
Partitioner	None

2. Tokenize the words of the words RDD with each word on a separate line:

scala> val wordsFlatMap = words.flatMap(_.split("W+"))

The following are the five parts of the wordsFlatMap RDD:

Part	Description
Partitions	Same as the parent RDD, that is, words (org.apache.spark.rdd.HadoopPartition)
Dependencies	Same as the parent RDD, that is, words (org.apache.spark.OneToOneDependency)
Compute function	To compute the parent and split each element, which flattens the results
Preferred location	Ask parent RDD
Partitioner	None

3. Transform each word in the wordsFlatMap RDD into the (word,1) tuple:

scala> val wordsMap = wordsFlatMap.map( w => (w,1))

The following are the five parts of the wordsMap RDD:

Part	Description
Partitions	Same as the parent RDD, that is, wordsFlatMap (org.apache.spark.rdd.HadoopPartition)
Dependencies	Same as the parent RDD, that is, wordsFlatMap (org.apache.spark.OneToOneDependency)
Compute function	To compute the parent and map it to PairRDD
Preferred Location	Ask parent RDD
Partitioner	None

4. Reduce all the values of a given key and sum them up:

scala> val wordCount = wordsMap.reduceByKey(_+_)

The following are the five parts of the wordCount RDD:

Part	Description
Partitions	One per reduce task (org.apache.spark.rdd.ShuffledRDDPartition)
Dependencies	Shuffle dependency on each parent (org.apache.spark.ShuffleDependency)
Compute function	To perform additions on shuffled data
Preferred location	None
Partitioner	HashPartitioner (org.apache.spark.HashPartitioner)

This is how an RDD graph of wordcount looks at the partition level view: