Testing Apache Spark Jobs

In this chapter, we will test Apache Spark jobs and learn how to separate logic from the Spark engine.

We will first cover unit testing of our code, which will then be used by the integration test in SparkSession. Later, we will be mocking data sources using partial functions, and then learn how to leverage ScalaCheck for property-based testing for a test as well as types in Scala. By the end of this chapter, we will have performed tests in different versions of Spark.

In this chapter, we will be covering the following topics:

• Separating logic from Spark engine-unit testing
• Integration testing using SparkSession
• Mocking data sources using partial functions
• Using ScalaCheck for property-based testing
• Testing in different versions of Spark

Separating logic from Spark engine-unit testing

Let's start by separating logic from the Spark engine.

In this section, we will cover the following topics:

• Creating a component with logic
• Unit testing of that component
• Using the case class from the model class for our domain logic

Let's look at the logic first and then the simple test.

So, we have a BonusVerifier object that has only one method, quaifyForBonus, that takes our userTransaction model class. According to our login in the following code, we load user transactions and filter all users that are qualified for a bonus. First, we need to test it to create an RDD and filter it. We need to create a SparkSession and also create data for mocking an RDD or DataFrame, and then test the whole Spark API. Since this involves logic, we will test it in isolation. The logic is as follows:

package com.tomekl007.chapter_6...

Integration testing using SparkSession

Let's now learn about integration testing using SparkSession.

In this section, we will cover the following topics:

• Leveraging SparkSession for integration testing
• Using a unit tested component

Here, we are creating the Spark engine. The following line is crucial for the integration test:

 val spark: SparkContext = SparkSession.builder().master("local[2]").getOrCreate().sparkContext

It is not a simple line just to create a lightweight object. SparkSession is a really heavy object and constructing it from scratch is an expensive operation from the perspective of resources and time. Tests such as creating SparkSession will take more time compared to the unit testing from the previous section.

For the same reason, we should use unit tests often to convert all edge cases and use integration testing only for the smaller part of...

Mocking data sources using partial functions

In this section, we will cover the following topics:

• Creating a Spark component that reads data from Hive
• Mocking the component
• Testing the mock component

Let's assume that the following code is our production line:

 ignore("loading data on prod from hive") {
 UserDataLogic.loadAndGetAmount(spark, HiveDataLoader.loadUserTransactions)
 }

Here, we are using the UserDataLogic.loadAndGetAmount function, which needs to load our user data transaction and get the amount of the transaction. This method takes two arguments. The first argument is a sparkSession and the second argument is the provider of sparkSession, which takes SparkSession and returns DataFrame, as shown in the following example:

object UserDataLogic {
  def loadAndGetAmount(sparkSession: SparkSession, provider: SparkSession => DataFrame): DataFrame = {
   ...

Using ScalaCheck for property-based testing

In this section, we will cover the following topics:

• Property-based testing
• Creating a property-based test

Let's look at a simple property-based test. We need to import a dependency before we define properties. We also need a dependency for the ScalaCheck library, which is a library for property-based tests.

In the previous section, every test extended FunSuite. We used functional tests, but we had to provide arguments explicitly. In this example, we're extending Properties from the ScalaCheck library and testing a StringType, as follows:

object PropertyBasedTesting extends Properties("StringType")

Our ScalaCheck will generate a random string for us. If we create a property-based test for a custom type, then that is not known to the ScalaCheck. We need to provide a generator that will generate instances of that...

Testing in different versions of Spark

In this section, we will cover the following topics:

• Changing the component to work with Spark pre-2.x
• Mock testing pre-2.x
• RDD mock testing

Let's start with the mocking data sources from the third section of this chapter—Mocking data sources using partial functions.

Since we were testing UserDataLogic.loadAndGetAmount, notice that everything operates on the DataFrame and thus we had a SparkSession and DataFrame.

Now, let's compare it to the Spark pre-2.x. We can see that this time, we are unable to use DataFrames. Let's assume that the following example shows our logic from the previous Sparks:

test("mock loading data from hive"){
 //given
 import spark.sqlContext.implicits._
 val df = spark.sparkContext
 .makeRDD(List(UserTransaction("a", 100), UserTransaction("b", 200)))
 .toDF()
 .rdd...

Summary

In this chapter, we first learned how to separate logic from the Spark engine. We then looked at a component that was well-tested in separation without the Spark engine, and we carried out integration testing using SparkSession. For this, we created a SparkSession test by reusing the component that was already well-tested. By doing that, we did not have to cover all edge cases in the integration test and our test was much faster. We then learned how to leverage partial functions to supply mocked data that's provided at the testing phase. We also covered ScalaCheck for property-based testing. By the end of this chapter, we had tested our code in different versions of Spark and learned how to change our DataFrame mock test to RDD.

In the next chapter, we will learn how to leverage the Spark GraphX API.