Spark Optimizations

In the previous chapters, we learned how to use Spark to implement a variety of use cases using features such as RDDs, DataFrames, Spark SQL, MLlib, GraphX/Graphframes, and Spark Streaming. We also discussed how to monitor your applications to better understand their behavior in production. However, sometimes, you would want your jobs to run efficiently. We measure the efficiency of any job on two parameters: runtime and storage space. In the Spark application, you might also be interested in the statistic of the data shuffles between the nodes. We discussed some of the optimizations in the earlier chapters, but, in this chapter, we'll discuss more optimizations that can help you achieve some performance benefits.

Most developers focus only on writing their applications on Spark and do not focus on optimizing their job for a variety of reasons. This chapter...

Cluster-level optimizations

As discussed in Chapter 1, Introduction to Apache Spark, Spark can scale horizontally. This means that performance will increase if you add more nodes to your cluster, because Spark can perform more operations in parallel. Spark also enables users to take good advantage of memory, and a fast network can also help in optimizing shuffle data. Because of all of these reasons, more hardware is always better.

Memory

Efficient use of memory is critical for good performance. In the earlier versions of Spark, the memory was used for three main purposes:

• RDD storage
• Shuffle and aggregation storage
• User code

Memory was divided among them with some fixed proportions. For example, RDD storage had 60%, shuffle...

Application optimizations

As discussed, there are multiple ways to improve the performance of your Spark applications. In the previous section, we covered some optimizations that were related to hardware. In this section, we'll discuss how you can apply some optimizations while writing your Spark applications.

Language choice

One of the first choices the developers have to make is to decide the language API they are going to write their applications in. In Chapter 1, Introduction to Apache Spark, we gave an overview of all the languages supported by Spark. The choice of language depends on the use case and the dynamics of the team. If you are part of a data science team and comfortable in writing your machine-learning...

Summary

In this chapter, we discussed some of the optimizations provided by Spark. First, we discussed some hardware level optimizations, such as setting the number of cores, executors, and the amount of memory for your Spark applications. We then gave an overview of project Tungsten and its optimizations. Then, we covered application-level optimizations, such as choosing the right language, API and the file format for your applications. Finally, we covered optimizations provided by RDD and DataFrame APIs.

This is the end of this book. I thank you for staying with me till the end, and I hope you enjoyed the content. If you wish to explore Apache Spark in more detail, you may also consider buying Learning Apache Spark 2, by Packt.