In this chapter, we cover the methods for accessing big data from Jupyter. Big data is meant to be large data files, often in the many millions of rows. Big data is a topic of discussion in many firms. Most firms have it in one form or another, and they are trying hard to draw some value from all of the data they have stored.
An up-and-coming language for dealing with large datasets is Spark. Spark is an open source toolset specifically made for dealing with large datasets. We can use Spark coding in Jupyter much like the other languages we have seen.
In Chapter 2,Adding an Engine, we dealt with installing Spark for use in Jupyter. For this chapter, we will be using the Python 3 engine for further work. As a reminder, we start a Notebook using the Python 3 engine and then import the Python-Spark library to invoke Spark functionality.
Most importantly, we will be using Spark to access big data.
