Getting Your Big Data into the Spark Environment Using RDDs

Primarily, this chapter will provide a brief overview of how to get your big data into the Spark environment using resilient distributed datasets (RDDs). We will be using a wide array of tools to interact with and modify this data so that useful insights can be extracted. We will first load the data on Spark RDDs and then carry out parallelization with Spark RDDs.

In this chapter, we will cover the following topics:

• Loading data onto Spark RDDs
• Parallelization with Spark RDDs
• Basics of RDD operation

Loading data on to Spark RDDs

In this section, we are going to look at loading data on to Spark RDDs, and will cover the following topics:

• The UCI machine learning data repository
• Getting data from the repository to Python
• Getting data into Spark

Let's start with an overview of the UCI machine learning data repository.

The UCI machine learning repository

We can access the UCI machine learning repository by navigating to https://archive.ics.uci.edu/ml/. So, what is the UCI machine learning repository? UCI stands for the University of California Irvine machine learning repository, and it is a very useful resource for getting open source and free datasets for machine learning. Although PySpark's main issue or solution...

Parallelization with Spark RDDs

Now that we know how to create RDDs within the text file that we received from the internet, we can look at a different way to create this RDD. Let's discuss parallelization with our Spark RDDs.

In this section, we will cover the following topics:

• What is parallelization?
• How do we parallelize Spark RDDs?

Let's start with parallelization.

What is parallelization?

The best way to understand Spark, or any language, is to look at the documentation. If we look at Spark's documentation, it clearly states that, for the textFile function that we used last time, it reads the text file from HDFS.

On the other hand, if we look at the definition of parallelize, we can see that this is...

Basics of RDD operation

Let's now go through some RDD operational basics. The best way to understand what something does is to look at the documentation so that we can get a rigorous understanding of what a function performs.

The reason why this is very important is that the documentation is the golden source of how a function is defined and what it is designed to be used as. By reading the documentation, we make sure that we are as close to the source as possible in our understanding. The link to the relevant documentation is https://spark.apache.org/docs/latest/rdd-programming-guide.html.

So, let's start with the map function. The map function returns an RDD by applying the f function to each element of this RDD. In other words, it works the same as the map function we see in Python. On the other hand, the filter function returns a new RDD containing only the elements...

Summary

In this chapter, we learned how to load data on Spark RDDs and also covered parallelization with Spark RDDs. We had a brief overview of the UCI machine learning repository before loading the data. We had an overview of the basic RDD operations, and also checked the functions from the official documentation.

In the next chapter, we will cover big data cleaning and data wrangling.