Manipulating Data with the Pandas Library

In the next few portions of the book, we are going to get our hands dirty by building the various kinds of recommender systems that were introduced in chapter one. However, before we do so, it is important that we know how to handle, manipulate, and analyze data efficiently in Python.

The datasets we'll be working with will be several megabytes in size. Historically, Python has never been well-known for its speed of execution. Therefore, analyzing such huge amounts of data using vanilla Python and the built-in data structures it provides us is simply impossible.

In this chapter, we're going to get ourselves acquainted with the pandas library, which aims to overcome the aforementioned limitations, making data analysis in Python extremely efficient and user-friendly. We'll also introduce ourselves to the Movies Dataset that...

Technical requirements

You will be required to have Python installed on a system. Finally, to use the Git repository of this book, the user needs to install Git.

The code files of this chapter can be found on GitHub:
https://github.com/PacktPublishing/Hands-On-Recommendation-Systems-with-Python.

Check out the following video to see the code in action:

http://bit.ly/2LoZEUj.

Setting up the environment

Before we start coding, we should probably set up our development environment. For data scientists and analysts using Python, the Jupyter Notebook is, by far, the most popular tool for development. Therefore, we strongly advise that you use this environment.

We will also need to download the pandas library. The easiest way to obtain both is to download Anaconda. Anaconda is a distribution that comes with the Jupyter software and the SciPy packages (which includes pandas).

You can download the distribution here: https://www.anaconda.com/download/.

The next step is to create a new folder (I'm going to name it RecoSys) in your desired location. This will be the master folder that contains all the code we write as part of this book. Within this folder, create another folder named Chapter2, which will contain all the code we write as part of this chapter...

The Pandas library

Pandas is a package that gives us access to high-performance, easy-to-use tools and data structures for data analysis in Python.

As we stated in the introduction, Python is a slow language. Pandas overcomes this by implementing heavy optimization using the C programming language. It also gives us access to Series and DataFrame, two extremely powerful and user-friendly data structures imported from the R Statistical Package.

Pandas also makes importing data from external files into the Python environment a breeze. It supports a wide variety of formats, such as JSON, CSV, HDF5, SQL, NPY, and XLSX.

As a first step toward working with pandas, let's import our movies data into our Jupyter Notebook. To do this, we need the path to where our dataset is located. This can be a URL on the internet or your local computer. We highly recommend downloading the data...

The Pandas DataFrame

As we saw in the previous section, the df.head() code outputted a table-like structure. In essence, the DataFrame is just that: a two-dimensional data structure with columns of different data types. You can think of it as an SQL Table. Of course, just being a table of rows and columns isn't what makes the DataFrame special. The DataFrame gives us access to a wide variety of functionality, some of which we're going to explore in this section.

Each row in our DataFrame represents a movie. But how many movies are there? We can find this out by running the following code:

#Output the shape of df
df.shape

OUTPUT:
(45466, 24)

The result gives us the number of rows and columns present in df. We can see that we have data on 45,466 movies.

We also see that we have 24 columns. Each column represents a feature or a piece of metadata about the movie. When we ran...

The Pandas Series

When we accessed the Jumanji movie using .loc and .iloc, the data structures returned to us were Pandas Series objects. You may have also noticed that we were accessing entire columns using df[column_name]. This, too, was a Pandas Series object:

type(small_df['year'])

OUTPUT:
pandas.core.series.Series

The Pandas Series is a one-dimensional labelled array capable of holding data of any type. You may think of it as a Python list on steroids. When we were using the .apply() and .astype() methods in the previous section, we were actually using them on these Series objects.

Therefore, like the DataFrame, the Series object comes with its own group of extremely useful methods that make data analysis a breeze.

First, let's check out the shortest- and longest-running movies of all time. We will do this by accessing the runtime column of the DataFrame as...

Summary

In this chapter, we gained an understanding of the limitations of using vanilla Python and its built-in data structures. We acquainted ourselves with the Pandas library and learned how it overcomes the aforementioned difficulties by giving us access to extremely powerful and easy-to-use data structures. We then explored the two main data structures, Series and DataFrame, by analyzing our movies-metadata dataset.

In the next chapter, we will use our newfound skills to build an IMDB Top 250 Clone and its variant, a type of knowledge-based recommender.