Chapter 4. Scientific Computing and Big Data Analysis with Python and Hadoop

In this chapter, we provide an introduction to Python and analyzing big data using Hadoop and Python packages. We will be looking at a basic Python installation, opening a Jupyter Notebook, and working through some examples.

In a nutshell, the following topics will be covered in this chapter:

• Installation:
  • Downloading and installing Python
  • Downloading and installing Anaconda
  • Installing Jupyter Notebook
• Data analysis

Installation

In this section, we will look at the steps involved in installing and setting up Jupyter Notebook with the Python interpreter to perform data analysis.

Installing standard Python

Go to the Python download page at http://www.python.org/download/ with your web browser. Python is supported on Windows, macOS, and Linux, and you will find the different installations: 

• Python releases for Windows at https://www.python.org/downloads/windows/
• Python releases for macOS X at https://www.python.org/downloads/mac-osx/
• Python source releases (Linux and Unix) at https://www.python.org/downloads/source/

When you click on the download page, you will see the following screen: 

If you click on a specific version, such as 3.6.5, then you will be taken to a different page, as shown in the following screenshot:

You can read the release notes and then proceed to download the Python version by simply scrolling down the page, as shown in the following screenshot:

Click the correct version for your operating...

Data analysis

Download OnlineRetail.csv from the link provided with the book. Then, you can load the file using Pandas.

The following is a simple way of reading a local file using Pandas:

import pandas as pd
path = '/Users/sridharalla/Documents/OnlineRetail.csv'
df = pd.read_csv(path)

However, since we are analyzing data in a Hadoop cluster, we should be using hdfs not a local system. The following is an example of how the hdfs file can be loaded into a pandas DataFrame:

import pandas as pd
from hdfs import InsecureClient
client_hdfs = InsecureClient('http://localhost:9870')
with client_hdfs.read('/user/normal/OnlineRetail.csv', encoding = 'utf-8') as reader:
 df = pd.read_csv(reader,index_col=0)

The following is what the following line of code does:

df.head(3)

You will get the following result:

Basically, it displays the top three entries in the DataFrame.

We can now experiment with the data. Enter the following:

len(df)

That should output this:

65499

That just means the length, or size, of the DataFrame...

Summary

In this chapter, we have discussed Python and how to use Python to perform data analysis with Jupyter Notebook. We also looked at several different operations that can be done using Python.

In the next chapter, we will look at another popular analytical language, R, and how R can be used to perform data analysis.