# 1. The Importance of Data Visualization and Data Exploration

Overview

This chapter introduces you to the basics of the statistical analysis of a dataset. We will look at basic operations for calculating the mean, median, and variance of different datasets and use NumPy and pandas to filter, sort, and shape the datasets to our requirements. The concepts we will cover will serve as a base of knowledge for the upcoming visualization chapters, in which we will work with real-world datasets. By the end of this chapter, you will be able to explain the importance of data visualization and calculate basic statistical values (such as median, mean, and variance), and use NumPy and pandas for data wrangling.

# Introduction

Unlike machines, people are usually not equipped for interpreting a large amount of information from a random set of numbers and messages in each piece of data. Out of all our logical capabilities, we understand things best through the visual processing of information. When data is represented visually, the probability of understanding complex builds and numbers increases.

Python has recently emerged as a programming language that performs well for data analysis. It has applications across data science pipelines that convert data into a usable format (such as pandas), analyzes it (such as NumPy), and extract useful conclusions from the data to represent it in a visually appealing manner (such as Matplotlib or Bokeh). Python provides data visualization libraries that can help you assemble graphical representations efficiently.

In this book, you will learn how to use Python in combination with various libraries, such as **NumPy**, **pandas**, **Matplotlib**, **seaborn**, and **geoplotlib**, to create impactful data visualizations using real-world data. Besides that, you will also learn about the features of different types of charts and compare their advantages and disadvantages. This will help you choose the chart type that's suited to visualizing your data.

Once we understand the basics, we can cover more advanced concepts, such as interactive visualizations and how **Bokeh** can be used to create animated visualizations that tell a story. Upon completing this book, you will be able to perform **data wrangling**, extract relevant information, and visualize your findings descriptively.

## Introduction to Data Visualization

Computers and smartphones store data such as names and numbers in a digital format. **Data representation** refers to the form in which you can store, process, and transmit data.

Representations can narrate a story and convey fundamental discoveries to your audience. Without appropriately modeling your information to use it to make meaningful findings, its value is reduced. Creating representations helps us achieve a more precise, more concise, and more direct perspective of information, making it easier for anyone to understand the data.

Information isn't equivalent to data. Representations are a useful apparatus to derive insights from the data. Thus, representations transform data into useful information.

## The Importance of Data Visualization

Instead of just looking at data in the columns of an Excel spreadsheet, we get a better idea of what our data contains by using visualization. For instance, it's easy to see a pattern emerge from the numerical data that's given in the following scatter plot. It shows the correlation between body mass and the maximum longevity of various animals grouped by class. There is a positive correlation between body mass and maximum longevity:

Visualizing data has many advantages, such as the following:

- Complex data can be easily understood.
- A simple visual representation of outliers, target audiences, and futures markets can be created.
- Storytelling can be done using dashboards and animations.
- Data can be explored through interactive visualizations.

## Data Wrangling

**Data wrangling** is the process of transforming raw data into a suitable representation for various tasks. It is the discipline of augmenting, cleaning, filtering, standardizing, and enriching data in a way that allows it to be used in a downstream task, which in our case is data visualization.

Look at the following data wrangling process flow diagram to understand how accurate and actionable data can be obtained for business analysts to work on:

In relation to the preceding figure, the following steps explain the flow of the data wrangling process:

- First, the Employee Engagement data is in its raw form.
- Then, the data gets imported as a DataFrame and is later cleaned.
- The cleaned data is then transformed into graphs, from which findings can be derived.
- Finally, we analyze this data to communicate the final results.

For example, employee engagement can be measured based on raw data gathered from feedback surveys, employee tenure, exit interviews, one-on-one meetings, and so on. This data is cleaned and made into graphs based on parameters such as referrals, faith in leadership, and scope of promotions. The percentages, that is, information derived from the graphs, help us reach our result, which is to determine the measure of employee engagement.

## Tools and Libraries for Visualization

There are several approaches to creating data visualizations. Depending on your requirements, you might want to use a non-coding tool such as **Tableau**, which allows you to get a good feel for your data. Besides Python, which will be used in this book, **MATLAB** and **R** are widely used in data analytics.

However, Python is the most popular language in the industry. Its ease of use and the speed at which you can manipulate and visualize data, combined with the availability of a number of libraries, make Python the best choice for data visualization.

Note

MATLAB (https://www.mathworks.com/products/matlab.html), R (https://www.r-project.org), and Tableau (https://www.tableau.com) are not part of this book; we will only cover the relevant tools and libraries for Python.

# Overview of Statistics

**Statistics** is a combination of the analysis, collection, interpretation, and representation of numerical data. **Probability** is a measure of the likelihood that an event will occur and is quantified as a number between 0 and 1.

A **probability distribution** is a function that provides the probability for every possible event. A probability distribution is frequently used for statistical analysis. The higher the probability, the more likely the event. There are two types of probability distributions, namely discrete and continuous.

A **discrete probability distribution** shows all the values that a random variable can take, together with their probability. The following diagram illustrates an example of a discrete probability distribution. If we have a six-sided die, we can roll each number between 1 and 6. We have six events that can occur based on the number that's rolled. There is an equal probability of rolling any of the numbers, and the individual probability of any of the six events occurring is 1/6:

A **continuous probability distribution** defines the probabilities of each possible value of a continuous random variable. The following diagram provides an example of a continuous probability distribution. This example illustrates the distribution of the time needed to drive home. In most cases, around 60 minutes is needed, but sometimes, less time is needed because there is no traffic, and sometimes, much more time is needed if there are traffic jams:

## Measures of Central Tendency

Measures of central tendency are often called **averages** and describe central or typical values for a probability distribution. We are going to discuss three kinds of averages in this chapter:

**Mean**: The arithmetic average is computed by summing up all measurements and dividing the sum by the number of observations. The mean is calculated as follows:

**Median**: This is the middle value of the ordered dataset. If there is an even number of observations, the median will be the average of the two middle values. The median is less prone to outliers compared to the mean, where outliers are distinct values in data.**Mode**: Our last measure of central tendency, the mode is defined as the most frequent value. There may be more than one mode in cases where multiple values are equally frequent.

For example, a die was rolled 10 times, and we got the following numbers: 4, 5, 4, 3, 4, 2, 1, 1, 2, and 1.

The mean is calculated by summing all the events and dividing them by the number of observations: *(4+5+4+3+4+2+1+1+2+1)/10=2.7*.

To calculate the median, the die rolls have to be ordered according to their values. The ordered values are as follows: 1, 1, 1, 2, 2, 3, 4, 4, 4, 5. Since we have an even number of die rolls, we need to take the average of the two middle values. The average of the two middle values is *(2+3)/2=2.5*.

The modes are 1 and 4 since they are the two most frequent events.

## Measures of Dispersion

**Dispersion**, also called **variability**, is the extent to which a probability distribution is stretched or squeezed.

The different measures of dispersion are as follows:

**Variance**: The variance is the expected value of the squared deviation from the mean. It describes how far a set of numbers is spread out from their mean. Variance is calculated as follows:

**Standard deviation**: This is the square root of the variance.**Range**: This is the difference between the largest and smallest values in a dataset.**Interquartile range**: Also called the**midspread**or**middle 50%**, this is the difference between the 75th and 25th percentiles, or between the upper and lower quartiles.

## Correlation

The measures we have discussed so far only considered single variables. In contrast, **correlation** describes the statistical relationship between two variables:

- In a positive correlation, both variables move in the same direction.
- In a negative correlation, the variables move in opposite directions.
- In zero correlation, the variables are not related.
Note

One thing you should be aware of is that correlation does not imply causation. Correlation describes the relationship between two or more variables, while causation describes how one event is caused by another. For example, consider a scenario in which ice cream sales are correlated with the number of drowning deaths. But that doesn't mean that ice cream consumption causes drowning. There could be a third variable, say temperature, that may be responsible for this correlation. Higher temperatures may cause an increase in both ice cream sales and more people engaging in swimming, which may be the real reason for the increase in deaths due to drowning.

**Example**

Consider you want to find a decent apartment to rent that is not too expensive compared to other apartments you've found. The other apartments (all belonging to the same locality) you found on a website are priced as follows: $700, $850, $1,500, and $750 per month. Let's calculate some values statistical measures to help us make a decision:

- The mean is
*($700 + $850 + $1,500 + $750) / 4 = $950*. - The median is
*($750 + $850) / 2 = $800*. - The standard deviation is .
- The range is
*$1,500 - $700 = $800*.

As an exercise, you can try and calculate the variance as well. However, note that compared with all the above values, the median value ($800) is a better statistical measure in this case since it is less prone to outliers (the rent amount of $1,500). Given that all apartments belong to the same locality, you can clearly see that the apartment costing $1500 is definitely priced much higher as compared with other apartments. A simple statistical analysis helped us to narrow down our choices.

## Types of Data

It is important to understand what kind of data you are dealing with so that you can select both the right statistical measure and the right visualization. We categorize data as categorical/qualitative and numerical/quantitative. Categorical data describes characteristics, for example, the color of an object or a person's gender. We can further divide categorical data into nominal and ordinal data. In contrast to nominal data, ordinal data has an order.

Numerical data can be divided into discrete and continuous data. We speak of discrete data if the data can only have certain values, whereas continuous data can take any value (sometimes limited to a range).

Another aspect to consider is whether the data has a temporal domain – in other words, is it bound to time or does it change over time? If the data is bound to a location, it might be interesting to show the spatial relationship, so you should keep that in mind as well. The following flowchart classifies the various data types:

## Summary Statistics

In real-world applications, we often encounter enormous datasets. Therefore, **summary statistics** are used to summarize important aspects of data. They are necessary to communicate large amounts of information in a compact and simple way.

We have already covered measures of central tendency and dispersion, which are both summary statistics. It is important to know that measures of central tendency show a center point in a set of data values, whereas measures of dispersion show how much the data varies.

The following table gives an overview of which measure of central tendency is best suited to a particular type of data:

In the next section, we will learn about the NumPy library and implement a few exercises using it.

# NumPy

When handling data, we often need a way to work with multidimensional arrays. As we discussed previously, we also have to apply some basic mathematical and statistical operations on that data. This is exactly where **NumPy** positions itself. It provides support for large n-dimensional arrays and has built-in support for many high-level mathematical and statistical operations.

Note

Before NumPy, there was a library called **Numeric**. However, it's no longer used, because NumPy's signature **ndarray** allows the performant handling of large and high-dimensional matrices.

Ndarrays are the essence of NumPy. They are what makes it faster than using Python's built-in lists. Other than the built-in list data type, ndarrays provide a stridden view of memory (for example, `int[]`

in Java). Since they are homogeneously typed, meaning all the elements must be of the same type, the stride is consistent, which results in less memory wastage and better access times.

The **stride** is the number of locations between the beginnings of two adjacent elements in an array. They are normally measured in bytes or in units of the size of the array elements. A stride can be larger or equal to the size of the element, but not smaller; otherwise, it would intersect the memory location of the next element.

Note

Remember that NumPy arrays have a defined data type. This means you are not able to insert strings into an integer type array. NumPy is mostly used with double-precision data types.

The following are some of the built-in methods that we will use in the exercises and activities of this chapter.

**mean**

NumPy provides implementations of all the mathematical operations we covered in the *Overview of Statistics* section of this chapter. The mean, or average, is the one we will look at in more detail in the upcoming exercise:

Note

The `#`

symbol in the code snippet below denotes a code comment. Comments are added into code to help explain specific bits of logic.

# mean value for the whole dataset np.mean(dataset) # mean value of the first row np.mean(dataset[0]) # mean value of the whole first column np.mean(dataset[:, 0] # mean value of the first 10 elements of the second row np.mean(dataset[1, 0:10])

**median**

Several of the mathematical operations have the same interface. This makes them easy to interchange if necessary. The `median`

, `var`

, and `std`

methods will be used in the upcoming exercises and activities:

# median value for the whole dataset np.median(dataset) # median value of the last row using reverse indexing np.median(dataset[-1]) # median value of values of rows >5 in the first column np.median(dataset[5:, 0])

Note that we can index every element from the end of our dataset as we can from the front by using **reverse indexing**. It's a simple way to get the last or several of the last elements of a list. Instead of `[0]`

for the first/last element, it starts with `dataset[-1]`

and then decreases until `dataset[-len(dataset)]`

, which is the first element in the dataset.

**var**

As we mentioned in the *Overview of Statistics* section, the variance describes how far a set of numbers is spread out from their mean. We can calculate the variance using the `var`

method of NumPy:

# variance value for the whole dataset np.var(dataset) # axis used to get variance per column np.var(dataset, axis=0) # axis used to get variance per row np.var(dataset, axis=1)

**std**

One of the advantages of the standard deviation is that it remains in the scalar system of the data. This means that the unit of the deviation will have the same unit as the data itself. The `std`

method works just like the others:

# standard deviation for the whole dataset np.std(dataset) # std value of values from the 2 first rows and columns np.std(dataset[:2, :2]) # axis used to get standard deviation per row np.std(dataset, axis=1)

Now we will do an exercise to load a dataset and calculate the mean using these methods.

Note

All the exercises and activities in this chapter will be developed in Jupyter Notebooks. Please download the GitHub repository with all the prepared templates from https://packt.live/31USkof. Make sure you have installed all the libraries as mentioned in the preface.

## Exercise 1.01: Loading a Sample Dataset and Calculating the Mean Using NumPy

In this exercise, we will be loading the `normal_distribution.csv`

dataset and calculating the mean of each row and each column in it:

- Using the Anaconda Navigator launch either Jupyter Labs or Jupyter Notebook. In the directory of your choice, create a
`Chapter01/Exercise1.01`

folder. - Create a new Jupyter Notebook and save it as
`Exercise1.01.ipynb`

in the`Chapter01/Exercise1.01`

folder. - Now, begin writing the code for this exercise as shown in the steps below. We begin with the import statements. Import
`numpy`

with an alias:import numpy as np

- Use the
`genfromtxt`

method of NumPy to load the dataset:Note

The code snippet shown here uses a backslash (

`\`

) to split the logic across multiple lines. When the code is executed, Python will ignore the backslash, and treat the code on the next line as a direct continuation of the current line.dataset = \ np.genfromtxt('../../Datasets/normal_distribution.csv', \ delimiter=',')

Note

In the preceding snippet, and for the rest of the book, we will be using a relative path to load the datasets. However, for the preceding code to work as intended, you need to follow the folder arrangement as present in this link: https://packt.live/3ftUu3P. Alternatively, you can also use the absolute path; for example,

`dataset = np.genfromtxt('C:/Datasets/normal_distribution.csv', delimiter=',')`

. If your Jupyter Notebook is saved in the same folder as the dataset, then you can simply use the filename:`dataset = np.genfromtxt('normal_distribution.csv', delimiter=',')`

The

`genfromtxt`

method helps load the data from a given text or`.csv`

file. If everything works as expected, the generation should run through without any error or output.Note

The

`numpy.genfromtext`

method is less efficient than the`pandas.read_csv`

method. We shall refrain from going into the details of why this is the case as this explanation is beyond the scope of this text. - Check the data you just imported by simply writing the name of the ndarray in the next cell. Simply executing a cell that returns a value, such as an ndarray, will use Jupyter formatting, which looks nice and, in most cases, displays more information than using
`print`

:# looking at the dataset dataset

A section of the output resulting from the preceding code is as follows:

- Print the shape using the
`dataset.shape`

command to get a quick overview of our dataset. This will give us output in the form (rows, columns):dataset.shape

We can also call the rows as instances and the columns as features. This means that our dataset has 24 instances and 8 features. The output of the preceding code is as follows:

(24,8)

- Calculate the mean after loading and checking our dataset. The first row in a NumPy array can be accessed by simply indexing it with zero; for example,
`dataset[0]`

. As we mentioned previously, NumPy has some built-in functions for calculations such as the mean. Call`np.mean()`

and pass in the dataset's first row to get the result:# calculating the mean for the first row np.mean(dataset[0])

The output of the preceding code is as follows:

100.177647525

- Now, calculate the mean of the first column by using
`np.mean()`

in combination with the column indexing`dataset[:, 0]`

:np.mean(dataset[:, 0])

The output of the preceding code is as follows:

99.76743510416668

Whenever we want to define a range to select from a dataset, we can use a colon,

`:`

, to provide start and end values for the to be selected range. If we don't provide start and end values, the default of*0*to*n*is used, where n is the length of the current axis. - Calculate the mean for every single row, aggregated in a list, using the
`axis`

tools of NumPy. Note that by simply passing the`axis`

parameter in the`np.mean()`

call, we can define the dimension our data will be aggregated on.`axis=0`

is horizontal and`axis=1`

is vertical. Get the result for each row by using`axis=1`

:np.mean(dataset, axis=1)

The output of the preceding code is as follows:

Get the mean of each column by using

`axis=0`

:np.mean(dataset, axis=0)

The output of the preceding code is as follows:

- Calculate the mean of the whole matrix by summing all the values we retrieved in the previous steps:
np.mean(dataset)

The output of the preceding code is as follows:

100.16536917390624

Note

To access the source code for this specific section, please refer to https://packt.live/30IkAMp.

You can also run this example online at https://packt.live/2Y4yHK1.

You are already one step closer to using NumPy in combination with plotting libraries and creating impactful visualizations. Since we've now covered the very basics and calculated the mean, it's now up to you to solve the upcoming activity.

## Activity 1.01: Using NumPy to Compute the Mean, Median, Variance, and Standard Deviation of a Dataset

In this activity, we will use the skills we've learned to import datasets and perform some basic calculations (mean, median, variance, and standard deviation) to compute our tasks.

Perform the following steps to implement this activity:

- Open the
`Activity1.01.ipynb`

Jupyter Notebook from the`Chapter01`

folder to do this activity. Import NumPy and give it the alias`np`

. - Load the
`normal_distribution.csv`

dataset by using the`genfromtxt`

method from NumPy. - Print a subset of the first two rows of the dataset.
- Load the dataset and calculate the mean of the third row. Access the third row by using index 2,
`dataset[2]`

. - Index the last element of an ndarray in the same way a regular Python list can be accessed.
`dataset[:, -1]`

will give us the last column of every row. - Get a submatrix of the first three elements of every row of the first three columns by using the double-indexing mechanism of NumPy.
- Calculate the median for the last row of the dataset.
- Use reverse indexing to define a range to get the last three columns. We can use
`dataset[:, -3:]`

here. - Aggregate the values along an
`axis`

to calculate the rows. We can use`axis=1`

here. - Calculate the variance for each column using
`axis 0`

. - Calculate the variance of the intersection of the last two rows and the first two columns.
- Calculate the standard deviation for the dataset.
Note

The solution for this activity can be found via this link.

You have now completed your first activity using NumPy. In the following activities, this knowledge will be consolidated further.

## Basic NumPy Operations

In this section, we will learn about basic NumPy operations such as indexing, slicing, splitting, and iterating and implement them in an exercise.

### Indexing

Indexing elements in a NumPy array, at a high level, works the same as with built-in Python lists. Therefore, we can index elements in multi-dimensional matrices:

# index single element in outermost dimension dataset[0] # index in reversed order in outermost dimension dataset[-1] # index single element in two-dimensional data dataset[1, 1] # index in reversed order in two-dimensional data dataset[-1, -1]

### Slicing

Slicing has also been adapted from Python's lists. Being able to easily slice parts of lists into new ndarrays is very helpful when handling large amounts of data:

# rows 1 and 2 dataset[1:3] # 2x2 subset of the data dataset[:2, :2] # last row with elements reversed dataset[-1, ::-1] # last 4 rows, every other element up to index 6 dataset[-5:-1, :6:2]

### Splitting

Splitting data can be helpful in many situations, from plotting only half of your time-series data to separating test and training data for machine learning algorithms.

There are two ways of splitting your data, horizontally and vertically. Horizontal splitting can be done with the `hsplit`

method. Vertical splitting can be done with the `vsplit`

method:

# split horizontally in 3 equal lists np.hsplit(dataset, (3)) # split vertically in 2 equal lists np.vsplit(dataset, (2))

### Iterating

Iterating the NumPy data structures, ndarrays, is also possible. It steps over the whole list of data one after another, visiting every single element in the ndarray once. Considering that they can have several dimensions, indexing gets very complex.

The `nditer`

is a multi-dimensional iterator object that iterates over a given number of arrays:

# iterating over whole dataset (each value in each row) for x in np.nditer(dataset): print(x)

The `ndenumerate`

will give us exactly this index, thus returning (`0`

, `1`

) for the second value in the first row:

Note

The triple-quotes ( `"""`

) shown in the code snippet below are used to denote the start and end points of a multi-line code comment. Comments are added into code to help explain specific bits of logic.

""" iterating over the whole dataset with indices matching the position in the dataset """ for index, value in np.ndenumerate(dataset): print(index, value)

Now, we will perform an exercise using these basic NumPy operations.

## Exercise 1.02: Indexing, Slicing, Splitting, and Iterating

In this exercise, we will use the features of NumPy to index, slice, split, and iterate ndarrays to consolidate what we've learned. A client has provided us with a dataset, `normal_distribution_splittable.csv`

, wants us to confirm that the values in the dataset are closely distributed around the mean value of 100.

Note

You can obviously plot a distribution and show the spread of data, but here we want to practice implementing the aforementioned operations using the NumPy library.

Let's use the features of NumPy to index, slice, split, and iterate ndarrays.

**Indexing**

- Create a new Jupyter Notebook and save it as
`Exercise1.02.ipynb`

in the`Chapter01/Exercise1.02`

folder. - Import the necessary libraries:
import numpy as np

- Load the
`normal_distribution.csv`

dataset using NumPy. Have a look at the ndarray to verify that everything works:dataset = np.genfromtxt('../../Datasets/'\ 'normal_distribution_splittable.csv', \ delimiter=',')

Note

As mentioned in the previous exercise, here too we have used a relative path to load the dataset. You can change the path depending on where you have saved the Jupyter Notebook and the dataset.

Remember that we need to show that our dataset is closely distributed around a mean of 100; that is, whatever value we wish to show/calculate should be around 100. For this purpose, first we will calculate the mean of the values of the second and the last row.

- Use simple indexing for the second row, as we did in our first exercise. For a clearer understanding, all the elements of the second row are saved to a variable and then we calculate the mean of these elements:
second_row = dataset[1] np.mean(second_row)

The output of the preceding code is as follows:

96.90038836444445

- Now, reverse index the last row and calculate the mean of that row. Always remember that providing a negative number as the index value will index the list from the end:
last_row = dataset[-1] np.mean(last_row)

The output of the preceding code is as follows:

100.18096645222221

From the outputs obtained in

*step 4*and*5*, we can say that these values indeed are close to 100. To further convince our client, we will access the first value of the first row and the last value of the second last row. - Index the first value of the first row using the Python standard syntax of [0][0]:
first_val_first_row = dataset[0][0] np.mean(first_val_first_row)

The output of the preceding code is as follows:

99.14931546

- Use reverse indexing to access the last value of the second last row (we want to use the combined access syntax here). Remember that
`-1`

means the last element:last_val_second_last_row = dataset[-2, -1] np.mean(last_val_second_last_row)

The output of the preceding code is as follows:

101.2226037

Note

For

*steps 6*and*7*, even if you had not used`np.mean()`

, you would have got the same values as presently shown. This is because the mean of a single value will be the value itself. You can try the above steps with the following code:`first_val_first_row = dataset[0][0]`

`first_val_first_row`

`last_val_second_last_row = dataset[-2, -1]`

`last_val_second_last_row`

From all the preceding outputs, we can confidently say that the values we obtained hover around a mean of 100. Next, we'll use slicing, splitting, and iterating to achieve our goal.

**Slicing**

- Create a 2x2 matrix that starts at the second row and second column using
`[1:3, 1:3]`

:""" slicing an intersection of 4 elements (2x2) of the first two rows and first two columns """ subsection_2x2 = dataset[1:3, 1:3] np.mean(subsection_2x2)

The output of the preceding code is as follows:

95.63393608250001

- In this task, we want to have every other element of the fifth row. Provide indexing of
`::2`

as our second element to get every second element of the given row:every_other_elem = dataset[4, ::2] np.mean(every_other_elem)

The output of the preceding code is as follows:

98.35235805800001

Introducing the second column into the indexing allows us to add another layer of complexity. The third value allows us to only select certain values (such as every other element) by providing a value of 2. This means it skips the values between and only takes each second element from the used list.

- Reverse the elements in a slice using negative numbers:
reversed_last_row = dataset[-1, ::-1] np.mean(reversed_last_row)

The output of the preceding code is as follows:

100.18096645222222

**Splitting**

- Use the
`hsplit`

method to split our dataset into three equal parts:hor_splits = np.hsplit(dataset,(3))

Note that if the dataset can't be split with the given number of slices, it will throw an error.

- Split the first third into two equal parts vertically. Use the
`vsplit`

method to vertically split the dataset in half. It works like`hsplit`

:ver_splits = np.vsplit(hor_splits[0],(2))

- Compare the shapes. We can see that the subset has the required half of the rows and the third half of the columns:
print("Dataset", dataset.shape) print("Subset", ver_splits[0].shape)

The output of the preceding code is as follows:

Dataset (24, 9) Subset (12, 3)

**Iterating**

- Iterate over the whole dataset (each value in each row):
curr_index = 0 for x in np.nditer(dataset): print(x, curr_index) curr_index += 1

The output of the preceding code is as follows:

Looking at the given piece of code, we can see that the index is simply incremented with each element. This only works with one-dimensional data. If we want to index multi-dimensional data, this won't work.

- Use the
`ndenumerate`

method to iterate over the whole dataset. It provides two positional values,`index`

and`value`

:for index, value in np.ndenumerate(dataset): print(index, value)

The output of the preceding code is as follows:

Notice that all the output values we obtained are close to our mean value of 100. Thus, we have successfully managed to convince our client using several NumPy methods that our data is closely distributed around the mean value of 100.

Note

To access the source code for this specific section, please refer to https://packt.live/2Neteuh.

You can also run this example online at https://packt.live/3e7K0qq.

We've already covered most of the basic data wrangling methods for NumPy. In the next section, we'll take a look at more advanced features that will give you the tools you need to get better at analyzing your data.

## Advanced NumPy Operations

In this section, we will learn about advanced NumPy operations such as filtering, sorting, combining, and reshaping and then implement them in an exercise.

### Filtering

Filtering is a very powerful tool that can be used to clean up your data if you want to avoid outlier values.

In addition to the `dataset[dataset > 10]`

shorthand notation, we can use the built-in NumPy `extract`

method, which does the same thing using a different notation, but gives us greater control with more complex examples.

If we only want to extract the indices of the values that match our given condition, we can use the built-in `where`

method. For example, `np.where(dataset > 5)`

will return a list of indices of the values from the initial dataset that is bigger than 5:

# values bigger than 10 dataset[dataset > 10] # alternative – values smaller than 3 np.extract((dataset < 3), dataset) # values bigger 5 and smaller 10 dataset[(dataset > 5) & (dataset < 10)] # indices of values bigger than 5 (rows and cols) np.where(dataset > 5)

### Sorting

Sorting each row of a dataset can be really useful. Using NumPy, we are also able to sort on other dimensions, such as columns.

In addition, `argsort`

gives us the possibility to get a list of indices, which would result in a sorted list:

# values sorted on last axis np.sort(dataset) # values sorted on axis 0 np.sort(dataset, axis=0) # indices of values in sorted list np.argsort(dataset)

### Combining

Stacking rows and columns onto an existing dataset can be helpful when you have two datasets of the same dimension saved to different files.

Given two datasets, we use **vstack** to stack `dataset_1`

on top of `dataset_2`

, which will give us a combined dataset with all the rows from `dataset_1`

, followed by all the rows from `dataset_2`

.

If we use **hstack**, we stack our datasets "next to each other," meaning that the elements from the first row of `dataset_1`

will be followed by the elements of the first row of `dataset_2`

. This will be applied to each row:

# combine datasets vertically np.vstack([dataset_1, dataset_2]) # combine datasets horizontally np.hstack([dataset_1, dataset_2]) # combine datasets on axis 0 np.stack([dataset_1, dataset_2], axis=0)

### Reshaping

Reshaping can be crucial for some algorithms. Depending on the nature of your data, it might help you to reduce dimensionality to make visualization easier:

# reshape dataset to two columns x rows dataset.reshape(-1, 2) # reshape dataset to one row x columns np.reshape(dataset, (1, -1))

Here, `-1`

is an unknown dimension that NumPy identifies automatically. NumPy will figure out the length of any given array and the remaining dimensions and will thus make sure that it satisfies the required standard.

Next, we will perform an exercise using advanced NumPy operations.

## Exercise 1.03: Filtering, Sorting, Combining, and Reshaping

This final exercise for NumPy provides some more complex tasks to consolidate our learning. It will also combine most of the previously learned methods as a recap. We'll use the filtering features of NumPy for sorting, stacking, combining, and reshaping our data:

- Create a new Jupyter Notebook and save it as
`Exercise1.03.ipynb`

in the`Chapter01/Exercise1.03`

folder. - Import the necessary libraries:
import numpy as np

- Load the
`normal_distribution_splittable.csv`

dataset using NumPy. Make sure that everything works by having a look at the ndarray:dataset = np.genfromtxt('../../Datasets/'\ 'normal_distribution_splittable.csv', \ delimiter=',') dataset

- You will obtain the following output:

Note

For ease of presentation, we have shown only a part of the output.

**Filtering**

- Get values greater than 105 by supplying the condition
`> 105`

in the brackets:vals_greater_five = dataset[dataset > 105] vals_greater_five

You will obtain the following output:

You can see in the preceding figure that all the values in the output are greater than 105.

- Extract the values of our dataset that are between the values 90 and 95. To use more complex conditions, we might want to use the
`extract`

method of NumPy:vals_between_90_95 = np.extract((dataset > 90) \ & (dataset < 95), dataset) vals_between_90_95

You will obtain the following output:

The preceding output clearly shows that only values lying between 90 and 95 are printed.

- Use the
`where`

method to get the indices of values that have a delta of less than 1; that is,*[individual value] – 100*should be less than 1. Use those indices (`row, col)`

in a list comprehension and print them out:rows, cols = np.where(abs(dataset - 100) < 1) one_away_indices = [[rows[index], \ cols[index]] for (index, _) \ in np.ndenumerate(rows)] one_away_indices

The

`where`

method from NumPy allows us to get`indices (rows, cols)`

for each of the matching values.Observe the following truncated output:

Let us confirm if we indeed obtained the right indices. The first set of indices `0,0`

refer to the very first value in the output shown in *Figure 1.14*. Indeed, this is the correct value as `abs (99.14931546 – 100) < 1`

. We can quickly check this for a couple of more values and conclude that indeed the code has worked as intended.

Note

**List comprehensions** are Python's way of mapping over data. They're a handy notation for creating a new list with some operation applied to every element of the old list.

For example, if we want to double the value of every element in this list, `list = [1, 2, 3, 4, 5]`

, we would use list comprehensions like this: `doubled_list=[x*x for x in list]`

. This gives us the following list: `[1, 4, 9, 16, 25]`

. To get a better understanding of list comprehensions, please visit https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions.

**Sorting**

- Sort each row in our dataset by using the
`sort`

method:row_sorted = np.sort(dataset) row_sorted

As described before, by default, the

**last axis**will be used. In a two-dimensional dataset, this is axis 1 which represents the rows. So, we can omit the`axis=1`

argument in the`np.sort`

method call. You will obtain the following output:Compare the preceding output with that in

*Figure 1.14*. What do you observe? The values along the rows have been sorted in an ascending order as expected. - With multi-dimensional data, we can use the
`axis`

parameter to define which dataset should be sorted. Use the`0`

axes to sort the values by column:col_sorted = np.sort(dataset, axis=0) col_sorted

A truncated version of the output is as follows:

As expected, the values along the columns (91.37294597, 91.02628776, 94.11176915, and so on) are now sorted in an ascending order.

- Create a sorted index list and use fancy indexing to get access to sorted elements easily. To keep the order of our dataset and obtain only the values of a sorted dataset, we will use
`argsort`

:index_sorted = np.argsort(dataset[0]) dataset[0][index_sorted]

The output is as follows:

As can be seen from the preceding output, we have obtained the first row with sorted values.

**Combining** - Use the combining features to add the second half of the first column back together, add the second column to our combined dataset, and add the third column to our combined dataset.
thirds = np.hsplit(dataset, (3)) halfed_first = np.vsplit(thirds[0], (2)) halfed_first[0]

The output of the preceding code is as follows:

- Use
`vstack`

to vertically combine the`halfed_first`

datasets:first_col = np.vstack([halfed_first[0], halfed_first[1]]) first_col

The output of the preceding code is as follows:

After stacking the second half of our split dataset, we have one-third of our initial dataset stacked together again. Now, we want to add the other two remaining datasets to our

`first_col`

dataset. - Use the
`hstack`

method to combine our already combined`first_col`

with the second of the three split datasets:first_second_col = np.hstack([first_col, thirds[1]]) first_second_col

A truncated version of the output resulting from the preceding code is as follows:

- Use
`hstack`

to combine the last one-third column with our dataset. This is the same thing we did with our second-third column in the previous step:full_data = np.hstack([first_second_col, thirds[2]]) full_data

A truncated version of the output resulting from the preceding code is as follows:

**Reshaping**

- Reshape our dataset into a single list using the
`reshape`

method:single_list = np.reshape(dataset, (1, -1)) single_list

A truncated version of the output resulting from the preceding code is as follows:

- Provide a
`-1`

for the dimension. This tells NumPy to figure the dimension out itself:# reshaping to a matrix with two columns two_col_dataset = dataset.reshape(-1, 2) two_col_dataset

A truncated version of the output resulting from the preceding code is as follows:

Note

To access the source code for this specific section, please refer to https://packt.live/2YD4AZn.

You can also run this example online at https://packt.live/3e6F7Ol.

You have now used many of the basic operations that are needed so that you can analyze a dataset. Next, we will be learning about pandas, which will provide several advantages when working with data that is more complex than simple multi-dimensional numerical data. pandas also support different data types in datasets, meaning that we can have columns that hold strings and others that have numbers.

NumPy, as you've seen, has some powerful tools. Some of them are even more powerful when combined with pandas DataFrames.

# pandas

The **pandas** Python library provides data structures and methods for manipulating different types of data, such as numerical and temporal data. These operations are easy to use and highly optimized for performance.

Data formats, such as **CSV** and **JSON**, and databases can be used to create **DataFrames**. DataFrames are the internal representations of data and are very similar to tables but are more powerful since they allow you to efficiently apply operations such as multiplications, aggregations, and even joins. Importing and reading both files and in-memory data is abstracted into a user-friendly interface. When it comes to handling missing data, pandas provide built-in solutions to clean up and augment your data, meaning it fills in missing values with reasonable values.

Integrated indexing and label-based slicing in combination with fancy indexing (what we already saw with NumPy) make handling data simple. More complex techniques, such as **reshaping**, **pivoting,** and **melting** data, together with the possibility of easily **joining** and **merging** data, provide powerful tooling so that you can handle your data correctly.

If you're working with **time-series data**, operations such as **date range generation**, **frequency conversion**, and **moving window statistics** can provide an advanced interface for data wrangling.

Note

The installation instructions for pandas can be found here: https://pandas.pydata.org/. The latest version is v0.25.3 (used in this book); however, every v0.25.x should be suitable.

## Advantages of pandas over NumPy

The following are some of the advantages of pandas:

**High level of abstraction**: pandas have a higher abstraction level than NumPy, which gives it a simpler interface for users to interact with. It abstracts away some of the more complex concepts, such as high-performance matrix multiplications and joining tables, and makes it easier to use and understand.**Less intuition**: Many methods, such as joining, selecting, and loading files, are used without much intuition and without taking away much of the powerful nature of pandas.**Faster processing**: The internal representation of DataFrames allows faster processing for some operations. Of course, this always depends on the data and its structure.**Easy DataFrame design**: DataFrames are designed for operations with and on large datasets.

## Disadvantages of pandas

The following are some of the disadvantages of pandas:

**Less applicable**: Due to its higher abstraction, it's generally less applicable than NumPy. Especially when used outside of its scope, operations can get complex.**More disk space**: Due to the internal representation of DataFrames and the way pandas trades disk space for a more performant execution, the memory usage of complex operations can spike.**Performance problems**: Especially when doing heavy joins, which is not recommended, memory usage can get critical and might lead to performance problems.**Hidden complexity**: Less experienced users often tend to overuse methods and execute them several times instead of reusing what they've already calculated. This hidden complexity makes users think that the operations themselves are simple, which is not the case.Note

Always try to think about how to design your workflows instead of excessively using operations.

Now, we will do an exercise to load a dataset and calculate the mean using pandas.

## Exercise 1.04 Loading a Sample Dataset and Calculating the Mean using Pandas

In this exercise, we will be loading the `world_population.csv`

dataset and calculating the mean of some rows and columns. Our dataset holds the yearly population density for every country. Let's use pandas to perform this exercise:

- Create a new Jupyter Notebook and save it as
`Exercise1.04.ipynb`

in the`Chapter01/Exercise1.04`

folder. - Import the
`pandas`

libraries:import pandas as pd

- Use the
`read_csv`

method to load the aforementioned dataset. We want to use the first column, containing the country names, as our index. We will use the`index_col`

parameter for that:dataset = \ pd.read_csv('../../Datasets/world_population.csv', \ index_col=0)

- Now, check the data you just imported by simply writing the name of the dataset in the next cell. pandas uses a data structure called DataFrames. Print some of the rows. To avoid filling the screen, use the pandas
`head()`

method:dataset.head()

The output of the preceding code is as follows:

Both

`head()`

and`tail()`

let you provide a number,`n`

, as a parameter, which describes how many rows should be returned.Note

Simply executing a cell that returns a value such as a DataFrame will use Jupyter formatting, which looks nicer and, in most cases, displays more information than using

`print`

. - Print out the shape of the dataset to get a quick overview using the
`dataset.shape`

command. This works the same as it does with NumPy ndarrays. It will give us the output in the form`(rows, columns)`

:dataset.shape

The output of the preceding code is as follows:

(264, 60)

- Index the column with the year 1961. pandas DataFrames have built-in functions for calculations, such as the
`mean`

. This means we can simply call`dataset.mean()`

to get the result.The printed output should look as follows:

dataset["1961"].mean()

The output of the preceding code is as follows:

176.91514132840555

- Check the difference in population density over the years by repeating the previous step with the column for the year 2015 (the population more than doubled in the given time range):
# calculating the mean for 2015 column dataset["2015"].mean()

The output of the preceding code is as follows:

368.70660104001837

- To get the mean for every single country (row), we can make use of pandas
`axis`

tools. Use the`mean()`

method on the dataset on`axis=1`

, meaning all the rows, and return the first 10 rows using the`head()`

method:dataset.mean(axis=1).head(10)

The output of the preceding code is as follows:

- Get the mean for each column and return the last 10 entries:
dataset.mean(axis=0).tail(10)

The output of the preceding code is as follows:

- Calculate the mean of the whole DataFrame:
# calculating the mean for the whole matrix dataset.mean()

The output of the preceding code is as follows:

Since pandas DataFrames can have different data types in each column, aggregating this value on the whole dataset out of the box makes no sense. By default, `axis=0`

will be used, which means that this will give us the same result as the cell prior to this.

Note

To access the source code for this specific section, please refer to https://packt.live/37z3Us1.

You can also run this example online at https://packt.live/2Bb0ks8.

We've now seen that the interface of pandas has some similar methods to NumPy, which makes it really easy to understand. We have now covered the very basics, which will help you solve the first exercise using pandas. In the following exercise, you will consolidate your basic knowledge of pandas and use the methods you just learned to solve several computational tasks.

## Exercise 1.05: Using pandas to Compute the Mean, Median, and Variance of a Dataset

In this exercise, we will take the previously learned skills of importing datasets and basic calculations and apply them to solve the tasks of our first exercise using pandas.

Let's use pandas features such as `mean`

, `median`

, and `variance`

to make some calculations on our data:

- Create a new Jupyter Notebook and save it as
`Exercise1.05.ipynb`

in the`Chapter01/Exercise1.05`

folder. - Import the necessary libraries:
import pandas as pd

- Use the
`read_csv`

method to load the aforementioned dataset and use the`index_col`

parameter to define the first column as our index:dataset = \ pd.read_csv('../../Datasets/world_population.csv', \ index_col=0)

- Print the first two rows of our dataset:
dataset[0:2]

The output of the preceding code is as follows:

- Now, index the third row by using
`dataset.iloc[[2]]`

. Use the`axis`

parameter to get the mean of the country rather than the yearly column:dataset.iloc[[2]].mean(axis=1)

The output of the preceding code is as follows:

- Index the last element of the DataFrame using
`-1`

as the index for the`iloc()`

method:dataset.iloc[[-1]].mean(axis=1)

The output of the preceding code is as follows:

- Calculate the mean value of the values labeled as Germany using
`loc`

, which works based on the index column:dataset.loc[["Germany"]].mean(axis=1)

The output of the preceding code is as follows:

- Calculate the median value of the last row by using reverse indexing and
`axis=1`

to aggregate the values in the row:dataset.iloc[[-1]].median(axis=1)

The output of the preceding code is as follows:

- Use reverse indexing to get the last three columns with
`dataset[-3:]`

and calculate the median for each of them:dataset[-3:].median(axis=1)

The output of the preceding code is as follows:

- Calculate the median population density values for the first 10 countries of the list using the
`head`

and`median`

methods:dataset.head(10).median(axis=1)

The output of the preceding code is as follows:

When handling larger datasets, the order in which methods are executed matters. Think about what

`head(10)`

does for a moment. It simply takes your dataset and returns the first 10 rows in it, cutting down your input to the`mean()`

method drastically.The last method we'll cover here is the variance. pandas provide a consistent API, which makes it easy to use.

- Calculate the variance of the dataset and return only the last five columns:
dataset.var().tail()

The output of the preceding code is as follows:

- Calculate the mean for the year 2015 using both NumPy and pandas separately:
# NumPy pandas interoperability import numpy as np print("pandas", dataset["2015"].mean()) print("numpy", np.mean(dataset["2015"]))

The output of the preceding code is as follows:

Note

To access the source code for this specific section, please refer to https://packt.live/2N7E2Kh.

You can also run this example online at https://packt.live/2Y3B2Fa.

This exercise of how to use NumPy's `mean`

method with a pandas DataFrame shows that, in some cases, NumPy has better functionality. However, the DataFrame format of pandas is more applicable, so we combine both libraries to get the best out of both.

You've completed your first exercise with pandas, which showed you some of the similarities, and also differences when working with NumPy and pandas. In the following exercise, this knowledge will be consolidated. You'll also be introduced to more complex features and methods of pandas.

## Basic Operations of pandas

In this section, we will learn about the basic pandas operations, such as indexing, slicing, and iterating, and implement them with an exercise.

### Indexing

Indexing with pandas is a bit more complex than with NumPy. We can only access columns with a single bracket. To use the indices of the rows to access them, we need the `iloc`

method. If we want to access them with `index_col`

(which was set in the `read_csv`

call), we need to use the `loc`

method:

# index the 2000 col dataset["2000"] # index the last row dataset.iloc[-1] # index the row with index Germany dataset.loc["Germany"] # index row Germany and column 2015 dataset[["2015"]].loc[["Germany"]]

### Slicing

Slicing with pandas is even more powerful. We can use the default slicing syntax we've already seen with NumPy or use multi-selection. If we want to slice different rows or columns by name, we can simply pass a list into the brackets:

# slice of the first 10 rows dataset.iloc[0:10] # slice of rows Germany and India dataset.loc[["Germany", "India"]] # subset of Germany and India with years 1970/90 dataset.loc[["Germany", "India"]][["1970", "1990"]]

### Iterating

Iterating DataFrames is also possible. Considering that they can have several `dimensions`

and `dtypes`

, the indexing is very high level and iterating over each row has to be done separately:

# iterating the whole dataset for index, row in dataset.iterrows(): print(index, row)

### Series

A pandas Series is a one-dimensional labeled array that is capable of holding any type of data. We can create a Series by loading datasets from a `.csv`

file, Excel spreadsheet, or SQL database. There are many different ways to create them, such as the following:

- NumPy arrays:
# import pandas import pandas as pd # import numpy import numpy as np # creating a numpy array numarr = np.array(['p','y','t','h','o','n']) ser = pd.Series(numarr) print(ser)

- pandas lists:
# import pandas import pandas as pd # creating a pandas list plist = ['p','y','t','h','o','n'] ser = pd.Series(plist) print(ser)

Now, we will use basic pandas operations in an exercise.

## Exercise 1.06: Indexing, Slicing, and Iterating Using pandas

In this exercise, we will use the previously discussed pandas features to index, slice, and iterate DataFrames using pandas Series. To derive some insights from our dataset, we need to be able to explicitly index, slice, and iterate our data. For example, we can compare several countries in terms of population density growth.

Let's use the indexing, slicing, and iterating operations to display the population density of Germany, Singapore, United States, and India for years 1970, 1990, and 2010.

**Indexing**

- Create a new Jupyter Notebook and save it as
`Exercise1.06.ipynb`

in the`Chapter01/Exercise1.06`

folder. - Import the necessary libraries:
import pandas as pd

- Use the
`read_csv`

method to load the`world_population.csv`

dataset and use the first column, (containing the country names) as our index using the`index_col`

parameter:dataset = \ pd.read_csv('../../Datasets/world_population.csv', \ index_col=0)

- Index the row with the
`index_col`

`"United States"`

using the`loc`

method:dataset.loc[["United States"]].head()

The output of the preceding code is as follows:

- Use reverse indexing in pandas to index the second to last row using the
`iloc`

method:dataset.iloc[[-2]]

The output of the preceding code is as follows:

- Columns are indexed using their header. This is the first line of the CSV file. Index the column with the header of
`2000`

as a Series:dataset["2000"].head()

The output of the preceding code is as follows:

Remember, the

`head()`

method simply returns the first five rows. - First, get the data for the year 2000 as a DataFrame and then select India using the
`loc()`

method using chaining:dataset[["2000"]].loc[["India"]]

The output of the preceding code is as follows:

Since the double brackets notation returns a DataFrame once again, we can chain method calls to get distinct elements.

- Use the single brackets notation to get the distinct value for the population density of India in 2000:
dataset["2000"].loc["India"]

If we want to only retrieve a Series object, we must replace the double brackets with single ones. The output of the preceding code is as follows:

354.326858357522

**Slicing**

- Create a slice with the rows 2 to 5 using the
`iloc()`

method again:# slicing countries of rows 2 to 5 dataset.iloc[1:5]

The output of the preceding code is as follows:

- Use the
`loc()`

method to access several rows in the DataFrame and use the nested brackets to provide a list of elements. Slice the dataset to get the rows for Germany, Singapore, United States, and India:dataset.loc[["Germany", "Singapore", "United States", "India"]]

The output of the preceding code is as follows:

- Use chaining to get the rows for Germany, Singapore, United States, and India and return only the values for the years 1970, 1990, and 2010. Since the double bracket queries return new DataFrames, we can chain methods and therefore access distinct subframes of our data:
country_list = ["Germany", "Singapore", "United States", "India"] dataset.loc[country_list][["1970", "1990", "2010"]]

The output of the preceding code is as follows:

**Iterating**

- Iterate our dataset and print out the countries up until
`Angola`

using the`iterrows()`

method. The index will be the name of our row, and the row will hold all the columns:for index, row in dataset.iterrows(): # only printing the rows until Angola if index == 'Angola': break print(index, '\n', \ row[["Country Code", "1970", "1990", "2010"]], '\n')

The output of the preceding code is as follows:

Note

To access the source code for this specific section, please refer to https://packt.live/2YKqHNM.

You can also run this example online at https://packt.live/2YD56Xo.

We've already covered most of the underlying data wrangling methods using pandas. In the next exercise, we'll take a look at more advanced features such as filtering, sorting, and reshaping to prepare you for the next chapter.

## Advanced pandas Operations

In this section, we will learn about some advanced pandas operations such as filtering, sorting, and reshaping and implement them in an exercise.

### Filtering

Filtering in pandas has a higher-level interface than NumPy. You can still use the simple brackets-based **conditional filtering**. However, you're also able to use more complex queries, for example, filter rows based on labels using likeness, which allows us to search for a substring using the `like`

argument and even full regular expressions using `regex`

:

# only column 1994 dataset.filter(items=["1990"]) # countries population density < 10 in 1999 dataset[(dataset["1990"] < 10)] # years containing an 8 dataset.filter(like="8", axis=1) # countries ending with a dataset.filter(regex="a$", axis=0)

### Sorting

Sorting each row or column based on a given row or column will help you analyze your data better and find the ranking of a given dataset. With pandas, we are able to do this pretty easily. Sorting in ascending and descending order can be done using the parameter known as **ascending**. The default sorting order is `ascending`

. Of course, you can do more complex sorting by providing more than one value in the `by = [ ]`

list. Those will then be used to sort values for which the first value is the same:

# values sorted by 1999 dataset.sort_values(by=["1999"]) # values sorted by 1999 descending dataset.sort_values(by=["1994"], ascending=False)

### Reshaping

Reshaping can be crucial for easier visualization and algorithms. However, depending on your data, this can get really complex:

dataset.pivot(index=["1999"] * len(dataset), \ columns="Country Code", values="1999")

Now, we will use advanced pandas operations to perform an exercise.

## Exercise 1.07: Filtering, Sorting, and Reshaping

This exercise provides some more complex tasks and also combines most of the methods we learned about previously as a recap. After this exercise, you should be able to read the most basic pandas code and understand its logic.

Let's use pandas to filter, sort, and reshape our data.

**Filtering**

- Create a new Jupyter Notebook and save it as
`Exercise1.07.ipynb`

in the`Chapter01/Exercise1.07`

folder. - Import the necessary libraries:
# importing the necessary dependencies import pandas as pd

- Use the
`read_csv`

method to load the dataset, again defining our first column as an index column:# loading the dataset dataset = \ pd.read_csv('../../Datasets/world_population.csv', \ index_col=0)

- Use
`filter`

instead of using the bracket syntax to filter for specific items. Filter the dataset for columns 1961, 2000, and 2015 using the items parameter:# filtering columns 1961, 2000, and 2015 dataset.filter(items=["1961", "2000", "2015"]).head()

The output of the preceding code is as follows:

- Use conditions to get all the countries that had a higher population density than
`500`

in`2000`

. Simply pass this condition in brackets:""" filtering countries that had a greater population density than 500 in 2000 """ dataset[(dataset["2000"] > 500)][["2000"]]

The output of the preceding code is as follows:

- Search for arbitrary columns or rows (depending on the index given) that match a certain
`regex`

. Get all the columns that start with 2 by passing`^2`

(meaning that it starts at`2`

):dataset.filter(regex="^2", axis=1).head()

The output of the preceding code is as follows:

- Filter the rows instead of the columns by passing
`axis=0`

. This will be helpful for situations when we want to filter all the rows that start with`A`

:dataset.filter(regex="^A", axis=0).head()

The output of the preceding code is as follows:

- Use the
`like`

query to find only the countries that contain the word`land`

, such as Switzerland:dataset.filter(like="land", axis=0).head()

The output of the preceding code is as follows:

**Sorting**

- Use the
`sort_values`

or`sort_index`

method to get the countries with the lowest population density for the year 1961:dataset.sort_values(by=["1961"])[["1961"]].head(10)

The output of the preceding code is as follows:

- Just for comparison, carry out sorting for
`2015`

:dataset.sort_values(by=["2015"])[["2015"]].head(10)

The output of the preceding code is as follows:

We can see that the order of the countries with the lowest population density has changed a bit, but that the first three entries remain unchanged.

- Sort column 2015 in
`descending`

order to show the biggest values first:dataset.sort_values(by=["2015"], \ ascending=False)[["2015"]].head(10)

The output of the preceding code is as follows:

**Reshaping**

- Get a DataFrame where the columns are
`country codes`

and the only row is the year`2015`

. Since we only have one`2015`

label, we need to duplicate it as many times as our dataset's length:# reshaping to 2015 as row and country codes as columns dataset_2015 = dataset[["Country Code", "2015"]] dataset_2015.pivot(index=["2015"] * len(dataset_2015), \ columns="Country Code", values="2015")

The output of the preceding code is as follows:

Note

To access the source code for this specific section, please refer to https://packt.live/2N0xHQZ.

You can also run this example online at https://packt.live/30Jeziw.

You now know the basic functionality of pandas and have already applied it to a real-world dataset. In the final activity for this chapter, we will try to analyze a forest fire dataset to get a feeling for mean forest fire sizes and whether the temperature of each month is proportional to the number of fires.

## Activity 1.02: Forest Fire Size and Temperature Analysis

In this activity, we will use pandas features to derive some insights from a forest fire dataset. We will get the mean size of forest fires, what the largest recorded fire in our dataset is, and whether the amount of forest fires grows proportionally to the temperature in each month.

Our forest fires dataset has the following structure:

`X`

: X-axis spatial coordinate within the Montesinho park map: 1 to 9`Y`

: Y-axis spatial coordinate within the Montesinho park map: 2 to 9`month`

: Month of the year: 'jan' to 'dec'`day`

: Day of the week: 'mon' to 'sun'`FFMC`

: FFMC index from the FWI system: 18.7 to 96.20`DMC`

: DMC index from the FWI system: 1.1 to 291.3`DC`

: DC index from the FWI system: 7.9 to 860.6`ISI`

: ISI index from the FWI system: 0.0 to 56.10`temp`

: Temperature in degrees Celsius: 2.2 to 33.30`RH`

: Relative humidity in %: 15.0 to 100`wind`

: Wind speed in km/h: 0.40 to 9.40`rain`

: Outside rain in mm/m2: 0.0 to 6.4`area`

: The burned area of the forest (in ha): 0.00 to 1090.84Note

We will only be using the

`month`

,`temp`

, and`area`

columns in this activity.

The following are the steps for this activity:

- Open the
`Activity1.02.ipynb`

Jupyter Notebook from the`Chapter01`

folder to complete this activity. Import`pandas`

using the`pd`

alias. - Load the
`forestfires.csv`

dataset using pandas. - Print the first two rows of the dataset to get a feeling for its structure.

**Derive insights from the sizes of forest fires**

- Filter the dataset so that it only contains entries that have an area larger than 0.
- Get the mean, min, max, and std of the area column and see what information this gives you.
- Sort the filtered dataset using the area column and print the last 20 entries using the
`tail`

method to see how many huge values it holds. - Then, get the median of the area column and visually compare it to the mean value.

**Finding the month with the most forest fires**

- Get a list of unique values from the month column of the dataset.
- Get the number of entries for the month of March using the
`shape`

member of our DataFrame. - Now, iterate over all the months, filter our dataset for the rows containing the given month, and calculate the mean temperature. Print a statement with the number of fires, the mean temperature, and the month.
Note

The solution for this activity can be found via this link.

You have now completed this topic all about pandas, which concludes this chapter. We have learned about the essential tools that help you wrangle and work with data. pandas is an incredibly powerful and widely used tool for wrangling and understanding data.

# Summary

NumPy and pandas are essential tools for data wrangling. Their user-friendly interfaces and performant implementation make data handling easy. Even though they only provide a little insight into our datasets, they are valuable for wrangling, augmenting, and cleaning our datasets. Mastering these skills will improve the quality of your visualizations.

In this chapter, we learned about the basics of NumPy, pandas, and statistics. Even though the statistical concepts we covered are basic, they are necessary to enrich our visualizations with information that, in most cases, is not directly provided in our datasets. This hands-on experience will help you implement the exercises and activities in the following chapters.

In the next chapter, we will focus on the different types of visualizations and how to decide which visualization would be best for our use case. This will give you theoretical knowledge so that you know when to use a specific chart type and why. It will also lay down the fundamentals of the remaining chapters in this book, which will focus on teaching you how to use Matplotlib and seaborn to create the plots we have discussed here. After we have covered basic visualization techniques with Matplotlib and seaborn, we will dive more in-depth and explore the possibilities of interactive and animated charts, which will introduce an element of storytelling into our visualizations.