Chapter 5: Data Selection – DataFrames

In this chapter, you will develop an understanding of the different forms of the pandas index, and how the index is involved in slicing, which is one way to get a subset of a pandas data structure. You will learn how to manipulate the index itself, as well as the different notations pandas provides for selection.

By the end of this chapter, you will be able to select subsets of data and work with the index efficiently. You will also learn how to implement pandas dot, bracket, .loc(), and .iloc() notations to slice and index.

This chapter covers the following topics:

• Introduction to DataFrames
• Data selection in pandas DataFrames
• Activity 5.01 – Creating a multi-index from columns
• Bracket and dot notation
• Changing DataFrame values using bracket and dot notation

Introduction to DataFrames

Imagine that you're working on a dataset that contains hundreds of columns and thousands of rows, of which only a small subset – say a dozen rows and two or three columns – matter to you for a particular analysis. In such cases, it's better to isolate and focus on those rows and columns rather than working with the entire dataset. In data analysis and data science, you will constantly need to work with a subset of a larger dataset. Thankfully, pandas provides selection methods that make this process easy and efficient. You will learn about these methods in this chapter. We will start by revisiting DataFrames and then see how pandas selection methods apply to DataFrames.

So far in this book, you have learned about the basics of the pandas data structures (Chapter 2, Data Structures), how to get data in or out of pandas (Chapter 3, Data I/O), and the different data types in pandas (Chapter 4, Data Types). Now, it is time to integrate...

Data selection in pandas DataFrames

In Chapter 3, Data Structures, we studied the two core pandas data structures, DataFrames and Series. There, we did some very basic data selection without digging into the details of how it works. In this section, we will do a deeper dive and explore the index, which is fundamental to many pandas operations.

As you may recall when we introduced the idea of DataFrames, we drew analogies to spreadsheets. Let's revisit that analogy. Here is the same figure from Chapter 2, Data Structures (which is the data from Figure 5.1 but in a spreadsheet):

Figure 5.2 – The industry GDP data in a spreadsheet

Here, we can see the same three columns of data that were shown in Figure 5.1, but we have annotated the key differences. In pandas, the standard row index starts at 0, while for most spreadsheets, it starts at row 1. This "0 indexing" is standard for Python. An index in pandas is a series of numbers or strings...

Activity 5.01 – Creating a multi-index from columns

In this activity, you will read in a DataFrame from a file and then use some of the columns to create a sorted multi-index. Suppose you have been given a .csv file containing data about mushrooms, which, as you understand it, contains a classification of edible or poisonous mushrooms, as well as many visual features to allow them to be identified. Since you are a mushroom hunting enthusiast, you are very interested in analyzing and summarizing the data. You begin by reading the data in. Let's get started:

1. For this activity, all you will need is the pandas library. Load it into the first cell of the notebook.
2. Read in the mushroom.csv data from the Datasets directory and list the first five rows using .head().
3. You will see the class column and many visible attributes. List all the columns to see what else there is to work with. The result should be as follows:

Figure 5.32 –...

Bracket and dot notation

In the previous section, we focused on the DataFrame.loc method. pandas offers two ways to select data – using just brackets, [], and using what is called dot notation (pandas also refers to the latter as attribute access since object.name is Python syntax for accessing the name attribute in object).

Bracket notation

We have already introduced one form of bracket notation, which is using a column name inside brackets. There are several ways to apply bracket notation to a DataFrame, as follows:

• Select entire columns: DataFrame['column_name'] or DataFrame[[list of column names]]. If a single column is selected, the result is a Series; otherwise, the result is a DataFrame. If an additional selection results in only one row, the result can be a Series. Also, if the DataFrame only contains one row, selecting one column returns a Series (even though the result is a single value).
• Selecting a range of rows: DataFrame[start:end...

Changing DataFrame values using bracket or dot notation

Many of the methods we've discussed can be used to change the values in a DataFrame, as well as select slices or ranges. In the following screenshot, we can see the GDP data that we have been working with for 2015:

Figure 5.50 – The new GDP_2015 DataFrame

Now, suppose that as part of the economic analysis, we want to increase all the GDP values by 5,000. We can do this by selecting the GDP column using bracket notation on the left, and then doing the same and adding 5,000 on the right:

GDP_2015['GDP'] = GDP_2015['GDP'] + 5000

GDP_2015

This will produce the following output:

Figure 5.51 – The GDP_2015 DataFrame with every value in the GDP column increased by 5,000

Here, we can see the expected result – that is, all our GDP figures have been increased by 5,000. Thus, using bracket notation, we can choose where new data goes into...

Summary

In this chapter, you learned about the pandas methods for data indexing and selection by using the primary pandas data structure – the DataFrame. You compared the DataFrame.loc() and DataFrame.iloc() methods to access items in DataFrames by labels and integer locations, respectively. You also looked at some pandas shortcut methods, including bracket notation, dot notation, and extended indexing. Along the way, you saw how the pandas index is used behind the scenes to align data, and how that can be changed by changing or resetting the index. In addition, we showed you that in many cases, you can assign new values to a subset of data by using it on the left-hand side of an assignment statement (using the equals operator). This creates a very compact and easy-to-read coding style. We saw that an important pandas capability that involved using labels for the row or column index produced more robust code – instead of "hardcoding" the column numbers, they...