Introduction to pandas and Data Analysis

We start the book and this chapter by discussing the contemporary data analytics landscape and how pandas fits into that landscape. pandas is the go-to tool for data scientists for data pre-processing tasks. We will learn about the technicalities of pandas in the later chapters. This chapter covers the context, origin, history, market share, and current standing of pandas.

The chapter has been divided into the following headers:

• Motivation for data analysis
• How Python and pandas can be used for data analysis
• Description of the pandas library
• Benefits of using pandas

Motivation for data analysis

In this section, we discuss the trends that are making data analysis an increasingly important field in today's fast-moving technological landscape.

We live in a big data world

The term big data has become one of the hottest technology buzzwords in the past two years. We now increasingly hear about big data in various media outlets, and big data start-ups have increasingly been attracting venture capital. A good example in the area of retail is Target Corporation, which has invested substantially in big data and is now able to identify potential customers by using big data to analyze people's shopping habits online; refer to a related article at http://nyti.ms/19LT8ic.

Loosely speaking...

Data analytics pipeline

Data modeling is the process of using data to build predictive models. Data can also be used for descriptive and prescriptive analysis. But before we make use of data, it has to be fetched from several sources, stored, assimilated, cleaned, and engineered to suit our goal. The sequential operations that need to be performed on data are akin to a manufacturing pipeline, where each subsequent step adds value to the potential end product and each progression requires a new person or skill set.

The various steps in a data analytics pipeline are shown in the following diagram:

Steps in data analytics pipeline

1. Extract Data
2. Transform Data
3. Load Data
4. Read & Process Data
5. Exploratory Data Analysis
6. Create Features
7. Build Predictive Models
8. Validate Models
9. Build Products

These steps can be combined into three high-level categories: data engineering, data science...

How Python and pandas fit into the data analytics pipeline

The Python programming language is one of the fastest-growing languages today in the emerging field of data science and analytics. Python was created by Guido van Rossum in 1991, and its key features include the following:

• Interpreted rather than compiled
• Dynamic type system
• Pass by value with object references
• Modular capability
• Comprehensive libraries
• Extensibility with respect to other languages
• Object orientation
• Most of the major programming paradigms: procedural, object-oriented, and, to a lesser extent, functional

For more information, refer to the following article on Python at https://www.python.org/about/.

Among the characteristics that make Python popular for data science are its very user-friendly (human-readable) syntax, the fact that it is interpreted rather than compiled (leading to faster development...

What is pandas?

The pandas we are going to obsess over in this book are not the cute and lazy animals that also do kung fu when needed.

pandas is a high-performance open source library for data analysis in Python developed by Wes McKinney in 2008. pandas stands for panel data, a reference to the tabular format in which it processes the data. It is available for free and is distributed with a 3-Clause BSD License under the open source initiative.

Over the years, it has become the de-facto standard library for data analysis using Python. There's been great adoption of the tool, and there's a large community behind it, (1,200+ contributors, 17,000+ commits, 23 versions, and 15,000+ stars) rapid iteration, features, and enhancements are continuously made.

Some key features of pandas include the following:

• It can process a variety of datasets in different formats: time series...

Where does pandas fit in the pipeline?

As discussed in the previous section, pandas can be used to perform Step 4 to Step 6 in the pipeline. And Step 4 to Step 6 are the backbone of any data science process, application, or product:

Where does pandas fit in the data analytics pipeline?

The Step 1 to Step 6 can be performed in pandas by some methods. Those in the Step 4 to Step 6 are the primary tasks while the Step 1 to Step 3 can also be done in some way or other in pandas.

pandas is an indispensable library if you're working with data, and it would be near impossible to find code for data modeling that doesn't import pandas into the working environment. Easy-to-use syntax in Python and the availability of a spreadsheet-like data structure called a dataframe make it amenable even to users who are too comfortable and too unwilling to move away from Excel. At the same...

Benefits of using pandas

pandas forms a core component of the Python data analysis corpus. The distinguishing feature of pandas is that the suite of data structures that it provides is naturally suited to data analysis, primarily the DataFrame and, to a lesser extent, series (1-D vectors) and panel (3D tables).

Simply put, pandas and statstools can be described as Python's answer to R, the data analysis and statistical programming language that provides both data structures, such as R-dataframes, and a rich statistical library for data analysis.

The benefits of pandas compared to using a language such as Java, C, or C++ for data analysis are manifold:

• Data representation: It can easily represent data in a form that's naturally suited for data analysis via its DataFrame and series data structures in a concise manner. Doing the equivalent in Java/C/C++ requires many lines...

History of pandas

The basic version of pandas was built in 2008 by Wes McKinney, an MIT grad with heavy quantitative finance experience. Now a celebrity in his own right, thanks to his open source contributions and the wildly popular book called Data Analysis with Python, he was reportedly frustrated with the time he had to waste doing simple data manipulation tasks at his job, such as reading a CSV file, with the popular tools at that time. He said he quickly fell in love with Python for its intuitive and accessible nature after not finding Excel and R suitable for his needs. But he found that it was missing key features that would make it the go-to tool for data analysis—for example, an intuitive format to deal with spreadsheet data or to create new calculated columns from existing columns.

According to an interview he gave to Quartz, the design considerations and vision...

Usage pattern and adoption of pandas

The popularity of Python has skyrocketed over the years, especially after 2012; a lot of this can be attributed to the popularity of pandas. Python-related questions make up around 12% of the total questions asked from high-income countries on Stack Overflow, a popular platform for developers to ask questions and get answers from other people in the community about how to get things done and fix bugs in different programming languages. Given that there are hundreds of programming languages, one language occupying 12% of market share is an extraordinary achievement:

The most popular data analytics tools based on a survey of Kaggle users conducted in 2017-18

According to this survey conducted by Kaggle, 60% of the respondents said that they were aware of or have used Python for their data science jobs.

According to the data recorded by Stack...

pandas on the technology adoption curve

According to a popular framework called Gartner Hype Cycle, there are five phases in the process of the proliferation and adoption of technologies:

• Technology trigger
• Peak of inflated expectations
• Trough of disillusionment
• Slope of enlightenment
• Plateau of productivity

The following link contains a chart that shows different technologies and the stage they are at on the technology adoption curve https://blogs-images.forbes.com/gartnergroup/files/2012/09/2012Emerging-Technologies-Graphic4.gif.

As can be seen, Predictive Analytics has already reached the steady plateau of productivity, which is where the optimum and stable return on investment can be extracted from a technology. Since pandas is an essential component of most predictive analytics initiatives, it is safe to say that pandas has reached the plateau of productivity.

...

Popular applications of pandas

pandas is built on top of NumPy. Some of the noteworthy uses of the pandas, apart from every other data science project of course, are the following:

• pandas is a dependency of statsmodels (http://www.statsmodels.org/stable/index.html), making it a significant part of Python's numerical computing ecosystem.
• pandas has been used extensively in the production of many financial applications.

Summary

We live in a big data era characterized by the four V's- volume, velocity, variety, and veracity. The volume and velocity of data are set to increase for the foreseeable future. Companies that can harness and analyze big data to extract information and take actionable decisions based on this information will be the winners in the marketplace. Python is a fast-growing, user-friendly, extensible language that is very popular for data analysis.

pandas is a core library of the Python toolkit for data analysis. It provides features and capabilities that make it much easier and faster than many other popular languages, such as Java, C, C++, and Ruby.

Thus, given the strengths of Python outlined in this chapter as a choice for the analysis of data and the popularity it has gained from users, contributors, and industry leaders, data analysis practitioners utilizing Python...

References

• https://activewizards.com/blog/top-20-python-libraries-for-data-science-in-2018/
  
• https://qz.com/1126615/the-story-of-the-most-important-tool-in-data-science/