Chapter 7. Descriptive Statistics

This and the following chapter are mainly aimed at SAS, SPSS, or Minitab users, and especially those employing the languages R or S for statistical computing. We will develop an environment for working effectively in the field of data analysis, with the aid of IPython sessions powered up with the following resources from the SciPy stack:

• The probability and statistics submodule of the library of symbolic computations, sympy.stats.

• The two libraries of statistical functions scipy.stats and scipy.stats.mstats (the latter for data provided by masked arrays), together with the module statsmodels, for data exploration, estimation on statistical models, and performing statistical tests in a numerical setting. The package statsmodels uses, under the hood, the powerful library patsy to describe statistical models and building design matrices in Python (R or S users will find patsy compatible with their formula mini-language).

• For statistical inference, we again use...

Motivation

On Tuesday, September 8, 1857, the steamboat SS Central America left Havana at 9 A.M. for New York, carrying about 600 passengers and crew members. Inside this vessel, precious cargo was stored—a set of manuscripts by John James Audubon, and three tons of gold bars and coins. The manuscripts documented an expedition through the yet uncharted southwestern United States and California, and contained 200 sketches and paintings of its wildlife. The gold, fruit of many years of prospecting and mining during the California Gold Rush, was meant to start anew the lives of many of the passengers aboard.

On the 9th, the vessel ran into a storm which developed into a hurricane. The steamboat endured four hard days at sea, and by Saturday morning the ship was doomed. The captain arranged to have women and children taken off to the brig Marine, which offered them assistance at about noon. In spite of the efforts of the remaining crew and passengers to save the ship, the inevitable happened...

Probability

In the SciPy stack, we have two means for determining probability: a symbolic setting and a numerical setting. In this brief section, we are going to compare both with a sequence of examples.

For the symbolic treatment of random variables, we employ the module sympy.stats, while for the numerical treatment, we use the module scipy.stats. In both cases, the goal is the same—the instantiation of any random variable, and the following three kinds of operations on them:

• Description of the probability distribution of a random variable with numbers (parameters).

• Description of a random variable in terms of functions.

• Computation of associated probabilities.

Let's observe several situations through the scope of the two different settings.

Symbolic setting

Let's start with discrete random variables. For instance, let's consider several random variables used to describe the process of rolling three 6-sided dice, one 100-sided dice, and the possible outcomes:

In [1]: from sympy import var; \...

Data exploration

Data exploration is generally performed by presenting a meaningful synthesis of its distribution—it could be through a sequence of graphs, by describing it with a set of numerical parameters, or by approximating it with simple functions. Now let's explore different possibilities, and how to accomplish them with different tools in the SciPy stack.

Picturing distributions with graphs

The type of graph depends on the type of variable (categorical, quantitative, or dates).

Bar plots and pie charts

When our data is described in terms of categorical variables, we often use pie charts or bar graphs to represent it. For example, we access the Consumer Complaint Database from the Consumer Financial Protection Bureau, at http://catalog.data.gov/dataset/consumer-complaint-database. The database was created in February 2014 to contain complaints received by the Bureau about financial products and services. In its updated version in March of the same year, it consisted of almost 300,000...

Summary

This concludes the first part of our two-chapter coverage of Data Analysis, where we have explored advanced Python tools in the SciPy stack for computation, and visualization of descriptive statistics. In the next chapter, we produce a similar treatment of inference statistics, data mining, and machine learning.