Chapter 15
Project 5.1: Modeling Base Application

The next step in the pipeline from acquisition to clean-and-convert is the analysis and some preliminary modeling of the data. This may lead us to use the data for a more complex model or perhaps machine learning. This chapter will guide you through creating another application in the three-stage pipeline to acquire, clean, and model a collection of data. This first project will create the application with placeholders for more detailed and application-specific modeling components. This makes it easier to insert small statistical models that can be replaced with more elaborate processing if needed.

In this chapter, we’ll look at two parts of data analysis:

• CLI architecture and how to design a more complex pipeline of processes for gathering and analyzing data

• The core concepts of creating a statistical model of the data

Viewed from a distance, all analytical work can be considered to be creating a simplified model of important...

15.1 Description

This application will create a report on a dataset presenting a number of statistics. This automates the ongoing monitoring aspect of an Analysis Notebook, reducing the manual steps and creating reproducible results. The automated computations stem from having a statistical model for the data, often created in an analysis notebook, where alternative models are explored. This reflects variables with values in an expected range.

For industrial monitoring, this is part of an activity called Gage repeatability and reproducibility. The activity seeks to confirm that measurements are repeatable and reproducible. This is described as looking at a “measurement instrument.” While we often think of an instrument as being a machine or a device, the definition is actually very broad. A survey or questionnaire is a measurement instrument focused on people’s responses to questions.

When these computed statistics deviate from expectations, it suggests something...

15.2 Approach

We’ll take some guidance from the C4 model ( https://c4model.com) when looking at our approach:

• Context: For this project, a context diagram would show a user creating analytical reports. You may find it helpful to draw this diagram.

• Containers: There only seems to be one container: the user’s personal computer.

• Components: We’ll address the components below.

• Code: We’ll touch on this to provide some suggested directions.

The heart of this application is a module to summarize data in a way that lets us test whether it fits the expectations of a model. The statistical model is a simplified reflection of the underlying real-world processes that created the source data. The model’s simplifications include assumptions about events, measurements, internal state changes, and other details of the processing being observed.

For very simple cases — like Anscombe’s Quartet data — there are only two variables, which leaves...

15.3 Deliverables

This project has the following deliverables:

• Documentation in the docs folder.

• Acceptance tests in the tests/features and tests/steps folders.

• Unit tests for model module classes in the tests folder.

• Mock objects for the csv_extract module tests will be part of the unit tests.

• Unit tests for the csv_extract module components that are in the tests folder.

• An application to summarize the cleaned data in a TOML file.

• An application secondary feature to transform the TOML file to an HTML page or PDF file with the summary.

In some cases, especially for particularly complicated applications, the summary statistics may be best implemented as a separate module. This module can then be expanded and modified without making significant changes to the overall application.

The idea is to distinguish between these aspects of this application:

• The CLI, which includes argument parsing and sensible handling of input and output paths.

• The statistical model, which evolves as our understanding...

15.4 Summary

In this chapter we have created a foundation for building and using a statistical model of source data. We’ve looked at the following topics:

• Designing and building a more complex pipeline of processes for gathering and analyzing data.

• Some of the core concepts behind creating a statistical model of some data.

• Use of the built-in statistics library.

• Publishing the results of the statistical measures.

This application tends to be relatively small. The actual computations of the various statistical values leverage the built-in statistics library and tend to be very small. It often seems like there’s far more programming involved in parsing the CLI argument values, and creating the required output file, than doing the “real work” of this application.

This is a consequence of the way we’ve been separating the various concerns in data acquisition, cleaning, and analysis. We’ve partitioned the work into several, isolated stages along...

15.5 Extras

Here are some ideas for you to add to this project.

15.5.1 Measures of shape

The measurements of shape often involve two computations for skewness and kurtosis. These functions are not part of Python’s built-in statistics library.

It’s important to note that there are a very large number of distinct, well-understood distributions of data. The normal distribution is one of many different ways data can be distributed.

See https://www.itl.nist.gov/div898/handbook/eda/section3/eda366.htm.

One measure of skewness is the following:

Where Ȳ is the mean, and s is the standard deviation.

A symmetric distribution will have a skewness, g₁, near zero. Larger numbers indicate a ”long tail” opposite a large concentration of data around the mean.

One measure of kurtosis is the following:

The kurtosis for the standard normal distribution is 3. A value larger than 3 suggests more data is in the tails; it’s ”flatter” or ”...