Chapter 7
Data Inspection Features

There are three broad kinds of data domains: cardinal, ordinal, and nominal. The first project in this chapter will guide you through the inspection of cardinal data; values like weights, measures, and durations where the data is continuous, as well as counts where the data is discrete. The second project will guide reasoners through the inspection of ordinal data involving things like dates, where order matters, but the data isn’t a proper measurement; it’s more of a code or designator. The nominal data is a code that happens to use digits but doesn’t represent numeric values. The third project will cover the more complex case of matching keys between separate data sources.

An inspection notebook is required when looking at new data. It’s a great place to keep notes and lessons learned. It’s helpful when diagnosing problems that arise in a more mature analysis pipeline.

This chapter will cover a number of skills...

Chapter 7
Data Inspection Features

There are three broad kinds of data domains: cardinal, ordinal, and nominal. The first project in this chapter will guide you through the inspection of cardinal data; values like weights, measures, and durations where the data is continuous, as well as counts where the data is discrete. The second project will guide reasoners through the inspection of ordinal data involving things like dates, where order matters, but the data isn’t a proper measurement; it’s more of a code or designator. The nominal data is a code that happens to use digits but doesn’t represent numeric values. The third project will cover the more complex case of matching keys between separate data sources.

An inspection notebook is required when looking at new data. It’s a great place to keep notes and lessons learned. It’s helpful when diagnosing problems that arise in a more mature analysis pipeline.

This chapter will cover a number of skills...

7.1.1 Description

This project’s intent is to inspect raw data to understand if it is actually cardinal data. In some cases, floating-point values may have been used to represent nominal data; the data appears to be a measurement but is actually a code.

Spreadsheet software tends to transform all data into floating-point numbers; many data items may look like cardinal data.

One example is US Postal Codes, which are strings of digits, but may be transformed into numeric values by a spreadsheet.

Another example is bank account numbers, which — while very long — can be converted into floating-point numbers. A floating-point value uses 8 bytes of storage, but will comfortably represent about 15 decimal digits. While this is a net saving in storage, it is a potential confusion of data types and there is a (small) possibility of having an account number altered by floating-point truncation rules.

The user experience is a Jupyter Lab notebook that can be used to examine...

7.1.2 Approach

This project is based on the initial inspection notebook from Chapter 6, Project 2.1: Data Inspection Notebook. Some of the essential cell content will be reused in this notebook. We’ll add components to the components shown in the earlier chapter – specifically, the samples_iter() function to iterate over samples in an open file. This feature will be central to working with the raw data.

In the previous chapter, we suggested avoiding conversion functions. When starting down the path of inspecting data, it’s best to assume nothing and look at the text values first.

There are some common patterns in the source data values:

• The values appear to be all numeric values. The int() or float() function works on all of the values. There are two sub-cases here:

  • All of the values seem to be proper counts or measures in some expected range. This is ideal.

  • A few “outlier” values are present. These are values that seem to be outside the expected...

7.1.3 Deliverables

This project has the following deliverables:

• A requirements-dev.txt file that identifies the tools used, usually jupyterlab==3.5.3.

• Documentation in the docs folder.

• Unit tests for any new changes to the modules in use.

• Any new application modules with code to be used by the inspection notebook.

• A notebook to inspect the attributes that appear to have cardinal data.

This project will require a notebooks directory. See List of deliverables for some more information on this structure.

We’ll look at a few of these deliverables in a little more detail.

Inspection module

You are encouraged to refactor functions like samples_iter(), non_numeric(), and numeric_filter() into a separate module. Additionally, the AttrSummary class and the closely related summary_iter() function are also good candidates for being moved to a separate module with useful inspection classes and functions.

Notebooks can be refactored to import these classes and functions from a separate...

Chapter 7
Data Inspection Features

There are three broad kinds of data domains: cardinal, ordinal, and nominal. The first project in this chapter will guide you through the inspection of cardinal data; values like weights, measures, and durations where the data is continuous, as well as counts where the data is discrete. The second project will guide reasoners through the inspection of ordinal data involving things like dates, where order matters, but the data isn’t a proper measurement; it’s more of a code or designator. The nominal data is a code that happens to use digits but doesn’t represent numeric values. The third project will cover the more complex case of matching keys between separate data sources.

An inspection notebook is required when looking at new data. It’s a great place to keep notes and lessons learned. It’s helpful when diagnosing problems that arise in a more mature analysis pipeline.

This chapter will cover a number of skills...

7.2.1 Description

In the previous project (Project 2.2: Validating cardinal domains — measures, counts, and durations), we looked at attributes that contained cardinal data – measures and counts. We also need to look at ordinal and nominal data. Ordinal data is generally used to provide ranks and ordering. Nominal data is best thought of as codes made up of strings of digits. Values like US postal codes and bank account numbers are nominal data.

When we look at the CO2 PPM — Trends in Atmospheric Carbon Dioxide data set, available at https://datahub.io/core/co2-ppm, it has dates that are provided in two forms: as a year-month-day string and as a decimal number. The decimal number positions the first day of the month within the year as a whole.

It’s instructive to use ordinal day numbers to compute unique values for each date and compare these with the supplied ”Decimal Date” value. An integer day number may be more useful than the decimal date...

7.2.2 Approach

Dates and times often have bewildering formats. This is particularly true in the US, where dates are often written as numbers in month/day/year format. Using year/month/day puts the values in order of significance. Using day/month/year is the reverse order of significance. The US ordering is simply strange.

This makes it difficult to do inspections on completely unknown data without any metadata to explain the serialization format. A date like 01/02/03 could mean almost anything.

In some cases, a survey of many date-like values will reveal a field with a range of 1-12 and another field with a range of 1-31, permitting analysts to distinguish between the month and day. The remaining field can be taken as a truncated year.

In cases where there is not enough data to make a positive identification of month or day, other clues will be needed. Ideally, there’s metadata to define the date format.

The datetime.strptime() function can be used to parse dates when the format...

7.2.3 Deliverables

This project has the following deliverables:

• A requirements-dev.txt file that identifies the tools used, usually jupyterlab==3.5.3.

• Documentation in the docs folder.

• Unit tests for any new changes to the modules in use.

• Any new application modules with code to be used by the inspection notebook.

• A notebook to inspect the attributes that appear to have ordinal or nominal data.

The project directory structure suggested in Chapter 1, Project Zero: A Template for Other Projects mentions a notebooks directory. See List of deliverables for some more information. For this project, the notebook directory is needed.

We’ll look at a few of these deliverables in a little more detail.

Revised inspection module

Functions for date conversions and cleaning up nominal data can be written in a separate module. Or they can be developed in a notebook, and then moved to the inspection module. As we noted in the Description section, this project’s objective is to...

Chapter 7
Data Inspection Features

There are three broad kinds of data domains: cardinal, ordinal, and nominal. The first project in this chapter will guide you through the inspection of cardinal data; values like weights, measures, and durations where the data is continuous, as well as counts where the data is discrete. The second project will guide reasoners through the inspection of ordinal data involving things like dates, where order matters, but the data isn’t a proper measurement; it’s more of a code or designator. The nominal data is a code that happens to use digits but doesn’t represent numeric values. The third project will cover the more complex case of matching keys between separate data sources.

An inspection notebook is required when looking at new data. It’s a great place to keep notes and lessons learned. It’s helpful when diagnosing problems that arise in a more mature analysis pipeline.

This chapter will cover a number of skills...

7.4 Summary

This chapter expanded on the core features of the inspection notebook. We looked at handling cardinal data (measures and counts), ordinal data (dates and ranks), and nominal data (codes like account numbers).

Our primary objective was to get a complete view of the data, prior to formalizing our analysis pipeline. A secondary objective was to leave notes for ourselves on outliers, anomalies, data formatting problems and other complications. A pleasant consequence of this effort is to be able to write some functions that can be used downstream to clean and normalize the data we’ve found.

Starting in Chapter 9, Project 3.1: Data Cleaning Base Application, we’ll look at refactoring these inspection functions to create a complete and automated data cleaning and normalization application. That application will be based on the lessons learned while creating inspection notebooks.

In the next chapter, we’ll look at one more lesson that’s often learned...

7.5 Extras

Here are some ideas for you to add to the projects in this chapter.

7.5.1 Markdown cells with dates and data source information

A minor feature of an inspection notebook is some identification of the date, time, and source of the data. It’s sometimes clear from the context what the data source is; there may, for example, be an obvious path to the data.

However, in many cases, it’s not perfectly clear what file is being inspected or how it was acquired. As a general solution, any processing application should produce a log. In some cases, a metadata file can include the details of the processing steps.

This additional metadata on the source and processing steps can be helpful when reviewing a data inspection notebook or sharing a preliminary inspection of data with others. In many cases, this extra data is pasted into ordinary markdown cells. In other cases, this data may be the result of scanning a log file for key INFO lines that summarize processing.

...