Chapter 1
Project Zero: A Template for Other Projects

This is a book of projects. To make each project a good portfolio piece, we’ll treat each project as an enterprise software product. You can build something that could be posted to a company’s (or organization’s) internal repository.

For this book, we’ll define some standards that will apply to all of these projects. The standards will identify deliverables as a combination of files, modules, applications, notebooks, and documentation files. While each enterprise is unique, the standards described here are consistent with my experience as a consultant with a variety of enterprises.

We want to draw an informal boundary to avoid some of the steps required to post to the PyPI website. Our emphasis is on a product with test cases and enough documentation to explain what it does. We don’t want to go all the way to creating a project in PyPI. This allows us to avoid the complications of a build system...

1.1 On quality

It helps to have a clear definition of expectations. For these expectations, we’ll rely on the ISO 25010 standard to define quality goals for each project. For more details, see https://iso25000.com/index.php/en/iso-25000-standards/iso-25010.

The ISO/IEC 25010:2011 standard describes Systems and software Quality Requirements and Evaluation (SQuaRE). This standard provides eight characteristics of software. These characteristics are as follows:

• Functional suitability. Does it do what we need? It is complete, correct, and appropriate for the user’s expressed (and implied) needs? This is the focus of each project’s description.

• Performance efficiency. Does it work quickly? Does it use the minimum resources? Does it have enough capacity to meet the user’s needs? We won’t address this deeply in this book. We’ll talk about writing performance tests and ways to address performance concerns.

• Compatibility. Does it co-exist with other...

1.2 Suggested project sprints

We hesitate to provide a detailed step-by-step process for building software. For more experienced developers, our sequence of steps may not match their current practices. For less experienced developers, the suggested process can help by providing a rational order in which the deliverables can be built.

There was a time when a ”statement of work” with a detailed list of specific tasks was a central part of a software development effort. This was often part of a ”waterfall” methodology where requirements flowed to analysts who wrote specifications that flowed down to designers who wrote high-level designs that flowed down to coders. This wasn’t a great way to build software, and has been largely supplanted by Agile methods. For more information on Agility, see https://agilemanifesto.org.

The Agile approach lets us examine a project both as a series of steps to be completed, as well as a collection of deliverables that need...

1.3 List of deliverables

We’ll take another look at the project, this time from the view of what files will be created. This will parallel the outline of the activities shown in the previous section.

The following outline shows many of the files in a completed project:

• The documentation in the docs directory. There will be other files in there, but you’ll be focused on the following files:

  • The Sphinx index.rst starter file with references to overview and API sections.

  • An overview.rst section with a summary of the project.

  • An api.rst section with .. automodule:: commands to pull in documentation from the application.

• A set of test cases in the tests directory.

  • Acceptance tests aimed at Behave (or the pytest-bdd plug-in for Gherkin). When using Behave, there will be two sub-directories: a features directory and a steps directory. Additionally, there will be an environment.py file.

  • Unit test modules written with the pytest framework. These all have a name that starts...

1.4 Development tool installation

Many of the projects in this book are focused on data analysis. The tooling for data analysis is often easiest to install with the conda tool. This isn’t a requirement, and readers familiar with the PIP tool will often be able to build their working environments without the help of the conda tool.

We suggest the following tools:

• Conda for installing and configuring each project’s unique virtual environment.

• Sphinx for writing documentation.

• Behave for acceptance tests.

• Pytest for unit tests. The pytest-cov plug-in can help to compute test coverage.

• Pip-Tool for building a few working files from the pyproject.toml project definition.

• Tox for running the suite of tests.

• Mypy for static analysis of the type annotations.

• Flake8 for static analysis of code, in general, to make sure it follows a consistent style.

One of the deliverables is the pyproject.toml file. This has all of the metadata about the project in a single place. It lists packages...

1.5 Project 0 – Hello World with test cases

This is our first project. This project will demonstrate the pattern for all of the book’s projects. It will include these three elements.

• Description: The description section will set out a problem, and why a user needs software to solve it. In some projects, the description will have very specific details. Other projects will require more imagination to create a solution.

• Approach: The approach section will offer some guidance on architectural and design choices. For some projects there are trade-offs, and an Extras section will explore some of the other choices.

• Deliverables: The deliverables section lists the expectations for the final application or module. It will often provide a few Gherkin feature definitions.

For this initial project, the description isn’t going to be very complicated. Similarly, the approach part of this first project will be brief. We’ll dwell on the deliverables with some additional...

1.6 Summary

In this chapter, we’ve looked at the following topics:

• An overview of the software quality principles that we’ll try to emphasize.

• A suggested approach to completing the project as a sequence of project sprints.

• A general overview of the list of deliverables for each project.

• The tools suggested for creating these examples.

• A sample project to act as a template for subsequent projects.

After creating this initial project, the next chapter will look at the general collection of projects. The idea is to create a complete data analysis tool set with a number of closely-related projects.

1.7 Extras

Here are some ideas for you to add to this project.

1.7.1 Static analysis - mypy, flake8

There are several common static analysis tools that are as essential as automated testing:

• mypy checks type annotations to be sure the functions and classes will interact properly.

• flake8 does other syntax checks to make sure the code avoids some of the more common Python mistakes.

• black can be used to check the formatting to make sure it follows the recommended style. The black application can also be used to reformat a new file.

• isort can be used to put a long collection of import statements into a consistent order.

Once the application passes the functional tests in the *.feature files, these additional non-functional tests can be applied. These additional tests are often helpful for spotting more nuanced problems that can make a program difficult to adapt or maintain.

1.7.2 CLI features

The command-language interface permits a single option, the --who option, to provide a...