4.3 Summary

This chapter’s projects have shown examples of the following features of a data acquisition application:

• Web API integration via the requests package. We’ve used the Kaggle API as an example of a RESTful API that provides data for download and analysis.

• Parsing an HTML web page using the Beautiful Soup package.

• Adding features to an existing application and extending the test suite to cover these new alternative data sources.

A challenging part of both of these projects is creating a suite of acceptance tests to describe the proper behavior. Pragmatically, a program without automated tests cannot be trusted. The tests are every bit as important as the code they’re exercising.

In some enterprises, the definition of done is breezy and informal. There may be a presentation or an internal memo or a whitepaper that describes the desired software. Formalizing these concepts into tangible test cases is often a significant effort. Achieving agreements can become...