Chapter 4
Data Acquisition Features: Web APIs and Scraping

Data analysis often works with data from numerous sources, including databases, web services, and files prepared by other applications. In this chapter, you will be guided through two projects to add additional data sources to the baseline application from the previous chapter. These new sources include a web service query, and scraping data from a web page.

This chapter’s projects cover the following essential skills:

• Using the requests package for Web API integration. We’ll look at the Kaggle API, which requires signing up to create an API token.

• Using the Beautiful Soup package to parse an HTML web page.

• Adding features to an existing application and extending the test suite to cover these new alternative data sources.

It’s important to recognize this application has a narrow focus on data acquisition. In later chapters, we’ll validate the data and convert it to a more useful form. This reflects...

Chapter 4
Data Acquisition Features: Web APIs and Scraping

Data analysis often works with data from numerous sources, including databases, web services, and files prepared by other applications. In this chapter, you will be guided through two projects to add additional data sources to the baseline application from the previous chapter. These new sources include a web service query, and scraping data from a web page.

This chapter’s projects cover the following essential skills:

• Using the requests package for Web API integration. We’ll look at the Kaggle API, which requires signing up to create an API token.

• Using the Beautiful Soup package to parse an HTML web page.

• Adding features to an existing application and extending the test suite to cover these new alternative data sources.

It’s important to recognize this application has a narrow focus on data acquisition. In later chapters, we’ll validate the data and convert it to a more useful form. This reflects...

Chapter 4
Data Acquisition Features: Web APIs and Scraping

Data analysis often works with data from numerous sources, including databases, web services, and files prepared by other applications. In this chapter, you will be guided through two projects to add additional data sources to the baseline application from the previous chapter. These new sources include a web service query, and scraping data from a web page.

This chapter’s projects cover the following essential skills:

• Using the requests package for Web API integration. We’ll look at the Kaggle API, which requires signing up to create an API token.

• Using the Beautiful Soup package to parse an HTML web page.

• Adding features to an existing application and extending the test suite to cover these new alternative data sources.

It’s important to recognize this application has a narrow focus on data acquisition. In later chapters, we’ll validate the data and convert it to a more useful form. This reflects...

4.3 Summary

This chapter’s projects have shown examples of the following features of a data acquisition application:

• Web API integration via the requests package. We’ve used the Kaggle API as an example of a RESTful API that provides data for download and analysis.

• Parsing an HTML web page using the Beautiful Soup package.

• Adding features to an existing application and extending the test suite to cover these new alternative data sources.

A challenging part of both of these projects is creating a suite of acceptance tests to describe the proper behavior. Pragmatically, a program without automated tests cannot be trusted. The tests are every bit as important as the code they’re exercising.

In some enterprises, the definition of done is breezy and informal. There may be a presentation or an internal memo or a whitepaper that describes the desired software. Formalizing these concepts into tangible test cases is often a significant effort. Achieving agreements can become...

4.4 Extras

Here are some ideas for you to add to these projects.

4.4.1 Locate more JSON-format data

A search of Kaggle will turn up some other interesting data sets in JSON format.

• https://www.kaggle.com/datasets/rtatman/iris-dataset-json-version: This data set is famous and available in a number of distinct formats.

• https://www.kaggle.com/datasets/conoor/stack-overflow-tags-usage

• https://www.kaggle.com/datasets/queyrusi/the-warship-dataset

One of these is a JSON download. The other two are ZIP archives that contain JSON-format content.

This will require revising the application’s architecture to extract the JSON format data instead of CSV format data.

An interesting complication here is the distinction between CSV data and JSON data:

• CSV data is pure text, and later conversions are required to make useful Python objects.

• Some JSON data is converted to Python objects by the parser. Some data (like datestamps) will be left as text.

At acquisition time, this doesn’...