Chapter 12
Project 3.8: Integrated Data Acquisition Web Service

In many enterprise applications, data is provided to several consumers. One way to do this is to define an API that provides data (and the metadata) for subsequent use. In this chapter, we guide you through the transformation of Project 2.5 schema information into a larger OpenAPI specification. We will also build a small Flask application that provides the core acquire-cleanse-convert process as a web service.

We’ll cover a number of skills in the chapter:

• Creating an OpenAPI specification for a service to acquire and download data

• Writing a web service application to implement the OpenAPI specification

• Using a processing pool to delegate long-running background tasks

This is a bit of a deviation from a straight path of acquiring and cleaning data. In some enterprises, this deviation is needed to publish useful data to a wider audience.

We’ll begin with a description of the behavior of this RESTful API...

12.1 Description

In Chapter 8, Project 2.5: Schema and Metadata, we used Pydantic to generate a schema for the analysis data model. This schema provides a formal, language-independent definition of the available data. This can then be shared widely to describe the data and resolve questions or ambiguities about the data, the processing provenance, the meaning of coded values, internal relationships, and other topics.

This specification for the schema can be extended to create a complete specification for a RESTful API that provides the data that meets the schema. The purpose of this API is to allow multiple users — via the requests module — to query the API for the analytical data as well as the results of the analysis. This can help users to avoid working with out-of-date data. An organization creates large JupyterLab servers to facilitate doing analysis processing on machines far larger than an ordinary laptop.

Further, an API provides a handy wrapper around the...

12.2 Overall approach

We’ll take some guidance from the C4 model ( https://c4model.com) when looking at our approach.

• Context For this project, the context diagram has several use cases: listing available data, downloading available data, starting a process to acquire data, and checking the status of a process acquiring data.

• Containers Ideally, this runs on a single container that hosts the web service as well as the processing. In some cases, multiple containers will be required because the processing demands are so huge.

• Components There are two significantly different collections of software components: the web service, and the application programs that run in the background to acquire and clean the data.

• Code The acquiring and cleaning applications have already been described as separate projects. We’ll focus on the web service.

We’ll decompose the web service application into several components. The following diagram shows the relationship between the RESTful...

12.3 Deliverables

This project has the following deliverables:

• Documentation in the docs folder

• Acceptance tests in the tests/features and tests/steps folders

• Unit tests for the application modules in the tests folder

• An application for the RESTful API processing

We’ll start by looking at the acceptance test cases, first. They’ll be rather complex because we need to start the RESTful API service before we can access it with a client request.

12.3.1 Acceptance test cases

Back in Chapter 4, Data Acquisition Features: Web APIs and Scraping, specifically Acceptance tests using a SQLite database, we looked at ways to describe a scenario that involved a database service.

For this project, we’ll need to write scenarios that will lead to step definitions that start the RESTful API service.

There’s an important question about setting the state of the RESTful API server. One approach to setting a state is by making a sequence of requests as part of the scenario...

12.4 Summary

This chapter integrated a number of application programs under the cover of a single RESTful API. To build a proper API, there were several important groups of skills:

• Creating an OpenAPI specification.

• Writing a web service application to implement the OpenAPI specification.

• Using a processing pool to delegate long-running background tasks. In this example, we used concurrent.futures to create a future promise of results, and then compute those results.

The number of processes involved can be quite daunting. In addition to the web service, there is a processing pool, with a number of sub-processes to do the work of acquiring and cleaning data.

In many cases, additional tools are built to monitor the API to be sure it’s running properly. Further, it’s also common to allocate dedicated servers to this work, and configure supervisord to start the overall service and ensure the service continues to run properly.

12.5 Extras

Here are some ideas for you to add to these projects.

12.5.1 Add filtering criteria to the POST request

The POST request that initiates acquire processing is quite complicated. See A POST request starts processing to see the processing it does.

We might name the function for this route creation_job_post() to make it clear that this creates jobs to acquire data in response to an HTTP POST request.

The list of tasks in this function includes the following:

1. Check the user’s permissions.

2. Validate the parameters.

3. Build an AcquireJob instance with the parameters.

4. Update the AcquireJob instance with the Future object. The future will evaluate the acquire_series() function that does the work of acquiring and cleaning the data.

5. Return a JSON object with details of the submitted job, as well as headers and a status code to redirect to a request to get the job’s status.

Some RESTful APIs will have even more complicated parameters. For example, users may...