Introduction

We have now reached an exciting inflection point in our learning about scraping. From this point on, we will learn about making scrapers as a service using several APIs, microservice, and container tools, all of which will allow the running of the scraper either locally or in the cloud, and to give access to the scraper through standardized REST APIs.60;

We will start this new journey in this chapter with the creation of a simple REST API using Flask-RESTful which we will eventually use to make requests to the service to scrape pages on demand. We will connect this API to a scraper function implemented in a Python module that reuses the concepts for scraping StackOverflow jobs, as discussed in Chapter 7, Text Wrangling and Analysis.

The final few recipes will focus on using Elasticsearch as a cache for these results, storing documents we retrieve from the scraper...