Introduction

In this chapter, we will introduce the use of data in JSON, CSV, and XML formats. This will include the means of parsing and converting this data to other formats, including storing that data in relational databases, search engines such as Elasticsearch, and cloud storage including AWS S3. We will also discuss the creation of distributed and large-scale scraping tasks through the use of messaging systems including AWS Simple Queue Service (SQS). The goal is to provide both an understanding of the various forms of data you may retrieve and need to parse, and an instruction the the various backends where you can store the data you have scraped. Finally, we get a first introduction to one and Amazon Web Service (AWS) offerings. By the end of the book we will be getting quite heavy into AWS and this gives a gentle introduction.

...