Ingesting the data

Before we move on to our ingestion code, there is some setup you will have to do to create a source from which to ingest. There are multiple ways to set up a Kafka service to provide a source for our process, but we chose to use Azure Event Hubs. However, you should use whichever service is most convenient for you. If you decided to use Azure Event Hubs, you will need to set up the service by following the instructions at the following link: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs.

Once you have an Event Hubs/Kafka service created, you’ll create a namespace and topic that will be used in this example. You’ll also have to load data into that topic so that our ingestion process can consume the data. The code for this is located on our GitHub repository in the Chapter 13 data_generator folder.

The code will be excluded from the text of this book as it’s written in Python. The main reason...