Configuring data sources
We're all set and ready to roll! We will build the Electroniz lakehouse one layer at a time starting with the bronze layer. The problem is that we do not have any data. Since this is a learning exercise, we will generate our own data; however, most likely, data will preexist for you when you work on actual customer projects.
Data preparation
To proceed with the lakehouse creation, we will create the following data sources with sample data:
- The stores database: This is stored in Microsoft SQL Server.
- E-commerce transactions: These are pushed to Azure Event Hubs.
- The e-commerce website tracking: This is downloaded from an Azure Blob storage location.
There is no need to generate data for the following datasets because they are downloaded in real time from external sources:
- Currency conversion data: This is downloaded from a REST API.
- Geo-location data: This is downloaded from an HTTP source.
