Ingesting data from Blob storage using Azure Event Grid

In our final example of data ingestion, we will enable streaming on our cluster and use Azure Event Grid and Event Hubs so we can ingest data whenever new files are placed in our storage account's blob container. A blob container is a location on the storage account used to store our data.

For this section, we need to create the following Azure resources:

• A storage account for storing files
• An event grid to emit blob creation events
• An event hub deliver the notification to Azure Data Explorer

Using JSON data, we will demonstrate how to create JSON-based mapping schemas.

When a file is uploaded to the storage account, a blob created event is generated and received by the event grid. The event grid then updates Azure Data Explorer to pull information from the storage account. In our example, the information is a JSON file.

This path of data ingestion is shown in the following figure:

...