Streaming data from log files

Nowadays, most enterprises are maintaining a lot of applications and each application has its log files. Extracting the required information from log files or monitoring the log file data is becoming a cumbersome task for administrators. To salvage some of the workloads from the shoulders of administrators, Azure Databricks is a great help. Using Databricks, you can process the log files and interactively query the log data to extract the required information.

Getting ready

You can follow along by running the steps in the "4-4.StreamingLogFiles" notebook in your local cloned repository in the Chapter04 folder (https://github.com/PacktPublishing/Azure-Databricks-Cookbook/tree/main/Chapter04).

Upload the LogFiles folder in the Common/LogData folder to the ADLS Gen-2 account in the rawdata filesystem under the Log Files folder.

In this recipe, the use case that we are going to examine is the application logs being generated in the ADLS...