Building a logical folder structure

We learned about efficient folder structures in Chapter 2, Designing a Data Storage Structure, where we explored the best practices for storing data for batch processing and streaming scenarios. The rule of thumb is to store the data in a hierarchical date folder structure, with the date part added toward the end, as shown here:

{Region}/{SubjectMatter(s)}/{yyyy}/{mm}/{dd}/{hh}/

We can have more intermediate folders in the folder path based on our business requirements. Please refer to Chapter 2, Designing a Data Storage Structure, to refresh your memory on designing efficient folder structures.

To create a container on Azure Data Lake Gen 2, you can use the following command via the Azure CLI:

az storage fs create -n <container name> --account-name <account name> --auth-mode login

Once you have created a container, you can easily create folders in it by using the following command:

az storage fs directory create -n...