Data flow

1. Data is ingested into the bronze layer by the Azure Data Factory pipelines of the Azure data lake and stored in its raw format.
2. Azure Data Factory pipelines or a combination of Azure Data Factory and Azure Databricks cleans and formats the data and moves it to the silver layer of the data lake.
3. The data in the silver layer can be directly consumed by machine learning algorithms via Azure Databricks or Azure Machine Learning.
4. For advanced analytics that needs analytical cubes or a star schema of fact and dimension tables (https://learn.microsoft.com/en-us/power-bi/guidance/star-schema), data from the silver layer is further transformed using Azure Data Factory and Azure Databricks before being moved to the gold layer.
5. Machine learning models trained on data from the silver layer are hosted on an AKS cluster for inferencing by applications.
6. Power BI reads data from the gold or silver layer to build visual dashboards.
7. Data that needs to be shared...