Understanding data consumption
Before we start verifying the aggregated data, we should focus on how our end users will be able to consume data for dashboarding, ML, and AI purposes. As per the laid-out architecture of the Electroniz lakehouse, we decided to publish data from both the gold and silver layers.
Publishing data from the gold layer is necessary; otherwise, how would users be able to access aggregated data? But why do we need to publish data from the silver layer? You guessed it – analytics is an ongoing process. In the future, users may want to create new dashboards and ML models for the betterment of the company.
Important
Publishing data from the gold and silver layers is acceptable because they store data that is in a clean and secure state. But the same cannot be said for data in the bronze layer. Publishing raw/unclean data not only throws a lot of work around standardization, validation, and deduplication at end users, but it also ends up exposing...
