Understanding the medallion architecture

The process of bringing in data, transforming it, and then preparing it for usage is the main focus of this section. There are many different logical models for this process, with various naming conventions. However, Databricks has proposed the medallion architecture, which can serve as a good model for you to use when thinking about data engineering pipelines. Let’s dive into it.

The medallion architecture, as shown in the following figure, is a data processing architecture that leverages a multi-layered approach to organize, refine, and deliver data for analytics and decision-making purposes. Each layer in this architecture serves a specific function and contributes to the overall data pipeline.

Figure 12.8 – The medallion architecture

Let us understand each layer in this architecture:

• The Bronze layer: At the foundation of the medallion architecture is the Bronze layer. This layer is...