The dbt Core workflow for ingesting and transforming data
In this chapter, we will build our first full-blown dbt project, taking data from a source system and making it available for use in reporting, but let’s start with the big picture by describing the reference architecture for a modern data platform.
In Chapter 4, we outlined the data life cycle. In the following image, we will start to visualize the layers of a modern data platform and start to tie them to how we build them with dbt:
Figure 5.1: Layers of a modern data platform
The layers are as follows:
- Data sources: We will use the full abilities of the underlying data platform to load the data coming from our data sources. In the case of Snowflake, this means being able to load data from files stored on major cloud providers, and of course, data residing in other Snowflake databases (DBs), whether our own or shared with us by others, such as partners, customers, or other departments...
