Concepts of data pipelines and data architecture
We rarely think about how water reaches the taps of our homes. After all, we are end users who pay and use the service with certain expectations and have little visibility and interest in what concerns the transport and management of drinking water.
But this is a good moment to stop for a few seconds to understand this process – it is a process that has many similarities with data pipelines.
What is a data pipeline?
To better understand what a data pipeline is, we can compare it to the components that carry water from the basins to our homes:
- There is a basin of water to draw from (the data sources)
- Various mechanisms are needed to recover, purify, and transport water (the data applications)
- The water reaches the taps of our houses (the data destination)
At this stage, let’s define what a data pipeline is. It is the flow of data that starts from one or several places where data is stored...
