Examining examples of real-world data pipelines
The data pipeline examples that we have used in this book have been based on common types of transformations and pipelines, but they have been relatively simple examples. As you can imagine, in large organizations, the types of data pipelines that are built can be a lot more complex and may end up processing extremely large sets of data.
In this section, we will examine two examples of more complex data engineering pipelines from two very well-known organizations – Spotify and Netflix. Both of these companies have public blogs that cover software and data engineering, and the details provided about their pipelines in this section have been taken from the public information that’s been made available in a variety of blog posts and articles.
By learning more about these real-world big data pipelines, you can be better prepared for what to expect when you start working with very large datasets. Also, these examples...
