Understanding the core concepts for pipeline orchestration

In Chapter 5, Architecting Data Engineering Pipelines, we architected a high-level overview of a data pipeline. We examined potential data sources, discussed the types of data transformations that may be required, and looked at how we could make transformed data available to our data consumers.

Then, we examined the topics of data ingestion, transformation, and how to load transformed data into data marts in more detail in the subsequent chapters. As we discussed previously, these steps are often referred to as an Extract, Transform, Load (ETL) process.

We have now come to the part where we need to combine the individual steps involved in our ETL processes to operationalize and automate how we process data. But before we look deeper at the AWS services to enable this, let’s examine some of the key concepts around pipeline orchestration.

What is a data pipeline, and how do you orchestrate it?

A simple...