Summary

In this chapter, we addressed some very important issues related to data observability. We focused on learning about the main concepts surrounding data pipelines and how they can be characterized, after which we understood the various types of data pipeline architectures.

Then, we learned how data observability can make a drastic contribution to containing and reducing costs associated with the evolution and maintenance of data pipelines.

After, we analyzed and understood the fundamental role of data lineage and when it is essential to automate the documentation updates, reduce data catalog management, anticipate propagation, mitigate the impacts of a data anomaly, and drastically reduce changing risk.