Summary
In this chapter, we learned about various deployment strategies such as time-based and event-based mechanisms. Additionally, using the event-based mechanism, we created the Electroniz master pipeline. After manually testing the master pipeline, we scheduled it to run every hour. We also learned how to monitor pipelines, add durability features, and deal with failure conditions. Finally, we examined how to add alerts to pipelines so that the administrative groups of users are alerted to failures.
The next chapter is very critical because it equips the data engineer with some critical skills. In Chapter 10, Solving Data Engineering Challenges, we will learn about some common challenges faced by modern-day data engineers along with the recommended ways in which to deal with them.
