Monitoring pipelines
To monitor the hourly runs using the panel on the left-hand side, click on Monitor. Then, click on Trigger runs:
Figure 9.13 – The first trigger run for the Electroniz master pipeline
Notice how the master pipeline successfully runs every hour to ingest the incremental data and recompute the aggregations. Hopefully, this should make Electroniz a very happy customer. As a data engineer, you should be proud of your achievements. Getting this far involves a lot of work.
In an ideal world after the deployment, pipelines should run automatically and seamlessly. The reality is that failures will happen, so you need to be prepared for them well in advance. After all, your pipelines ingest data from varying sources that may or may not be under your direct control. One day, the database that your pipeline ingests from might be down for maintenance, or a REST API that you subscribe to might go offline.
Important
A smart data engineer...
