Summary

In this chapter, we covered the important elements that we need to collect to implement observability at the data level from within the application. This observability was exposed in a data model, where we distinguished several categories of observations.

First is the elements related to the context – that is, what application is running, what version it is using, who created it and who runs it, and where and when it was run. These elements are important to create a structure around the data transformations. Second is the data itself. We saw that the metadata can be defined by some attribute of the data source and its schema. Third are the data transformations and operations, which we have described as lineages. These lineages are also the link between the data sources, their schemas, and their applications. Finally, once we have associated the lineage with the right execution, some observation metrics can be computed.

We also looked at some specific elements related...