Fundamentals of Data Observability

Perfect! You have set up new objectives for the marketing report data so Lucy, the marketing analyst we met in Chapter 1, is delighted. Your team is now aware of the expectations of the marketing team, and the trust relationship between the marketing and data engineering teams can be rebuilt. You are now thinking about ways your data engineering team will monitor quality, and it will start by analyzing the datasets you provide to the business in order to prove they are correct. Every day, you check the quality of the output table in a Snowflake database. Quickly, an error appears. The values in a column seem to have been divided by a factor of 10. What a shame! Lucy is already using the dataset. Hopefully, you can proactively contact her and avoid errors in her report. Nevertheless, you must now fix the issue. This won’t be a piece of cake… Looking at the central data catalog, you are already losing hope… How should you start...

Technical requirements

This chapter contains interactive examples that you can also run. All the code examples for this chapter can be found at https://github.com/PacktPublishing/Data-Observability-for-Data-Engineering.

From data quality monitoring to data observability

The general way of conducting data quality involves manual and automated checks, also called tests, on process inputs and outputs. In this paradigm, on the one hand, the consumer is responsible for checking the validity of their raw material according to their proper needs – for instance, by validating the schema you are receiving. On the other hand, the producer checks the conformity of the output data regarding consumers’ needs by ensuring, for instance, that data manipulation did not deteriorate its completeness. Often, if the data team arranges a well-running data quality program, the inputs won’t be checked by the consumers as they expect the inputs to be already validated.

The following figure explains this model; the data quality process ensures that the inputs and outputs are in line with quality expectations:

Figure 2.1 – Data quality outside the application

...

Three principles of data observability

By definition, observability comes from the system itself and reporting its state to the outside world. In data observability, the root of the system is the data process. A process is almost always an application, so a script, a notebook, or a program, for instance. The application reports its inner activity, which allows the external observer to understand what happens inside the process. More than creating a data product output, a layer of data observability allows the application to produce data on its proper execution.

Data observability comes with three main principles:

Data observability = Contextual + Synchronous + Continuous validation

This is all about avoiding using unnecessary observations, avoiding data error cascade, and ensuring non-propagation of known issues.

Let’s explore these principles in detail:

• Observability must be put into context: This point is about contextualizing your indicators. Indeed, a...

Data observability in IT observability

Observability is not a new asset in the monitoring toolkit. Indeed, making a system observable is already implemented by the operational and DevOps community to report on application or system availability. Observability helps us to monitor complex distributed systems by collecting logs, metrics, or traces.

If we take observability in the broadest sense, every component of the data pipeline can be monitored — not only the data, but the application, the system, or even the users. Data observability helps us to build the bridge between the data engineering world and application and system observability, with some common elements allowing a deeper analysis of data issues in the context of full observability.

Besides the problems that come from the data itself, other issues may arise — for instance, from an unresized cluster, a failing or outdated application, and so on. The combination of all kinds of observability can increase...

Key components of data observability

In this section, we will see some examples of data observability metrics that are collected from inside applications and issues that can be raised from such quality issues. We will focus on detecting issues and to do so, we are going to create visuals of data observability issues in a Jupyter notebook.

If you want to follow the example, you can find it in the Chapter2 section of the GitHub repository. The name of the notebook is Visualise_Observability_Issues.ipynb.

In this part, we will focus on a timeliness, a completeness, and an accuracy issue.

The dataset that we provide is a basic example of marketing and sales data. The data represents the orders made on a web shop and consists of the following fields:

• date: The date of the order
• guid: A unique ID for the order
• email: The email address linked to the order
• page_visited: The number of pages the customer visited on the website
• duration: How long the customer...

Data observability in the enterprise ecosystem

As we have seen in this chapter, through the context, the synchronization with the process, and the continuous validation of data, data observability manages to ground all the potential that often cannot be expressed with traditional data quality approaches. In this section, in particular, we’ll understand how businesses can take direct advantage of it.

Investments in data quality are becoming more and more huge, as well as the perception that keeping control of data quality is already a critical aspect of the success of a business. This sense of a need for protection is transforming into investments in qualified personnel and, often, in expensive and sophisticated tools that promise to help companies acquire the means and skills necessary to guarantee certified data products on which to make reliable decisions on an ongoing basis.

If, on the one hand, it is true that enthusiasm and attention are growing exponentially, on...

Summary

In this chapter, we learned about the ties between data quality and observability. We saw that data quality monitoring by itself is often not sufficient to ensure good data and trust in the pipeline.

We introduced the concept of data observability, which will help the monitoring of the data application by applying these three principles:

• Observability is put into context: Data issues must be put into context in order to avoid interfering with other lineages and applications
• Observability needs synchronicity: The sooner you detect the issue, the better you avoid other applications modifying the data source, as long as the appropriate mechanisms to do so are implemented in your pipeline
• Observability allows continuous validation: Use rules to validate data and ensure data quality at runtime

We learned how to measure the success of our projects and investments in data observability, identifying the fundamental objectives and metrics to monitor in order...