Root Cause Analysis

Creating rules and expectations is one thing, as it allows you to detect any issues in your data, but troubleshooting is another.

An observed system should give you many clues and means to check the origin of the error, which will lead to efficient data issue resolution.

In a company, resources are key. The team’s time should be dedicated to value creation, not maintenance or troubleshooting under pressure. You need to know how to use the resources efficiently to avoid wasting time and, ultimately, money.

The best way to keep these costs under control is to evaluate them using key performance indicators (KPIs). Some interesting team or project metrics that you may like to follow include the mean time to detect and the mean time to resolve. The former designs the period between the incident’s occurrence and its detection, while the latter describes the amount of time spent resolving the issue. The goal of the head of data, and all data engineering...

Data incident management

When an issue is detected, the team’s productivity can be affected as resources mobilized on issue resolution cannot be used to create value with new projects. Therefore, to avoid working under pressure and troubleshooting the issue in an unsustainable way, the method we propose is as follows:

1. Detect the issue.
2. Evaluate its impact.
3. Find the root cause.
4. Troubleshoot.
5. Avoid future similar issues.

Thanks to observability, each step will be supported by logs, metrics, and traces that you can use to reduce the time the team spends on resolving issues.

Let’s explore each step in more detail.

Detecting the issue

An issue can be detected by several means. Let’s say that, before you read this book, the majority of the issues are reported to you. One of your data providers or internal customers could come to you to signal that something is fishy in the data you are consuming or, even worse, that the data you...

Anomaly detection

In this section, you’ll learn how an anomaly can be detected based on the metrics you have gathered and can be the basis for rules starting from the SLIs.

Anomalies can be inferred from the following:

• Simple indicator deterministic cases
• Multiple indicators deterministic cases
• Time series analysis

Let’s dig into these categories to explain how the observability metrics can be leveraged to find out the root cause and, in fine, new points of attention and rules for your data sources.

Simple indicator deterministic cases

Anomalies can be detected by adding a series of basic checks to the rules based on the type of metrics you gather, as well as the business logic.

By handling missing values effectively, organizations can prevent potential misinterpretations or errors in data analysis. For example, if the data producer or consumer expects no missing values in the data source, a deterministic rule addressing the number of...

Summary

In this chapter, we saw the real value of data observability for data engineers, where they troubleshoot or even firefight issues in their day-to-day jobs. Days or weeks of tedious manual checks can be avoided by adding proactive and at-the-source observability. The observability metrics that are collected by applications moving, reading, and transforming the data are great assets for performing analyses in case any issues occur.

Furthermore, we have seen that the more observable the system is, the easier it is to evaluate the impact of any issue, allowing the team to work efficiently on what requires the most attention. The in-context collected metrics allow us to easily overview the content of the data through the lineage so that we can correctly identify the faulty application or data and fix it faster.

This is only one of the main advantages of implementing data observability. In the next chapter, we will explore how data observability can be used to optimize pipelines...