Optimizing Data Pipelines

The importance of data in companies has significantly increased the investments in data platforms by companies. Over time, this has increased companies’ priority of being aware of what their data pipelines do and how they do it and therefore monitoring not only the quality of the outcomes but also the state of health of the pipelines. At the same time, they are also monitoring the usage of the resources and tracking the associated costs.

In this chapter, we will understand how data observability offers us a way to make the governance of our data pipelines scalable and sustainable. First, we will focus on understanding the key data pipelines, their main components, and the types of data pipelines, as well as their characteristics. Then, we will learn how data observability and, in particular, data lineage can be used to manage several aspects of the data pipeline life cycle, such as the costs and the risks.

In this chapter, we’ll cover the...

Concepts of data pipelines and data architecture

We rarely think about how water reaches the taps of our homes. After all, we are end users who pay and use the service with certain expectations and have little visibility and interest in what concerns the transport and management of drinking water.

But this is a good moment to stop for a few seconds to understand this process – it is a process that has many similarities with data pipelines.

What is a data pipeline?

To better understand what a data pipeline is, we can compare it to the components that carry water from the basins to our homes:

• There is a basin of water to draw from (the data sources)
• Various mechanisms are needed to recover, purify, and transport water (the data applications)
• The water reaches the taps of our houses (the data destination)

At this stage, let’s define what a data pipeline is. It is the flow of data that starts from one or several places where data is stored...

Rationalizing the costs

At this point, most companies have been building data pipelines for decades, and what initially started as a simple process of transforming and uploading dashboards has now evolved into real data departments with tens, hundreds, and thousands of people working with data. We started by having and maintaining a few pipelines, but today, we have companies with thousands of pipelines that read and write from thousands of different data sources. Therefore, a critical aspect is governing this ecosystem of data pipelines and data stakeholders as well as governing the associated costs. This is especially true when we speak about cloud data architectures based on Software-as-a-Service (SaaS) being available on demand, a kind of provisioning well known for being difficult to measure, control, and predict costs.

Due to this, rationalizing data pipeline costs has become not only important but crucial to guaranteeing the right return on investment and making data analysis...

Summary

In this chapter, we addressed some very important issues related to data observability. We focused on learning about the main concepts surrounding data pipelines and how they can be characterized, after which we understood the various types of data pipeline architectures.

Then, we learned how data observability can make a drastic contribution to containing and reducing costs associated with the evolution and maintenance of data pipelines.

After, we analyzed and understood the fundamental role of data lineage and when it is essential to automate the documentation updates, reduce data catalog management, anticipate propagation, mitigate the impacts of a data anomaly, and drastically reduce changing risk.