Pathway to Data Observability

Throughout this book, we have seen how data observability can be included in diverse industry standard tools. In the first part of this chapter, we will share a technical roadmap so that you can include data observability in your applications based on what we’ve observed from many past projects that we’ve conducted over the years.

This technical roadmap will also cover what can be done once pure observability has been implemented, as well as what can be added to complete the picture and make the most of it. We will discover the links with data management, AI, ML, and data quality programs.

The second part of this chapter will focus on applying what you have learned in this book by considering a company example and explaining how you can implement data observability in it.

This final chapter covers the following topics:

• Technical roadmap to include data observability
• Project

Technical roadmap to include data observability

In this section, we will describe how you can add step-by-step observability to your data applications while summarizing all the elements covered in this book. We will see what we need to consider when starting with data observability from a technical point of view. We will cover the following topics:

• Allocating the right resources to your data observability project
• Defining clear objectives with the team
• Implementing data observability in applications
• Continuously improving observability
• Scaling data observability

Let’s get started!

Allocating the right resources to your data observability project

To implement data observability in your data applications, allocating the right resources is key to success. Such resources include budget, staffing, tools, and time.

Budget

First of all, defining a budget for data observability can be tricky. From our experience, we often see this budget as...

Implementing data observability in a project

In this section, we will learn how to implement data observability at scale in an organization, using the technical roadmap presented in the first part of this chapter.

Here, we will provide a macro view, at the organization level, of how observability will be progressively adopted. For the micro level, which includes inside a project, pipeline, or data product, please refer to Part 2 – Implementing Data Observability.

We will consider a company called PetCie, a pet store company that’s active in five countries and produces and sells dog food to resellers.

This company currently has three projects:

• Sales reporting: A financial dashboard that’s used by the CFO so that they can follow the day-to-day sales to the resellers. This report is the outcome of a Tableau dashboard that’s linked to a PostgreSQL table and fed by a Spark application. They plan on changing Tableau into a Qlik process this quarter...

Summary

This chapter summarized all the steps a company must consider to deploy data observability in their pipelines. We have seen that the following technical roadmap can be set to ensure the success of the implementation:

1. Allocate the right resources to your data observability project: This consists of evaluating the right budget, staffing the right people, choosing the right tools, and defining a good period to start a first implementation.
2. Define clear objectives with the team: We saw how objectives can be set from a business and a technical point of view.
3. Implement data observability in your applications: During this process, we saw that success can be achieved by going incrementally in the implementation.
4. Continuously improve observability: Data observability is not a one-time process – it must be periodically reviewed to ensure permanent success.
5. Scale data observability: Finally, we saw how observability can be scaled in the company by proposing...