Introducing DataOps

DataOps is a collaborative data management framework that can be applied to a Data Fabric architecture by managing the stages from the creation to the deployment of data. It offers an operational model for data to deliver quality analytics. DataOps and Data Fabric can be executed independently of one another, but when used together, they create a high-quality, automated, cost-effective, and well-governed data environment on steroids. This chapter will introduce the DataOps framework and its principles. It will also look at its evolution from other popular frameworks such as DevOps, Statistical Process Control (SPC), and Agile. We will discuss the applicability of Data Quality to data observability, a subcomponent of DataOps, and the alignment of DataOps and Data Fabric.

By the end of this chapter, you will understand the basics of DataOps and how it aligns with Data Fabric.

In this chapter, we will cover the following topics:

• What is DataOps?
• DataOps...

What is DataOps?

DataOps is a framework that applies best practices, processes, and technologies using a collaborative approach to achieve fast, high-quality, and cost-efficient data delivery. Similar to Data Fabric, DataOps is not a tool or specific technology. Rather, it’s a set of principles focused on managing data as code, which in turn enables a deep level of automation necessary for scaling out data management. It emphasizes teamwork across a diverse set of data roles working on data analytics to eliminate misalignment between teams. Its bedrock is the ongoing quality monitoring of both data and processes to achieve customer satisfaction and efficiency. It advocates reusability, iterative short deployment cycles, and feedback loops to achieve business and customer excellence.

DataOps is applicable to raw and business-ready data. It streamlines the development, testing, deployment, and monitoring of data and its pipelines by applying proven, successful quality control...

DataOps’ value

DataOps accelerates operations in building and delivering data solutions. It can be viewed as a layer sitting on a Data Fabric architecture that expedites data processing and delivery to achieve an organization’s digital transformation journey. As discussed in Chapter 2, Show Me the Business Value, there are four key ingredients to achieve profitable data monetization:

• Trusted quality data
• Meaningful insights
• Action-oriented business plan
• High execution speed

DataOps with Data Fabric focuses on the delivery of trusted, quality data and meaningful insights, that is, insights that can be acted upon to derive value and reliability. Both the first and second points enable the creation of a lucrative business plan that can be executed on. Time is money, and DataOps and Data Fabric specialize in achieving the preceding four goals quickly.

The following is a summary of the key objectives of a DataOps discipline:

• Customer satisfaction...

Data Fabric with DataOps

To support one of the key principles of a Data Fabric architecture, data are assets that evolve into Data Products, Data Fabric needs to be married with DataOps principles that focus on quality control and efficiencies in the development and delivery of data. DataOps applies principles in how data and data pipelines are developed, such as with embedded automation and orchestration. It expects the testing of data, assumptions, and integrations at a grand scale. It establishes quality controls from initial data transport from the source to the destination(s), and then after that, executes ongoing data monitoring. Data is fluid; the business evolves and requirements change, which requires proactive monitoring and analysis to ensure the successful delivery of data to consumers.

The DataOps phases are iterative and are not always executed in sequence. Sometimes, a phase may be executed multiple times. For example, during testing, you might find data validation...

Summary

Applying a Data Fabric architecture with a DataOps framework dramatically raises the bar to deliver data with high agility, quality, and governance at a lower cost. In this chapter, we provided an introduction to DataOps, its value, and the 18 driving principles. We briefly introduced Agile, SPC, and DevOps from which DataOps borrows many of its principles. We reviewed the distinction between traditional Data Quality and modern Data Quality, and how modern Data Quality can leverage a foundational Data Quality framework. We also defined data observability and its relationship to Data Quality. We concluded by conceptualizing the use of a Data Fabric architecture together with a DataOps framework. Both offer tremendous value and should be applied in the life cycle management of data.

At this point in the book, we have introduced Data Fabric, Data Mesh, DataOps, and foundational Data Governance concepts. In the next chapter, we will focus on a key business artifact, a data strategy...