You're reading from Driving Data Quality with Data Contracts

Product type Book

Published in Jun 2023

Publisher Packt

ISBN-13 9781837635009

Pages 206 pages

Edition 1st Edition

Languages

Python

Concepts

Data Engineering

Author (1):

Andrew Jones

Table of Contents (16) Chapters

Preface

Part 1: Why Data Contracts?

Chapter 1: A Brief History of Data Platforms

Chapter 2: Introducing Data Contracts

Part 2: Driving Data Culture Change with Data Contracts

Chapter 3: How to Get Adoption in Your Organization

Chapter 4: Bringing Data Consumers and Generators Closer Together

Chapter 5: Embedding Data Governance

Part 3: Designing and Implementing a Data Architecture Based on Data Contracts

Chapter 6: What Makes Up a Data Contract

Chapter 7: A Contract-Driven Data Architecture

Chapter 8: A Sample Implementation

Chapter 9: Implementing Data Contracts in Your Organization

Chapter 10: Data Contracts in Practice

Index

Why subscribe?

Other Books You May Enjoy

Embedding Data Governance

In this chapter, we’re going to look at the rather broad area of data governance. We’ll explore what we mean by data governance, and why it’s so important.

As we’ll see, effective governance of data is critical to an organization, but rarely achieved. With data contracts, we can embed our data governance controls alongside the data. We’ll show you how to do this and how powerful it can be, as it unlocks an array of potential automation and tooling that makes it easy to manage our data.

Finally, we’ll assign responsibility for data governance. As we learned in the previous chapter, it’s by clearly defining roles and responsibilities that we enable groups of people to work together closely and do so effectively with the knowledge of what is expected of them.

These roles and responsibilities look different in a data contract-backed architecture than in more traditional implementations of data governance...

Why we need data governance

We’ll start by discussing what we mean by data governance, what it covers, and why it is needed. Once we have a shared understanding, we’ll look at how we can promote effective data governance through data contracts, and finish by discussing the roles and responsibilities involved.

There are many definitions of data governance, and organizations implement it in different ways. Broadly, it is a combination of people, processes, standards, and technology that supports and promotes data that is accessible, usable, accurate, consistent, secure, and compliant.

The following diagram illustrates the data requirements supported by data governance using a combination of different programs:

Figure 5.1 – Data requirements supported by the different data governance programs

To gain a better understanding of what data governance is and why we need it, let’s explore the following topics in more detail:

...

Promoting data governance through data contracts

Data contracts are the perfect place to promote data governance. In this section, we’ll explore how you can use data contracts to improve the visibility of data governance, collect up-to-date categorization of data, and provide the tooling and guardrails the data generators need to handle their data effectively.

As mentioned in Chapter 2, Introducing Data Contracts, the contract definition can be extended to capture any metadata we need to ensure we’re handling our data in accordance with our standards. This could include the following:

Whether it is personal data.
What entity the data relates to (for example, is it about a customer, an order, or another entity?).
Who has access to the data, and when that access expires.
How the data is classified according to your organization’s policies (for example, is it confidential, secret, or public knowledge?).
How long we keep this data for (the...

Assigning responsibility for data governance

To implement effective data governance in our organization, we need to be clear on the roles and responsibilities involved. In this section, we’ll define those roles and responsibilities and how they work together.

We will cover this in the following subsections:

Responsibilities of the data generators
Introducing a data architecture council
Working together to implement federated data governance

Responsibilities of the data generators

By using a data contracts-backed architecture, we promote a more decentralized operating model. We give data generators the autonomy and responsibility to own and manage their data, supported by the right self-served tooling and guardrails.

Consequently, we need to rethink our approach to data governance. We don’t want to create a central team to try to take control of the data. These teams become a bottleneck, slowing down access to and use of data. They also...

Summary

In this chapter, we looked at what data governance is and discussed why the effective governance of data is critical. This is particularly important when we look at how we handle our data and manage the risks associated with that. But data governance is more than managing risk, and when done well can help promote a data-driven culture in your organization.

We then looked at how, with data contracts, we can embed our data governance alongside the data. This ensures the classifications and other metadata are correct, accurate, and kept up to date as the data evolves. That metadata can also be used to drive tooling and services to support the effective management and handling of our data, ideally by automating a lot of it away.

With that in place, we’re able to assign the responsibility of data governance to the data generators. They are the best placed to carry out that task, as only they have the full context of the data, what it contains, and why we are generating...