Data Governance in GCP

In the information age, data has become essential in businesses and organizations worldwide. Having a data ecosystem that has the ability to collect, store, process, and analyze data is essential for making informed decisions, gaining a competitive edge, and achieving strategic goals. Google Cloud Platform (GCP), with its powerful suite of data management and analysis tools, offers a robust foundation for harnessing the potential of data. However, as data grows in volume and complexity, ensuring its usability, security, and accountability becomes important. This is where data governance in GCP plays a pivotal role.

Data governance is the set of tools, practices, and policies that dictate how data is managed, maintained, and used within an organization. GCP encompasses a wide array of tools, processes, and best practices designed to ensure that data is accurate, secure, and compliant with regulations. Whether you’re a data scientist, a data engineer...

Technical requirements

For this chapter’s exercises, we will need the following services:

• BigQuery
• Dataform
• Dataplex
• Sensitive Data Protection
• Example code from https://github.com/PacktPublishing/Data-Engineering-with-Google-Cloud-Platform-Second-Edition/tree/main/chapter-10

Steps on how to access, create, or configure the technical requirements will be provided later in each exercise.

Introduction to data governance

Data governance is the set of processes, policies, standards, and practices that organizations use to manage their data ecosystem.

The roles involved in data governance vary depending on the size of the organization. In some cases, when the organization is not that big, data engineers are responsible for data governance. In other cases, when the organization is very big and requires a deeper understanding of legal aspects, usually there is a dedicated team called the data governance team.

data governance encompasses a wide spectrum of practices and principles, much like the vast realm of big data. If we want to implement data governance, understanding the underlying motivations behind implementing it is crucial. Implementing data governance with the wrong motivation usually leads to a wasted effort.

The motivations can be defined into three primary pillars:

• Usability
• Security
• Accountability

The preceding three primary...

A deeper understanding of data usability

The first motivation for implementing data governance is that data should be usable. I think the statement alone is already very clear: we build a data platform so that it can be useful. But if we think more about it, how can data be usable by the end users?

The following are the two aspects that we should think about to improve data usability:

• Users should know how to find the data
• Users should easily understand the data

Let’s discuss the first aspect, that is, the users should know how to find the data.

A data engineering team and the data governance team will be successful if only the data that they produce is used by the end users. It’s nearly impossible to achieve this goal if the end users find it difficult to retrieve the data to get the information that they need.

You may think that it does not make sense that the end users can’t find the data. Unfortunately, this is one of the most common...

A deeper understanding of being accountable

The third pillar of data governance is accountability. Accountability for data is established when the processes and track records for all actions that happen in the data ecosystem are clear. In other words, data is not accountable when no one has a clear idea of why and how things happen in your data ecosystem.

The word “clear” can be expanded to some of the aspects:

• Clear traceability
• Clear data ownership
• Data lineage
• Clear data quality process

Clear traceability

Clear traceability means that for whatever event or actions occur on the data, you have a clear view of who does what and when. This is crucially important for examples such as these: events when a table that contains sensitive data is created, a list of queries that take most of the BigQuery capacity in a day, or a user that costs the most queries in a month.

Please note that the main point of this aspect of data governance is...

Summary

In this chapter, we learned about data governance. We started with a general understanding of data governance, where there are many aspects of data governance. To simplify it, we classified these aspects into three pillars: Usability, Security, and Accountability. Using these pillars, we then drill down deeper to understand the principles, the tools, and the practice of implementing them.

To better understand data usability, we learned that we need to make sure the data products that we have created are easy to use, which includes making the data easy to find and easy to understand. We then learn how to use Dataplex search and metadata tagging to support the requirements.

We then moved on to data security, which consists of two main parts, data encryption and data protection. We learned how to manage access control in BigQuery up to the column level using taxonomy and data policies. We also learned how to use SDP to find the sensitive data in our datasets.

Then we...