Designing Data Governance

The job of the Data Governance layer in a Data Fabric architecture is to govern, protect, secure, and enable the discovery and understanding of data. A Data Governance layer in a Data Fabric builds a connected data ecosystem with metadata, embedding Data Quality, Data Lineage, Data Security, Protection and Privacy, and Master Data Management (MDM) into every phase of the life cycle of data. It is driven by an active metadata-infused framework that enables enforcement and automation.

In this chapter, we’ll review the design of the Data Governance layer in a Data Fabric architecture. We’ll discuss its metadata-driven and Event-Driven Architecture (EDA) patterns. Finally, we’ll step through how the Data Governance layer is applied to the data life cycle.

In this chapter, we’ll cover the following topics:

• Data Governance architecture
• Metadata as a service
• The Data Governance layer
• Data Fabric’s governance...

Data Governance architecture

The Data Governance layer establishes the foundational data architecture for a Data Fabric, which both the Data Integration and Self-Service layers rely on. The following are two key architecture patterns in the Data Governance layer:

• Metadata-driven
• Event-driven

Let’s discuss each of them in detail.

Metadata-driven architecture

Metadata in data management is defined classically as data about data. It provides context and information relevant to data so that it can be understood and used effectively. Metadata collection has a long history, dating back to the late 1900s (https://www.dataversity.net/a-brief-history-of-metadata/). The reliance on and use of metadata since then have increased drastically throughout the years with digital data management. Companies such as Google and Adobe were pioneers in tapping into the value of leveraging metadata to create a world of discoverable and usable data at a grand scale to realize...

Metadata as a service

Metadata as a service needs to be designed into a platform to drive automation and enable data sharing, governance, and Data Integration. The collection of metadata needs to take place throughout the life cycle of data, across all data management functions such as the development and deployment of data, data ingestion, Data Integration, and data consumption until its end of life. Metadata collection at every stage of the data life cycle enables us to make intelligent decisions to increase business value. In the following section, we will dive into some important points about metadata collection.

Metadata collection

To enable data consumption and enforce the right level of Data Governance, data must be discoverable and understood. This starts by collecting metadata wrapped around data. Metadata should be leveraged to create a 360-degree view that represents everything there is to know about assets, whether data assets, data related assets, and Data Products...

The Data Governance layer

The Data Governance layer in a Data Fabric architecture represents different architectural patterns brought together to ultimately create a fabric of connected metadata and data. There are two key components in the design of the Data Governance layer – active metadata and life cycle governance. Each of these components at a physical level can be made up of several microservices that communicate with one another by leveraging different styles of communication such as APIs and event-based communication. The key architectural pattern in the Data Governance layer relies on metadata. From an implementation perspective, several technologies and tools are involved in this space, such as a data catalog, which leverages a streaming platform to communicate metadata updates. Leveraging an event-based tool for metadata updates ensures the latest most up-to-date metadata is collected.

Figure 7.4 represents the Data Governance layer, consisting of active metadata...

The Data Fabric’s governance applied

The data life cycle is typically represented by the journey data takes from when it comes into existence to the point at which it is destroyed. As this applies to the Data Governance layer in a Data Fabric architecture, certain specific steps and considerations are taken. Each of the actions taken respects the Data Fabric principles as part of the overall architecture. Figure 7.6 represents the logical data life cycle phases in which Data Governance must be applied. Data won’t always flow across every phase. However, the Data Governance layer must account for every phase with the necessary capabilities to ensure it is properly protected and secured.

Figure 7.6 – Data life cycle phases

Note

The life cycle of data can be represented in different ways with various points of view depending on the scope and objectives. There isn’t one standard, as all have their own level of focus. Some might...

Summary

Data Governance must be designed end to end across all the phases (Create, Ingest, Integrate, Consume, Archive, and Destroy) in a data life cycle. In this chapter, we defined a Data Governance architecture as being metadata-driven and event-driven We also defined what it means to manage metadata as a service, which focuses on the collection, integration, and storage of distributed metadata to derive insights and take action. We learned that a Data Governance layer consists of two components: active metadata and life cycle governance. We also understand that the brain of the Data Fabric architecture is active metadata, which is represented by a Metadata Knowledge Graph. The life cycle governance component consists of capabilities that are centered around Data Governance pillars such as Data Privacy, Protection, and Security, Data Quality, Data Lineage, MDM, and Metadata Management.

In the next chapter, we will focus on the two other layers in a Data Fabric architecture:...