Realizing a Data Fabric Technical Architecture

An implementation of a Data Fabric architecture consists of an ecosystem of different tools, data systems, and technologies. Beyond that, it requires the right set of capabilities and data processes that support Data Fabric principles. All these pieces work together as a cohesive automated ecosystem driven by a Data Fabric architecture design to create a connected fabric of data. There isn’t one strict path toward the implementation of a Data Fabric architecture. There are several variations of how you might choose to implement it guided by business goals and strategy. What’s important is to leverage the design guardrails discussed in this book to set you on the path toward the realization of a successful Data Fabric technical architecture.

In this chapter, we will cover the technical architecture of a Data Fabric and what tools, technologies, and capabilities are necessary and should be considered for what reasons. A...

Technical Data Fabric architecture

A Data Fabric platform is infused with intelligent metadata that acts as a parallel universe, tracing and understanding the flow of data across its various stages, tools, systems, and processes. You can think about it as the kids’ game “copy me,” where one copies everything another says or does. In the case of metadata, it works in a similar manner except that it also derives insights and knowledge as the metadata is collected. It learns from its navigation and proposes improved approaches to data management. Active metadata is at the center of the Data Fabric universe. It focuses on enabling data that is fit for purpose, secure, and delivered with speed while increasing efficiency. Active metadata captures, from creation to end of life, all data details such as facts, decisions, relationships, and policies, to then take action by influencing its strategic and efficient direction.

A technical architecture facilitates the implementation...

Use cases

The business use cases that could be realized using a Data Fabric solution are numerous. They include Customer 360, migration to the cloud, regulatory compliance, data democratization, and more. The theme throughout this book has been the criticality and emphasis on mature Data Governance, where active metadata is the glue that binds distributed data during its life cycle. Data Fabric is a technical approach that is positioned to address a diverse set of use cases.

At the time of writing this book, there has been a big shift toward distributed data management and achieving this via a Data Mesh, an organizational approach with specific data management principles. The accountability and responsibility of data management are moved to each business domain as opposed to a central IT organization leveraged across business domains. It has a federated Data Governance model that is accountable for global data policies and decision-making in order to adhere to government regulations...

Data Mesh multi-plane requirements

A Data Mesh architecture defines three logical architecture layers, referred to as a multi-plane architecture. Each layer has an expected set of capabilities, personas supported, and required interactions. Let’s discuss the objectives of this logical architecture.

Multi-plane architecture

A Data Mesh multi-plane logical architecture consists of three planes: mesh experience, Data Product experience, and infrastructure utility. The mesh experience plane oversees and manages, as a whole, the connected Data Products as part of the Data Mesh(es). The Data Product experience plane is granularly focused at the Data Product level and manages the Data Product life cycle. The infrastructure utility plane is the technical infrastructure that supports both Mesh and Data Product experience planes. Figure 9.5 illustrates a Data Mesh’s multi-plane data platform.

Figure 9.5 – Data Mesh multi-plane data platform...

Data Fabric with Data Mesh reference architecture

Data Mesh and Data Fabric both aim to achieve similar objectives; their approach is different but complementary. A Data Mesh architecture can build on a Data Fabric’s technical approach. There is a misconception in the market today that only one architecture should be selected to execute data management. The reality is that there should be different styles of architecture incorporated into data management to achieve success. As architects, data practitioners, and technologists, we will have different perspectives and points of view that may spark debates. That’s part of us learning, sharing, and collaborating with one another and helps us grow as professionals. I offer a point of view of how a Data Fabric architecture can be used together with a Data Mesh architecture.

A Data Fabric’s active metadata-driven architecture is a differentiator that complements and accelerates the Data Mesh’s principles. Figure...

Summary

A technical Data Fabric architecture is modular and composable of several tools and technologies. There are three logical layers in a Data Fabric architecture consisting of Data Governance, Data Integration, and Self-Service.

In this chapter, we reviewed capabilities and the kinds of tools that could be used to implement each layer in a Data Fabric architecture. We have also talked about the requirements and assumptions in two use cases: distributed data management via Data Mesh, and regulatory compliance. A reference architecture was presented that offers a point of view on how to apply both Data Fabric with Data Mesh architectures with a federated operational model. These architectures have similar objectives and call for similar capabilities where Data Fabric is active metadata-driven and takes a more technical approach when compared to Data Mesh, which is more organizationally driven.

In the next and last chapter, we will review industry best practices and recap...