Choosing between Data Fabric and Data Mesh

Just like Data Fabric, Data Mesh has become a buzzword in data management. Data Mesh focuses on principles such as managing data as a product, Self-Service, and providing a federated model at the organization and Data Governance level. The premise is to dismantle the concept of relying on central teams for data operations and move away from centralized data management to a decentralized approach. The Data Fabric and Data Mesh architecture approaches are often confused or referred to interchangeably; however, they are two separate design concepts. The question often asked is, do we need to choose one over the other? The answer is that they are complementary, and Data Fabric can be used together with Data Mesh.

In this chapter, we’ll focus on discussing how Data Fabric and Data Mesh share similar best practices and principles, and we’ll look at where they differ. I’ll provide a perspective of Data Mesh based on Zhamak...

Introducing Data Mesh

Data Mesh is a design concept based on federated data and business domains. It applies product management thinking to data management with the outcome being Data Products. It’s technology agnostic and calls for a domain-centric organization with federated Data Governance. Data management and local governance are owned and implemented at the business domain level. Cross-cutting business domain policies are created by a federated governance team at a global level but executed locally by the domain teams. The business pain points Data Mesh focuses on are resolving organizational bottlenecks, data silos, data swamps, and alignment between business domains and Data Governance teams. The following is Dehghani’s definition of Data Mesh:

Data mesh is a decentralized sociotechnical approach to share, access, and manage analytical data in complex and large-scale environments within or across organizations.

In her book, Dehghani positions Data Mesh as...

Comparing Data Fabric and Data Mesh

Data management best practices and principles have evolved over decades to address common pain points faced today with managing data and getting access to data quickly with the goal of achieving high business value. Organizations have learned the value of data through pain, failure, and revenue loss. This has led to an organizational culture change where data is no longer treated as a by-product but instead as a valuable asset, and data is seen as a product that needs to be treated with care, strategic planning, and much attention to ultimately achieve data monetization.

The reality is that both the Data Fabric and Data Mesh designs have evolved from lessons learned from previous data management approaches. They are based on a collection of best practices and principles to execute superior data management and achieve high-scale data sharing. They address the growing needs and complexities of data management across the four Vs – volume,...

Data Fabric and Data Mesh’s friendship

Data Fabric focuses on Self-Service data access via active metadata leveraging a composable set of tools and technologies. It offers the ability to discover, understand, and access data across hybrid and multi-cloud data landscapes with automation and Data Governance. It is primarily process and technology centric with flexibility in supporting diverse organizational models. On the other hand, Data Mesh is organizationally and process driven. It requires a technical implementation approach to execute its design. Data Mesh is at a higher level and Data Fabric is at a lower level. Data Fabric is capable of fulfilling Data Mesh’s key principles.

Figure 3.2 provides a conceptual view where both the Data Fabric and Data Mesh designs are applied together in a Self-Service data platform.

Figure 3.2 – Data Fabric with a Data Mesh Self-Service data platform

In the next four sections, I’ll provide...

Summary

Data Fabric and Data Mesh are two separate but harmonious data architecture approaches. Both approaches represent sophisticated designs that speed up data delivery while ensuring data can be trusted and relied upon with the goal of extracting business value.

In this chapter, we highlighted how Data Mesh takes an organizational approach to managing data while Data Fabric takes a technical, metadata-driven approach. We identified how Data Fabric and Data Mesh focus on addressing data silos, data access bottlenecks, quality, security, and scalability issues. Both recognize the criticality of having strong Data Governance with embedded automation in the life cycle management of data. Data Fabric can be used together with Data Mesh to realize its core principles (domain ownership, data as a product, Self-Service data platform, and federated computational governance). We discussed how one approach does not need to be selected over the other. Both Data Fabric and Data Mesh can...