Data Mesh

In this chapter, we will look at the following key topics:

• Data mesh concepts
• Data mesh on AWS
• Data mesh on an Amazon S3-based data lake
• Data mesh on Amazon Redshift

Data mesh concepts

If you recall from Chapter 8, Data Sharing, we kept the important topic of a distributed data lake that spans multiple AWS accounts open-ended. Now is a good time to complete that story. Even today, the vast majority of use cases that require a data lake can be solved by building a centralized data lake. However, as organizations become bigger, new lines of businesses (LOBs) that work as autonomous units become a reality. All these LOBs add more data sources to grow their business units, resulting in the exponential growth of data at the enterprise level.

Sharing data within an enterprise presents its fair share of challenges. Different LOBs have invested in cloud-based data lakes, along with customized analytics solutions, tailored to address their specific business needs. However, these systems are often designed to cater to particular types of data and may not seamlessly translate to other problem domains.

For many large organizations with many LOBs, a centralized...

Data mesh on AWS

To translate the concepts of data mesh to a data platform built using AWS services, we need to look at how the data is ingested, proceeded, and shared for consumption. The core purpose-built AWS analytics services remain the same, each performing specific tasks in the data platform. However, instead of placing all such services inside a single AWS account, they are all spread into different AWS accounts, owned and managed by different teams or business units. These accounts are constantly producing and/or consuming data, with the eventual goal of deriving value for the whole organization.

All the analytics services and architectures we’ve discussed in this book remain the same – it’s just the design philosophy around data production, data sharing, and data governance all become distributed and completely decoupled in nature. Instead of point-to-point data sharing across AWS accounts using bucket and IAM policies, a completely different mechanism...

Data mesh on an Amazon S3-based data lake

If you recall from our previous chapter on data governance, we used AWS Lake Formation (LF) as a tool to provide fine-grained access control to data that resides in the S3 data lake via the Glue Data Catalog. The same LF permissions mechanism can be leveraged to share data but in a cross-AWS account manner, which opens the doors to implementing a true data mesh architecture, where the data lake doesn’t have to be a central repository for the whole enterprise. Each LOB can establish its own data lake on S3 inside its own AWS account. Some LOB accounts will be data owners, meaning they will produce, store, and consume their data for analytics purposes, from their own data lake on S3. However, if another LOB needs access to some datasets that belong to a different LOB, instead of copying data around, both the producer and consumer LOBs can leverage LF’s cross-account sharing mechanism.

Let’s introduce the use case for implementing...

Data mesh on Amazon Redshift

A data mesh is an architecture pattern that’s not just limited to a single kind of analytics setup. A data lake is a prominent architecture that benefits from a data mesh in a large organization setup, with many independent analytics environments. However, data warehouses are also a foundational data store for analytics operations, and many times, data warehouses are the primary driving force of a data platform. Let’s look at how to establish a data mesh architecture using Amazon Redshift and Amazon DataZone.

The Redshift datashare feature plays a huge role in creating a data mesh using just Redshift. Any number of Redshift clusters, in any AWS account and region, can share datasets with other such clusters. This allows data producers to share data just by using SQL statements inside Redshift. Also, the consumers in other Redshift clusters use SQL statements to gain access to such shared assets.

However, distributed federated governance...

Summary

In this chapter, we looked at what data mesh is and how the four principles of data mesh help create a highly distributed, scalable, and governed data platform. AWS analytics services such as Amazon Redshift, S3 data lakes, AWS Lake Formation, and Amazon Athena contribute toward building a data mesh architecture; many features of these services assist in enabling a data mesh pattern.

We then looked at how, using AWS Lake Formation, organizations can create a cross-account permissions model that helps create a data mesh on an S3 data lake. Using Amazon DataZone, the process of publishing and subscribing to data assets become even easier to manage.

Finally, we looked at how you can use the Amazon Redshift datashare feature to create a data mesh pattern by allowing Redshift clusters in different AWS accounts and regions to share data assets. DataZone helps here too by simplifying the process of federated governance and fostering a self-service analytics culture.

In the...

References

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• Amazon Lake Formation cross-account workshop: https://catalog.us-east-1.prod.workshops.aws/workshops/78572df7-d2ee-4f78-b698-7cafdb55135d/en-US/configure-lakeformation/default-catalog-settings
• Data mesh on AWS blog: https://aws.amazon.com/blogs/big-data/design-a-data-mesh-architecture-using-aws-lake-formation-and-aws-glue/.