Argentina
Australia
Austria
Belgium
Brazil
Bulgaria
Canada
Chile
Colombia
Cyprus
Czechia
Denmark
Ecuador
Egypt
Estonia
Finland
France
Germany
Great Britain
Greece
Hungary
India
Indonesia
Ireland
Italy
Japan
Latvia
Lithuania
Luxembourg
Malaysia
Malta
Mexico
Netherlands
New Zealand
Norway
Philippines
Poland
Portugal
Romania
Russia
Singapore
Slovakia
Slovenia
South Africa
South Korea
Spain
Sweden
Switzerland
Taiwan
Thailand
Turkey
Ukraine
United States
Behram Irani
In this chapter, we will look at how organizations can build a data platform foundation by creating data lakes on AWS.
We will cover the following main topics:
Before we dive deep into the actual data and analytics use cases and explore how to design data lakes on AWS, it is first important to understand why Amazon Simple Storage Service (Amazon S3) is the preferred choice for building a data lake and why it is used as a storage layer to store all kinds of data in a centralized location.
If you recall from the discussions we had in Chapter 1, an ideal storage for building a data lake should inherently be scalable, durable, highly performant, easy to use, secure, cost-effective, and integrated with other building blocks of the data lake ecosystem. So, we ask a very important question: why choose Amazon S3 as a data lake store?
S3 checks all the boxes on what we look for in a store for building data lakes. Here are some of the features of S3:
The flow of this book is kept in such a way that it helps you get to the end state of building a modern data platform using AWS, with the ultimate goal of solving business use cases. To demonstrate all the building blocks of the data platform, it is important that I assume a fictitious entity and build a story around it. It’s easier to understand concepts if there is steady progression and continuity in the storyline.
For this book, I will consider a financial organization and all its use cases. You can apply most of the design and architecture techniques to other sector use cases too. The bottom line is that organizations may have different business models, but the concepts that go into building a modern data platform on AWS, to a large extent, remain the same irrespective of the business domain. In other words, the same AWS services and functionalities will be leveraged in building any kind of data platform.
Now that we have a broader business use case for setting up a data lake, let’s look at a use case that will help us define what the different layers of a typical data lake are and why they are required.
Use case for creating data lake layers
GreatFin has different LOBs, and within each of these LOBs, multiple personas have different tasks to perform on the data. Each persona may need specific access to different sets of data. They will all need the data to be formatted and stored in a certain way for them to do their day-to-day operations easily. For example, data engineers may need access to the raw source data so that they can profile the data and understand the quality of the data. Data scientists may need access to a standardized form of datasets so that they can do feature engineering for creating machine learning (ML) models. Data analysts may need access to business-friendly datasets so that they can derive insights from the data.
Before we get...
There are two types of data lake patterns, as follows:
Let’s discuss each of them. Note that you can use a hybrid pattern too, depending on your use case.
In a centralized pattern, the business data is stored and accessed from a central location, to be used throughout the enterprise. For example, it may be easy to manage entity information in a centralized location; entity information such as name, address, gender, age, and profession of a person. It’s easier to manage such datasets in a centralized way, from a governance point of view as well as to avoid data duplication.
Certain LOBs may have additional properties of the data that are relevant only to their use cases. For example, the marketing department may also want to see customer lifetime value (CLV), net promoter score (NPS), marketing preferences, and so on for a person. These additional attributes can then...
We talked about a data lake in AWS being a combination of the data in S3 buckets and the metadata of this data stored in a catalog. We will solve the mystery of creating a technical catalog in AWS by introducing another critical service for building a modern data platform, AWS Glue—a serverless data integration service. Now, Glue is actually an umbrella service consisting of multiple parts. It has the Glue ETL part, which is used for building data integration work, and we have multiple chapters on data ingestion and integration. The component of Glue that is relevant to our data catalog discussion is Glue Data Catalog. Let’s unfold more about the catalog in Glue and how it helps with our data lake in S3.
As the data passes through layers of the data lake in S3, the metadata of the data is captured and stored in Glue Data Catalog. It creates and stores the technical metadata in the form of data definition language (DDL) statements...
Let’s introduce this topic with a use case from GreatFin.
Use case for a transactional data lake
GreatFin wants to comply with the right to be forgotten General Data Protection Regulation (GDPR) compliance in Europe. It wants to have the ability in all its systems, including its analytics environments, to easily locate, update, or delete records as and when required.
The need to create transactional data lakes came about due to many business use cases and the challenges associated with them, such as the following:
|
Use Case |
Challenge |
|
Compliance requirements |
Compliance and privacy laws—for example, the GDPR requires the deletion of certain data within a specific timeframe and/or across all datasets |
|
Change data capture (CDC) |
CDC from the source databases and incremental... |
