Search icon
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
Data Engineering with Apache Spark, Delta Lake, and Lakehouse

You're reading from  Data Engineering with Apache Spark, Delta Lake, and Lakehouse

Product type Book
Published in Oct 2021
Publisher Packt
ISBN-13 9781801077743
Pages 480 pages
Edition 1st Edition
Languages
Author (1):
Manoj Kukreja Manoj Kukreja
Profile icon Manoj Kukreja

Table of Contents (17) Chapters

Preface Section 1: Modern Data Engineering and Tools
Chapter 1: The Story of Data Engineering and Analytics Chapter 2: Discovering Storage and Compute Data Lakes Chapter 3: Data Engineering on Microsoft Azure Section 2: Data Pipelines and Stages of Data Engineering
Chapter 4: Understanding Data Pipelines Chapter 5: Data Collection Stage – The Bronze Layer Chapter 6: Understanding Delta Lake Chapter 7: Data Curation Stage – The Silver Layer Chapter 8: Data Aggregation Stage – The Gold Layer Section 3: Data Engineering Challenges and Effective Deployment Strategies
Chapter 9: Deploying and Monitoring Pipelines in Production Chapter 10: Solving Data Engineering Challenges Chapter 11: Infrastructure Provisioning Chapter 12: Continuous Integration and Deployment (CI/CD) of Data Pipelines Other Books You May Enjoy

Summary

In this chapter, we learned about Delta Lake, known to be the driving force behind the modern lakehouse architecture – it truly enables the lakehouse. We learned in detail about some key features of Delta Lake, such as ACID compliance and open standards. After that, we performed common data lake table operations and showed how Delta Lake controls data isolation and concurrency levels.

In Chapter 7, Data Curation Stage – The Silver Layer, we will put the knowledge gained in this chapter to good use. Using Delta Lake, we will create the silver layer of the Electroniz lakehouse. You may recall from previous chapters that the silver layer in the lakehouse stores the curated, deduplicated, and standardized data. The silver layer represents the single source of truth of data, stored in a state that is readily consumable by diverse data analytics workloads.

lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $15.99/month. Cancel anytime}