Reader small image

You're reading from  Data Engineering with Apache Spark, Delta Lake, and Lakehouse

Product typeBook
Published inOct 2021
PublisherPackt
ISBN-139781801077743
Edition1st Edition
Right arrow
Author (1)
Manoj Kukreja
Manoj Kukreja
author image
Manoj Kukreja

Manoj Kukreja is a Principal Architect at Northbay Solutions who specializes in creating complex Data Lakes and Data Analytics Pipelines for large-scale organizations such as banks, insurance companies, universities, and US/Canadian government agencies. Previously, he worked for Pythian, a large managed service provider where he was leading the MySQL and MongoDB DBA group and supporting large-scale data infrastructure for enterprises across the globe. With over 25 years of IT experience, he has delivered Data Lake solutions using all major cloud providers including AWS, Azure, GCP, and Alibaba Cloud. On weekends, he trains groups of aspiring Data Engineers and Data Scientists on Hadoop, Spark, Kafka and Data Analytics on AWS and Azure Cloud.
Read more about Manoj Kukreja

Right arrow

Verifying aggregated data in the gold layer

Assuming the previous invocation of electroniz_batch_aggregation_pipeline was successful, you should see the following external tables and views in the silver container of the Azure Data Lake Storage account:

  1. Using the Azure portal, navigate to Home > All Resources > trainingsynapse100.

    Now, click on Open in the Open Synapse Studio section:

    Figure 8.31 – The Open Synapse Studio section

  2. Using the menu on the left, click on Data. Now, keep clicking on the arrow beside Database, then gold (SQL) and External Tables.

    You should now see the following pane:

    Figure 8.32 – Silver layer external tables in the Synapse serverless SQL pool

    Figure 8.32 – Silver layer external tables in the Synapse serverless SQL pool

  3. Similarly, the following external tables and views represent the aggregated data in the gold layer:

    Figure 8.33 – Gold layer external tables and views in the Synapse serverless SQL pool

  4. Let's go even further and fetch data from a view. To query the view, click on the three...
lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Data Engineering with Apache Spark, Delta Lake, and Lakehouse
Published in: Oct 2021Publisher: PacktISBN-13: 9781801077743

Author (1)

author image
Manoj Kukreja

Manoj Kukreja is a Principal Architect at Northbay Solutions who specializes in creating complex Data Lakes and Data Analytics Pipelines for large-scale organizations such as banks, insurance companies, universities, and US/Canadian government agencies. Previously, he worked for Pythian, a large managed service provider where he was leading the MySQL and MongoDB DBA group and supporting large-scale data infrastructure for enterprises across the globe. With over 25 years of IT experience, he has delivered Data Lake solutions using all major cloud providers including AWS, Azure, GCP, and Alibaba Cloud. On weekends, he trains groups of aspiring Data Engineers and Data Scientists on Hadoop, Spark, Kafka and Data Analytics on AWS and Azure Cloud.
Read more about Manoj Kukreja