Preface

With data quickly becoming an essential asset of any business, the need for cloud data and AI architects has never been higher. The Azure Data and AI Architect Handbook will assist any data professional or academic who is looking to advance their skill set in cloud data platform design. This book will help you understand all the individual components of an end-to-end data architecture and how to piece them together into a scalable and robust solution.

The book introduces core data architecture design concepts and Azure data and AI services. Cloud landing zones and best practices are explained to build up an enterprise-scale data platform from scratch. Next, you will get a deep dive into various data domains, such as data engineering, business intelligence, data science, and data governance. You’ll learn about various methods for ingesting data into the cloud, designing the right data warehousing solution, managing large-scale data transformations, extracting valuable insights, and how to leverage cloud computing and (generative) AI to drive advanced analytical workloads. Finally, you will discover how to add data governance, compliance, and security to a solution.

By the end of this book, you will have gained the necessary expertise to become a well-rounded Azure data and AI architect.

Who is this book for?

This book is for anyone looking to elevate their skill set to the level of an architect. Data engineers, data scientists, business intelligence developers, and database administrators will learn how to design end-to-end data solutions and get a bird’s-eye view of the entire data platform. Although not required, basic knowledge of databases and data engineering workloads is recommended.

What this book covers

Chapter 1, Introduction to Data Architectures, introduces methods of getting business value from data to solidify any long-term data strategy. You will then get an introduction to our architecture reference diagram to give a first glance at what a bare-bones data architecture may look like. You will then learn what challenges businesses can face when retaining an on-premise-only data strategy.

Chapter 2, Preparing for Cloud Adoption, explains the economic and technical benefits of using the Azure cloud and gives an introduction to Microsoft’s Well-Architected Framework (WAF), which is used by all data and AI architects at Microsoft to guarantee high quality in the design of any data platform. Finally, you will learn how to set up a data and AI landing zone to start your journey to the Azure cloud.

Chapter 3, Ingesting Data into the Cloud, discusses the different ways of ingesting data in Azure and various reference architectures (e.g., Lambda, Kappa, and so on) to best match any requirements. You will learn how to land any streaming or batch data in scalable Azure data lakes according to best practices.

Chapter 4, Transforming Data on Azure, covers data pipelines – the key components to move data between on-premises and Azure and between various Azure components. Pipelines can move data based on a specific event, on a schedule, or in near real time, also called a streaming pipeline. You will learn about the various techniques for automating such data pipelines utilizing orchestration of the pipelines as jobs. You will also learn how to handle both batch and streaming data when orchestrating data transformations in Azure.

Chapter 5, Storing Data for Consumption, looks at best practices for early data orchestration and storage design. You will also learn about the different types of data, the requirements for different data serving methods, and the Azure resources that can be used to meet the functional and technical storage requirements for a data platform.

Chapter 6, Data Warehousing, covers the different ways of creating data warehouses in Azure, where every warehouse comes with its own pros and cons. You will learn what metrics are taken into account when choosing the right warehousing option.

Chapter 7, The Semantic Layer, explains how to implement a semantic layer in a data warehouse to improve the ease of use for end/business users. The semantic layer will hide many of the underlying complexities occurring in earlier stages of the data processing, allowing a wider audience to perform queries against the data warehouse.

Chapter 8, Visualizing Data Using Power BI, explains the options for designing enterprise dashboards and reports to render KPIs. You will learn various ways of integrating Power BI with other components of the data platform to allow for fast and easy visualization of key data.

Chapter 9, Advanced Analytics Using AI, looks at how to leverage the Azure AI services to analyze or transform data or generate new data. You will learn key questions to ask yourself to set up a solid AI strategy and get an in-depth view of the Azure OpenAI service, Azure Cognitive Services, and the Azure Machine Learning workspace, along with knowledge of the entire MLOps process.

Chapter 10, Enterprise-Level Data Governance and Compliance, covers data governance, which has quickly become a key component of every cloud data platform at scale. You will learn about core concepts within the world of data governance and how Microsoft Purview addresses many of the needs in this area. Furthermore, you will learn about data governance frameworks to help get you started on your governance journey.

Chapter 11, Introduction to Data Security, looks at how Azure was designed with security in mind. You will learn about the different layers of data security, along with some core Microsoft and Azure services to make security and monitoring airtight.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Dedicated SQL pools allow the usage of INSERT and UPDATE T-SQL statements like relational databases, but also have a lot of dissimilarities.”

A block of code is set as follows:

AttemptedLoginLogs
| where Timestamp >= ago(7d)
| sort by Timestamp, Identity desc

| take 100When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

CREATE TABLE dbo.DimCustomer
(
    CustomerKey INT IDENTITY NOT NULL,
    CustomerAlternateKey NVARCHAR(15) NULL,
    CustomerName NVARCHAR(80) NOT NULL,
    EmailAddress NVARCHAR(50) NULL,
    Phone NVARCHAR(25) NULL,
    StreetAddress NVARCHAR(100),
    City NVARCHAR(20),
    PostalCode NVARCHAR(10),
    CountryRegion NVARCHAR(20)
)
WITH
(
    DISTRIBUTION = REPLICATE,
    CLUSTERED COLUMNSTORE INDEX
);

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “To connect to on-premises files in a folder, use the File system linked service, as illustrated in Figure 3.11.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [e-mail] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packtpub.com.

Share Your Thoughts

Once you’ve read Azure Data and AI Architect Handbook, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

1. Scan the QR code or visit the link below
   

https://packt.link/free-ebook/9781803234861

2. Submit your proof of purchase
3. That’s it! We’ll send your free PDF and other benefits to your email directly