Preface

Azure Databricks provides the latest and older versions of Apache Spark and allows you to integrate with various Azure resources for orchestrating, deploying, and monitoring your big data solution. This book shows you how you can ingest and transform data coming from various sources and formats and build a modern data warehouse solution that meets the demands of near real-time data requirements in the data warehouse.

You will begin with how to spin up an Azure Databricks service and what cluster options are available. You will gain knowledge of how to process data from various files formats and sources, including Kafka, Event Hub, Azure SQL Databases, Azure Synapse Analytics, and Cosmos DB. Once you know how to read and write data from and to various sources, you will be building end-to-end big data solutions using large datasets and streaming data.

Once the big data solution has been built, you will learn how to deploy notebooks to various environments such as UAT and production. Later on, you will learn security aspects associated with data isolation, where you will learn how to restrict access to the data in ADLS that AAD users can see while reading the data from Azure Databricks. Finally, you will learn how to monitor your Azure Databricks cluster utilization by learning about Ganglia reports.

Who this book is for

This book is for anyone who wants to learn how to ingest and process data from various Azure sources using Azure Databricks and build a modern data warehouse. This book takes a recipe-based approach to help you learn how to ingest, process, secure, deploy, and monitor the big data solution you have built on Azure Databricks. Working knowledge of Spark and a familiarity with the basics of Azure should be sufficient to get started on this book.

What this book covers

Chapter 1, Creating an Azure Databricks Service, explains how to create an Azure Databricks service from the Azure portal, Azure CLI, and ARM templates. You will get to understand the different types of clusters in Azure Databricks, while also learning how to add users and groups and how to authenticate to Azure Databricks using a personal access token.

Chapter 2, Reading and Writing Data from and to Various Azure Services and File Formats, explains how to read and write data from and to various data sources, including Azure SQL DB, Azure Synapse Analytics, ADLS Gen2, Storage Blob, Azure Cosmos, CSV, JSON, and Parquet formats. You will be using native connectors for Azure SQL and an Azure Synapse dedicated pool to read and write the data for improved performance.

Chapter 3, Understanding Spark Query Execution, dives deep into query execution and explains how to check the Spark execution plan and learn more about input, shuffle, and output partitions. You will get to understand the different types of joins while working with data frames. Also, you will learn about a few commonly used session-level configurations for changing partitions.

Chapter 4, Working with Streaming Data, explains how to ingest streaming data from HDInsight Kafka clusters, Event Hub, and Event Hub for Kafka, and how to perform certain transformations and write the output to Spark tables and Delta tables for downstream consumers. You will get to know the various options to use while ingesting data from streaming sources.

Chapter 5, Integrating with Azure Key Vault, App Configuration, and Log Analytics, integrates Azure Databricks with Azure resources such as Azure Key Vault and App Configuration to store credentials and secrets that will be read from Azure Databricks notebooks and explains how to integrate Azure Databricks with Log Analytics for telemetry.

Chapter 6, Exploring Delta Lake in Azure Databricks, explains how to use Delta for batch and streaming data in Azure Databricks. You will also understand how Delta Engine will assist in making your queries run faster.

Chapter 7, Implementing Near-Real-Time Analytics and Building a Modern Data Warehouse, explains how to implement an end-to-end big data solution where you read data from streaming sources such as Event Hub for Kafka, as well as from batch sources such as ADLS Gen-2, and perform various data transformations on data and later store the data in destinations such as Azure Cosmos DB, Azure Synapse Analytics, and Delta Lake. You will build a modern data warehouse and orchestrate the end-to-end pipeline using Azure Data Factory. You will be performing near real-time analytics using notebook visualizations and Power BI.

Chapter 8, Databricks SQL, explains how to run ad hoc SQL queries on the data in your data lake. You will get to know how to create SQL endpoints with multi clusters, write queries, create various visualizations in Data Lake, and create dashboards.

Chapter 9, DevOps Integrations and Implementing CI/CD for Azure Databricks, details how to integrate Azure DevOps Repo and GitHub with Databricks notebooks. You will learn how to implement Azure DevOps CI/CD for deploying notebooks across various environments (UAT and production) as well as how to deploy Azure Databricks resources using ARM templates and automate deployment using the Azure DevOps release pipeline.

Chapter 10, Understanding Security and Monitoring in Azure Databricks, covers pass-through authentication in Azure Databricks and how to restrict access to ADLS Gen-2 using RBAC and ACLs so that users can read data to which they have access from Azure Databricks. You will learn how to deploy Azure Databricks in a VNet as well as how to securely access the data in an ADLS Gen-2 storage account.

To get the most out of this book

This book is for data engineers, data analysts, and data scientists who are familiar with Spark and have some knowledge of Azure. If you understand the basics of Spark and want to learn how to build an end-to-end data pipeline in Azure Databricks and productionize it, then this book is for you. Data scientists and business analysts with some knowledge of SQL who want to run ad hoc queries on large volumes of data using Databricks SQL will also find this book useful.

Basic knowledge of Python and PySpark is all you need to understand and execute the code:

If you are using the digital version of this book, we advise you to type the code yourself or access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Azure-Databricks-Cookbook. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781789809718_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "--location -l: The Azure region where the workspace will be created."

A block of code is set as follows:

$appId="sdasdasdsdfsa7-xxx-xxx-xx-xx"
$appSecret="xxxxxxx~.xxxxxxjgx"
$tenantId="xxxxx-xxx-xx-xxxx-xxxcdxxxxxx"
$subscriptionName="Pay As You Go"
$resourceGroup = "CookbookRG

Any command-line input or output is written as follows:

az databricks workspace update

Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "We will select New Job Cluster here, select Edit option and provide the cluster configuration for the new cluster."

Tips or important notes

Appear like this.

Sections

In this book, you will find several headings that appear frequently (Getting ready, How to do it..., How it works..., There's more..., and See also).

To give clear instructions on how to complete a recipe, use these sections as follows:

Getting ready

This section tells you what to expect in the recipe and describes how to set up any software or any preliminary settings required for the recipe.

How to do it…

This section contains the steps required to follow the recipe.

How it works…

This section usually consists of a detailed explanation of what happened in the previous section.

There's more…

This section consists of additional information about the recipe in order to make you more knowledgeable about the recipe.

See also

This section provides helpful links to other useful information for the recipe.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [e-mail].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Azure Databricks Cookbook, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content..