Preface

Data engineering is a growing field that focuses on preparing data for analysis. This book uses various Azure services to implement and maintain infrastructure to extract data from multiple sources and then transform and load it for data analysis.

This book takes you through different techniques for performing big data engineering using Microsoft cloud services. It begins by showing you how Azure Blob storage can be used for storing large amounts of unstructured data and how to use it for orchestrating a data workflow. You'll then work with different Cosmos DB APIs and Azure SQL Database. Moving on, you'll discover how to provision an Azure Synapse database and find out how to ingest and analyze data in Azure Synapse. As you advance, you'll cover the design and implementation of batch processing solutions using Azure Data Factory, and understand how to manage, maintain, and secure Azure Data Factory pipelines. You'll also design and implement batch processing solutions using Azure Databricks and then manage and secure Azure Databricks clusters and jobs. In the concluding chapters, you'll learn how to process streaming data using Azure Stream Analytics and Data Explorer.

By the end of this Azure book, you'll have gained the knowledge you need to be able to orchestrate batch and real-time extract, load, transform (ETL) workflows in Microsoft Azure.

Who this book is for

This book is for database administrators, database developers, and ETL developers looking to build expertise in Azure data engineering using a recipe-based approach. Technical architects and database architects with experience in designing data or ETL applications either on-premises or on any other cloud vendor who want to learn Azure data engineering concepts will also find this book useful. Prior knowledge of Azure fundamentals and data engineering concepts is required.

What this book covers

Chapter 1, Working with Azure Blob Storage, covers how to work with Azure Blob storage and understand how it is used when orchestrating a data workflow.

Chapter 2, Working with Relational Databases in Azure, explains how to provision and work with Azure SQL Database.

Chapter 3, Analyzing Data with Azure Synapse Analytics, describes how to provision an Azure Synapse database and ingest and analyze data in Azure Synapse.

Chapter 4, Control Flow Activities in Azure Data Factory, explains how to implement different control activities available in Azure Data Factory.

Chapter 5, Control Flow Transformation and the Copy Data Activity in Azure Data Factory, explains how to work with the Azure Data Factory integration runtime. You'll also learn to use the SSIS package with Azure Data Factory.

Chapter 6, Data Flow in Azure Data Factory, explains how to use Azure Data Factory mapping and wrangling data flow to extract, transform, and load data.

Chapter 7, Azure Data Factory Integration Runtime, details the different integration runtimes available and their use cases.

Chapter 8, Deploying Azure Data Factory Pipelines, describes how to manually and automatically deploy Azure Data Factory pipelines using the Azure portal and Azure DevOps, respectively.

Chapter 9, Batch and Streaming Data Processing with Azure Databricks, covers recipes to perform batch and streaming data processing using Azure Databricks.

To get the most out of this book

You'll need an Azure subscription along with prior knowledge of Azure fundamentals and data engineering concepts.

If you are using the digital version of this book, we advise you to type the code yourself or access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/azure-data-engineering-cookbook. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781800206557_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "The orders.txt file doesn't have headers, so column names will be assigned as _col1_, _col2_, and so on."

A block of code is set as follows:

#get blob reference
$blobs = Get-AzStorageBlob -Container $destcontainername -Context $storagecontext
#change the access tier of all the blobs in the container
$blobs.icloudblob.setstandardblobtier("Cool")
#verify the access tier
$blobs

Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "The Performance tier can be either Standard or Premium."

Tips or important notes

Appear like this.

Sections

In this book, you will find several headings that appear frequently (Getting ready, How to do it..., How it works..., There's more..., and See also).

To give clear instructions on how to complete a recipe, use these sections as follows:

Getting ready

This section tells you what to expect in the recipe and describes how to set up any software or any preliminary settings required for the recipe.

How to do it…

This section contains the steps required to follow the recipe.

How it works…

This section usually consists of a detailed explanation of what happened in the previous section.

There's more…

This section consists of additional information about the recipe in order to make you more knowledgeable about the recipe.

See also

This section provides helpful links to other useful information for the recipe.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [e-mail].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.