Preface

Azure Data Explorer (ADX) enables developers and data scientists to make data-driven business decisions. This book will help you rapidly get insights from your applications by querying data at scale and implementing best practices for securing your ADX clusters.

The book begins by introducing ADX and discussing its architecture, core features, and benefits. You'll learn how to securely deploy ADX instances and be comfortable navigating and using the ADX Web UI. You'll focus on data ingestion and how to query and visualize your data using the powerful Kusto Query Language (KQL). You'll cover KQL operators and functions to efficiently query and explore your data. You'll learn to perform time series analysis and how to search for anomalies and trends in your data. Later, you'll focus on advanced ADX topics, starting with deploying your ADX instances using Infrastructure as Code (IaC). You will manage your cluster performance and monthly ADX costs by handling cluster scaling and data retention periods. Finally, you will cover how to secure your ADX environment by restricting access using subnet delegation and cover some of the best practices for improving your KQL query performance.

By the end of this book, you will be able to securely deploy your own ADX instance, ingest data from multiple sources, rapidly query your data, and produce reports with KQL and Power BI.

Who this book is for

This book is for data analysts, data engineers, and data scientists who are responsible for analyzing and querying their team's large volumes of data on Azure. This book will also be helpful for SRE and DevOps engineers that are responsible for deploying, maintaining, and securing the infrastructure. Some previous Azure experience and basic data querying knowledge will be beneficial.

What this book covers

Chapter 1, Introduction to Azure Data Explorer, covers what ADX is, the core features of ADX, and where ADX fits in Microsoft's suite of data services. The chapter then discusses some of the different use cases of when to use ADX and demonstrates how to execute your first KQL query.

Chapter 2, Building Your Azure Data Explorer Environment, explains how to quickly deploy and configure ADX clusters and databases using the Azure portal, PowerShell, and Azure ARM templates. By the end of this chapter, you will be ready to start ingesting and analyzing your data.

Chapter 3, Exploring Azure Data Explorer UI, presents the ADX UI to you. You will spend the majority of your time using the ADX UI to query and analyze your data. By the end of this chapter, you will be familiar with the windows and panes in the ADX Web UI.

Chapter 4, Ingesting Data in Azure Data Explorer, discusses the concept of data ingestion and demonstrates how to ingest data from multiple data sources such as Blob storage and Azure Event Hubs, how to create new table schemas, and explains how data maps to those tables. At the end of this chapter, you will understand how ADX ingests data and how to configure the data ingestion.

Chapter 5, Introducing the Kusto Query Language, introduces you to KQL and demonstrates how to query data. The chapter begins by introducing the language, explains the basics of KQL such as searching, filtering, aggregating, and joining tables. By the end of the chapter, you will know enough KQL to comfortably query data.

Chapter 6, Introducing Time Series Analysis, introduces you to ADX's time series features, beginning by defining what time series analysis is, and then demonstrating how to query your time series data using the make-series operator. Finally, we discuss some of the most important and useful time series functions provided by ADX.

Chapter 7, Identifying Patterns, Anomalies, and Trends in Your Data, builds on the previous chapter by discussing how to detect anomalies and trends in your data. The chapter first begins by introducing some of the anomaly functions available within ADX and then covers some of the machine learning capabilities of ADX.

Chapter 8, Data Visualization with Azure Data Explorer and Power BI, explains and demonstrates how to integrate ADX with Power BI. Power BI is a powerful reporting tool used to share rich graphs and reports. By the end of the chapter, you will know how to integrate ADX with Power BI and how to create reports in Power BI powered by ADX datasets.

Chapter 9, Monitoring and Troubleshooting Azure Data Explorer, teaches you how to monitor your ADX clusters using Azure Monitor and ADX Insights. The chapter teaches you how to configure alerts using KQL and action groups and explains how to troubleshoot issues by enabling the ADX diagnostics and examining those logs using Log Analytics. In the troubleshooting section, we will demonstrate how to troubleshoot and resolve a data ingestion problem.

Chapter 10, Azure Data Explorer Security, discusses how to secure your ADX instances using both identity management and virtual networks with subnet delegation. We begin by explaining why security is important on the public cloud and then we discuss identity management at the management and data plane. Next, we will introduce securing ADX instances using virtual networks and subnet delegation and demonstrate how to filter network traffic using network security groups (NSGs).

Chapter 11, Performance Tuning in Azure Data Explorer, begins by explaining why performance matters, and then discusses the KQL best practices and revisits the ADX architecture to explain how time filtering can provide performance improvements. You will also learn how to monitor the performance of your clusters, queries, and external applications.

Chapter 12, Cost Management in Azure Data Explorer, discusses how to plan and manage production deployments. The chapter first discusses how to manage your clusters and what requirements you should take into consideration when planning your deployment, and finally discusses how to estimate your Azure costs.

To get the most out of this book

To get the most out of the book, we recommend that you create an Azure account and take advantage of Microsoft's 30-day free trial to follow along with the practical examples. We will spend most of our time in the Azure portal, Azure Cloud Shell, and the Data Explorer Web UI. We also recommend that you clone the repository to your local machine and use Visual Studio Code to experiment and modify the code samples.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book's GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Please remember to turn off/deallocate your resources in Azure to avoid incurring extra charges.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Scalable-Data-Analytics-with-Azure-Data-Explorer. If there's an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Code in Action

The Code in Action videos for this book can be viewed at https://bit.ly/3uw1w2U.

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781801078542_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Since we know there is a failed ingestion, the table we are interested in is aptly called FailedIngestion."

A block of code is set as follows:

StormEvents | where State =~ "California"
 | summarize event=count() by EventType | render columnchart

Any command-line input or output is written as follows:

Get-AzRoleDefinition | Select-Object Name, Description

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: "Next, click Review + create. Finally, click Create once the validation is complete."

Tips and Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [e-mail] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Scalable Data Analytics with Azure Data Explorer, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.