Chapter 12: Cost Management in Azure Data Explorer

Azure provides access to a lot of powerful resources at the tips of our fingers—with a couple of clicks, you can have access to high-performing virtual machines (VMs), Structured Query Language (SQL) servers, and, of course, Azure Data Explorer (ADX). The problem is, if you do not pay close attention to your consumption, the cost can quickly grow, and you may end up with a huge bill at the end of the month. Luckily, Azure provides a couple of very useful features for managing your Azure costs.

In this chapter, we will begin by introducing scaling and how it relates to cost in Azure, then we will learn about the different ADX cluster stock-keeping units (SKUs) and how to select the correct SKU for your use case. We will then discuss ADX's scaling capabilities and what you should be aware of when using either manual or automatic scaling.

Next, we will introduce a useful Azure feature called Azure Advisor. It is, in...

Technical requirements

There are no code examples in this chapter, but I would like to point out that pricing on Azure varies from region to region. All prices listed in the book are for the West Europe region and I have used my local currency, which is the Euro (EUR). Please note that at the time of writing, Azure Advisor only displays cost savings in US Dollars (USD).

Scaling and cost management

One of the aspects that we have not discussed in detail so far is scaling. One of the design principles of cloud computing is elasticity, and Azure allows us to scale our resources on demand. Scaling in the context of elasticity comes in two dimensions. One is vertical scaling, which refers to increasing the specification of a VM or ADX engine node. For instance, changing the engine SKU from Standard_E64i_v3 to Standard_E80ids_v4 is an example of vertical scaling.

The second dimension is horizontal scaling. Horizontal scaling refers to adding more VMs or engines. For instance, increasing the number of engines is a form of horizontal scaling.

ADX can take care of scaling for us and this is referred to as auto-scaling, but there can be up to 10 minutes of downtime. If downtime is an issue, then you can also use manual scaling and decide when to scale your cluster—for instance, you could manually scale your cluster outside of peak hours.

If...

Selecting the correct ADX cluster SKU

We learned in Chapter 1, Introducing Azure Data Explorer, that ADX clusters consist of two VM clusters—the engine cluster, which is primarily responsible for querying our data, and the data management cluster, which is primarily responsible for data ingestion. When we build ADX clusters and specify the cluster size, we are specifying the VM SKU for our engine and data management clusters. The question is: How do we select the right SKU?

Azure provides several configurations, each optimized for different use cases. For instance, if you are ingesting huge volumes of data, then perhaps the storage-optimized SKUs are a better fit for your requirements. On the other hand, if you will be running a lot of jobs and queries, then one of the compute-optimized SKUs might be the right choice.

Before we explore the different SKUs, it is important to know Azure provides clusters with two different service levels. The first is dev/test, which is...

Introducing Azure Advisor

Azure Advisor may be one of the most underrated services on Azure. Azure Advisor provides insights and recommendations, based on your usage, on how to optimize your Azure deployments from a security, cost, performance, and operational excellence perspective. Another great benefit of Azure Advisor is that it is free. Simply search for Azure Advisor under All services in the Azure portal.

As shown in the following screenshot, Azure Advisor provides a clean dashboard showing the number of items that can potentially help optimize your deployment. Each item is ranked based on the impact of the saving:

Figure 12.4 – Azure Advisor

As you can see from the previous screenshot, we have two recommendations under Cost, which could save us USD 1,071 annually, and one Medium impact recommendation under the Operational excellence category. To see more information on the recommendations, simply click on one of the categories to open it....

Introducing Cost Management + Billing

Azure provides some very good cost management services directly in the Azure portal. As mentioned earlier, Azure Advisor analyzes our usage and makes recommendations on how we can save money. As shown in Figure 12.7, the Subscription overview blade provides a rich view in terms of current spending and forecasts.

The blade also provides the ability to download your invoices:

Figure 12.7 – Subscription billing overview

Let us review each of the tiles on the overview blade, as follows:

1. Latest billed amount—This displays the amount of the last invoice. The current invoice can be downloaded using the Download button, and a list of historical invoices can be retrieved by clicking View invoices.
2. Invoices over time—This displays a historical view of the last 4 months' invoices.
3. Shortcuts—This provides some shortcut links that allow you to email the invoices and view your...

Summary

In this chapter, we learned about different ADX cluster SKUs and how to select the correct SKU for your use case. The different SKUs are optimized for data ingestion/storage and high-volume workloads.

We then discussed ADX's scaling capabilities and what you should be aware of when using either manual or automatic scaling, and how workload groups can help reduce the risk of bad-performing queries/jobs impacting how we auto-scale.

Next, we introduced a useful Azure feature called Azure Advisor. Based on our usage patterns, Azure Advisor analyzes our usage patterns and makes cost-saving and performance-related recommendations based on how we use the platform.

Finally, we introduced Azure's cost management and billing features and learned how to view our invoices, generate billing reports and forecasts, and how to configure budgets and alerts so that we are notified when we are close to or have exceeded our budgets.

Before we end the book, I would like to...