Chapter 13: Managing and Monitoring Synapse Workloads

Microsoft Azure Platform as a Service (PaaS) enables users to manage their resources on the cloud efficiently and with minimal effort. Azure Synapse Analytics provides you with a centralized experience to manage and monitor all the resources and workloads in your Synapse workspace.

In this chapter, we are going to learn how to manage pools, connections, pipelines, security, and source control, all in a single unified space – Synapse Studio. We will also learn how we can monitor different activities happening across the Synapse workspace.

The following topics will be covered in this chapter in order to understand how we can manage and monitor Synapse workloads:

• Managing Synapse resources
• Monitoring Synapse workloads
• Managing maintenance schedules
• Creating alerts for Azure Synapse Analytics

Technical requirements

Before you start orchestrating your data, here are certain prerequisites that you should meet:

• You should have your Azure subscription, or access to any other subscription, with contributor-level access.
• Create your Synapse workspace on this subscription. You can follow the instructions from Chapter 1, Introduction to Azure Synapse, to create your Synapse workspace.
• Create your SQL pool on Azure Synapse. This has been covered in Chapter 2, Consideration for Your Compute Environment.
• Create the logic app in the Azure portal, which is required to create the alert rule. Please refer to the following link (https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-overview) to explore Logic Apps if you have never used it before.
• You must have an Azure DevOps account created in order to configure source control for Azure Synapse Analytics.

Managing Synapse resources

The Manage hub within Azure Synapse Studio allows you to manage your workspace within a unified experience. It gives you the option to create a new pool, pause an existing provisioned pool, or delete your provisioned pools. You can also manage linked services, triggers, and integration runtimes here.

You can also set up Synapse RBAC within the Manage hub on Synapse Studio and you can manage Git configuration for your code artifacts as well. We are going to learn about all these topics in the following sections.

Although we have already covered how to create a SQL or Spark pool in our previous chapters, in the following section, we will try to learn how we can create these pools using Synapse Studio.

Analytics pools

Azure Synapse supports two types of analytics pool, namely, SQL and Spark pools. These pools are basically two different compute environments that can be used independently as per business requirements. We will learn how to manage...

Monitoring Synapse workloads

Monitoring workloads is critical for understanding any unexpected behavior with the workload. Synapse pipelines, triggers, integration runtimes, Apache Spark applications, SQL requests, data flow debugging, and analytical pools can be monitored directly on the Monitor hub of Synapse Studio. You can also change the time range for certain workloads if you want to monitor workloads for a specific time range. This section is going to cover the monitoring of all activities and workloads within Azure Synapse Analytics.

You can glimpse the Monitor hub within Synapse Studio in the following screenshot:

Figure 13.22 – Monitoring various resources of a Synapse workspace in Synapse Studio

Integration is associated with Synapse pipelines, and we will learn how to monitor pipelines, triggers, and integration runtimes in the following section.

Integration

Synapse pipelines has various components that are stitched together...

Managing maintenance schedules

As soon as the SQL pool deployment is completed, SQL pools have an 8-hour primary and 8-hour secondary maintenance window applied by default. During this maintenance window, all the patch updates will happen internally, and you may not be able to access your resources for the time being. You will be notified when the maintenance job is complete so that you can resume your work after that. Thus, you need to make sure that the maintenance schedule is out of your regular business hours. In order to change the maintenance schedules, perform the following steps:

1. Go to your Synapse workspace in the Azure portal.
2. Click on SQL pool and select your SQL pool.
3. Select the Maintenance schedule link on the Overview blade to open the page for maintenance schedule settings.
4. Select Primary maintenance window and Secondary maintenance window settings as per your business requirements and then click on Save to save your changes.

   The following screenshot...

Creating alerts for Azure Synapse Analytics

Azure has provided a built-in functionality to set up alerts for monitoring Azure resources efficiently. Alerts can be set up for an individual Azure resource separately, and in this section, we are going to learn how to set up alerts for Azure Synapse Analytics.

To create an alert, we need to select the target resource we need to monitor, which is called the scope. After that, we need to define conditions for the alert, define the actions that will be invoked when the alert rule triggers, and finally provide details on the alert rule.

Perform the following steps to configure an alert rule for your Synapse workspace:

1. Go to your Synapse workspace in the Azure portal.
2. Click on the Alerts link of the Overview blade of your Synapse workspace.
3. Click on +New alert rule to configure a new alert rule:
   

   Figure 13.26 – Creating a new alert rule for Azure Synapse Analytics

4. Now we need to establish the scope for the alert...

Summary

In this chapter, we learned how we can manage different resources on Azure Synapse. We learned how to manage pipelines, triggers, various activities, and analytical pools within an Azure Synapse workspace. We also learned how we can configure a code repository for our workspace. With the built-in monitoring experience provided in Synapse Studio, we learned how to monitor various workloads and integration-related activities. We learned about creating alert rules for the Synapse workspace. We also got to know that we can modify maintenance windows even after creating the SQL pool as per our business requirements.

In the next chapter, we are going to learn about the best practices for using Azure Synapse Analytics.