Chapter 11: Developing and Maintaining the Presentation Layer

In the previous chapter, Chapter 10, Loading the Presentation Layer, and this one, we investigate how to implement and maintain the Presentation Layer of your Modern Data Warehouse (MDWH).

With all the versatile modules of your modern data estate, you have acquired data, transformed it, and even used advanced analytics to predict behavior based on the data that you have collected.

In this chapter, we will examine how to use Azure Synapse, and particularly Synapse Studio, when you implement your Presentation Layer. You will see how to integrate Azure Synapse with Azure DevOps and how you can automate your deployments. In your role as an MDWH developer, you will also enjoy the developer productivity features that Synapse Studio offers.

You will see how to implement backup and Disaster Recovery (DR) and how you can monitor your Azure Synapse environment.

Finally, you will learn about the two aspects of security...

Developing with Synapse Studio

One of the most important areas for efficient development is the capability to collect all your sources in a central repository, collaborate on artifacts during development, and automate deployments from development to test to production environments.

Synapse Studio is no exception here and integrates with either Azure DevOps or GitHub to enable these features. Other repositories are not yet supported.

Integrating Synapse Studio with Azure DevOps

Let's examine how to integrate your Synapse Studio with Azure DevOps and version your work.

First, you will need to connect your Azure Synapse workspace to your Azure DevOps environment. In Chapter 5, Integrating Data in Your Modern Data Warehouse, you saw how to do this with Azure Data Factory. With Azure Synapse, it is pretty much the same. Let's take these steps:

1. Please navigate to your DevOps environment (https://dev.azure.com/) and create a new project. Name it something such...

Backing up and DR in Azure Synapse

There are several aspects that you need to consider when you are planning backup and DR in your Azure Synapse workspace. You'll need to take care of three main areas:

• Data
• Developed artifacts
• Infrastructure setup

Backing up data

You will first look after your data and take care that you don't lose your biggest assets here. The two areas that you want to consider could be the following:

• Storage accounts/data lakes
• Dedicated SQL pools

As the serverless SQL pools and Spark pools are talking to the data lake, mainly you will cover those by backing up the data lake.

Backing up storage accounts/data lakes

A data lake/storage account can be prepared for DR in different ways. There are the following:

• Automated redundancy built into the service. You have already learned about this in Chapter 3, Understanding the Data Lake Storage Layer. You can decide on the zone or geo-redundant storage...

Monitoring your MDWH

Monitoring your workloads with different services on Azure might be a challenging idea when you consider the isolated service. But as mentioned in other chapters already, all Azure services that we have touched on integrate with Azure Monitor. Their logs and metrics can be sent to Log Analytics and can be analyzed together with other services at a higher level.

When we check the Azure Synapse-dedicated SQL pools, for example, you will have the option to select and push from the following log information to Log Analytics (and two other options; see the following):

• SQLRequests: These are insights about the queries run in the pool, especially the distribution of the steps.
• RequestSteps: Here, you will find the details about the query steps.
• ExecRequests: High-level information about the queries.
• DmsWorkers: Insights about the worker nodes that are executing steps during query execution.
• Waits: Wait states during query runs; you will also...

Understanding security in your MDWH

When you are using Azure services, there are always two aspects regarding security. You can set up access control where you grant or revoke Role-Based Access Control (RBAC) roles or Access Control Lists (ACLs).

We have touched on these concepts already in Chapter 3, Understanding the Data Lake Storage Layer, and in other chapters too when we have set up services and their connections.

The other perspective in the security topic is networking, such as when you want to hide your services completely from the outside world and the so-called public internet. You can peer your on-premise network to Azure Virtual Network. Typically, you will set up a so-called landing zone from where you will route traffic to the target services, such as your data lake, for example, or your Azure Synapse workspace with its computes.

Additionally, you will then implement IP firewall rules for the services that you are securing.

Implementing access control

...

Summary

This chapter took you through the integration between Azure Synapse and Azure DevOps. You learned how to connect Azure Synapse to Azure DevOps and how to automate deployments when creating new, or updating existing, artifacts.

You gained some insights into developer productivity with Synapse Studio and how to benefit from supporting functions of Synapse Studio.

Afterward, you examined how to implement backup and DR and how to monitor your Azure Synapse environment.

Finally, you saw and examined the security features of Azure Synapse.

In the upcoming chapter, Chapter 12, Distributing Data, you will learn how to connect your Power BI service with Azure Synapse and create Power BI reports directly in Synapse Studio.

Further reading

• Continuous integration: https://docs.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-deployment
• Native predict with dedicated SQL pools: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/sql-data-warehouse-predict
• Dedicated SQL pools user-defined snapshots: https://docs.microsoft.com/en-us/powershell/module/az.sql/new-azsqldatabaserestorepoint
• Backup and restore: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/backup-and-restore
• Dedicated SQL pool metrics: https://docs.microsoft.com/en-us/azure/synapse-analytics/monitoring/how-to-monitor-using-azure-monitor
• Synapse workspace RBAC roles:
• a) https://docs.microsoft.com/en-us/azure/synapse-analytics/security/synapse-workspace-synapse-rbac-roles
• b) https://docs.microsoft.com/en-us/azure/synapse-analytics/security/synapse-workspace-understand-what-role-you-need
• SQL authentication: https://docs.microsoft.com...