Automating Deployment through Azure Resource Manager and Azure DevOps

Federation is about providing autonomy to each data product owner to make their own decisions about the storage, computing, and sharing of data. However, this autonomy cannot come at a risk to the security and compliance standards of the company. So, how do we ensure that all the landing zones and resources inside those zones are configured as per the enterprise security and privacy policy? How do we ensure that the networking for these landing zones is standardized? The answer to these questions is automation. Building standard deployment templates, maintaining and versioning them, and deploying them through a standardized pipeline is the solution to ensure that even the most complex data mesh is maintained as per enterprise standards with no room for human error.

The networking, security, and actual deployment aspects of various services are beyond the scope of this book. What this book will cover are the techniques...

Azure Resource Manager templates for landing zones

If you have never built landing zones before using Azure Resource Manager (ARM) templates, then I suggest you first start with the landing zone templates provided by the Azure Cloud Adoption Framework (Chapter 3). This will cover all the core requirements, such as networking, security, resource organization, and monitoring, of a landing zone. You can start with the ready deployment template that you can find here: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/landing-zone/implementation-options#environment-development-approaches.

In a data mesh, product teams typically request a landing zone. These requests can come through a central portal or an email with a completed requirements template attached. Irrespective of the mechanism to submit a requirement for a landing zone, you should have a complete list of the services required to build a landing zone before you start building the automation templates.

...

Source code control for ARM templates

Source code control is a system of storing code and system artifacts in a central repository. This central repository is used by the team to do the following:

• Track modifications
• Maintain different versions of the same code
• Collaborate
• Audit changes
• Automate deployment

Microsoft Azure provides two options to create a centralized code repository: Git and Team Foundation Version Control (TFVC). Which one of these two options is best for you? TFVC is a mature source code management system with a rich set of features, as Microsoft has been working on TFVC for many years. GitHub is a more recent acquisition of Microsoft and implements source code with a different architecture. TFVC is centralized. Git is distributed.

One of the important concepts in source code control is branching. Branching involves copying the source code and tagging it as a separate version. From the time a repository is branched, all the changes...

Azure DevOps pipelines for deploying infrastructure

In Chapter 3, in the Streamlining deployment through DevOps section, we learned that DevOps pipelines are made of stages. Each stage is made of steps. These steps can be a task or a script that is executed by an agent. In this section, let’s understand how to construct a DevOps pipeline for infrastructure deployments.

The first step is to create an Azure Pipeline service instance if you don’t already have one. This can be done by going to the Azure Pipelines landing page here: https://azure.microsoft.com/en-us/products/devops/pipelines/.

Click on Start free or Start free with GitHub based on your repository decision in the previous section:

Figure 6.2 – Azure Pipelines

As a part of the setup process, you will be asked to create an organization and a project (see https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/create-organization). Once a project is created...

Summary

In this chapter, we covered the techniques and technologies used for automating the deployment and management of data mesh landing zones. We saw how to standardize the deployments using the ARM templates and then how to automate the deployment and testing of the deployments using Azure DevOps.

The deployment of a landing zone starts with a request coming from a data product team. We saw that this request can be submitted in multiple ways. They can send an email for the details. Or fill out a form (typically a Microsoft Excel template). Or the data mesh team hosts a centralized portal where teams can fill out this form and submit a request. The last option of submitting a request through a central portal is the most convenient way. This central portal can also serve many other purposes and make it very easy for the data product teams, and data mesh IT Ops teams to work together efficiently.

In the next chapter, we will focus on this central data mesh portal. We will discuss...