Continuous Integration and Continuous Deployment (CI/CD)

Continuous integration (CI) is a part of the DevOps process that allows you to automate the building and testing of code every time a team member commits and approves a code change to source code version control such as a GitHub repository. On the other hand, continuous deployment (CD) is the process of building, testing, configuring, and deploying code items/artifacts from the lower environment (such as development) to higher environments (such as test or production). In this chapter, we will learn about application lifecycle management (ALM) or DevOps processes support in Fabric and implement CI/CD to move our code items from lower environments to higher environments. When combined, this provides an effective and robust process for delivering new changes, features, or bug fixes quicker to your end users.

Specifically, we will cover the following topics:

• Understanding the end-to-end flow
• Connecting to a Git repo...

Technical requirements

Azure DevOps is a project management and DevOps tool, and out of the many features it provides, the ability to set up a CI/CD pipeline along with version control is at the heart of it. We will be focusing on these features in this chapter. You can learn more about Azure DevOps and the whole gamut of features it provides here: https://azure.microsoft.com/en-us/products/devops.

Before we get into the details, please create an Azure DevOps project and Git repository by following the instructions at https://learn.microsoft.com/en-us/azure/devops/repos/git/create-new-repo and name it Learn Microsoft Fabric. At this time, your GitHub repository will be empty.

What is Git?

Git is a distributed version control system (VCS) that allows you to back up and version your work, revert to the previous version if needed, collaborate with other developers, and so on. You can learn more about Git and its benefits here: https://learn.microsoft.com/en-us/devops/develop...

Understanding the end-to-end flow

When we create a Git repository, it gets created with a collaboration or main branch. You commit all your reviewed or approved code items in this branch. Next, whenever a developer or team is going to work on a new feature or release, you create a feature or working branch by using the main branch as the source. The team works in this feature branch during development, and once they have completed their work, they create a pull request to request merging changes from the feature branch to the main branch. This typically goes through review, and when it gets approved, these changes are merged into the main branch, as shown in Figure 10.1. This flow represents the CI part of the DevOps process. Based on the approach an organization has taken, this process repeats for a single feature or a single release with multiple features:

Figure 10.1 – End-to-end DevOps for Microsoft Fabric

For the CD part of the DevOps process...

Connecting to a Git repo with Azure DevOps

Native Git integration capability in Fabric makes it easy for developers to collaborate and release changes quickly and continuously. Let’s set this up and get started.

Open the Fabric workspace we created in Chapter 3, Building an End-to-End Analytics System – Lakehouse, click on the ellipsis (…), and then click on Workspace settings, as shown in Figure 10.2:

Figure 10.2 – Reviewing workspace settings

On the Workspace settings pane, click on Git integration on the left and connect the Azure DevOps account. Next, select the organization, DevOps project, and repository you created earlier, specify the main branch, and then click on Connect and sync, as shown in Figure 10.3:

Figure 10.3 – Connecting your workspace to a Git repository

Clicking on Connect and sync in Figure 10.3 will start syncing your code items (such as notebook, lakehouse, and...

Working on a new feature or release

Now, suppose you are going to work on a new feature or a release, and for that, you want to create a feature branch. In the current workspace, you can do that by clicking on Source control and then clicking on Checkout new branch, as shown in Figure 10.6:

Figure 10.6 – Managing source control in workspace

Specify the name of the branch and click on Checkout branch, as shown in Figure 10.7. This will create a new branch from your main branch as the source and connect your workspace to the newly created branch:

Figure 10.7 – Creating a new branch and checking out

If you have a workspace already created but not connected to the Git repository yet, you can create a new branch while connecting to the Git repository, as shown in Figure 10.8, and connect this workspace to this newly created branch:

Figure 10.8 – Creating a new branch from Workspace...

Creating and executing a deployment pipeline

Now that you have learned about how CI works in Fabric, it’s time to switch gears and look at the CD of your code items from a lower environment such as development to higher environments such as test and production.

When you are in the workspace view, you will notice a Create deployment pipeline option at the top of the screen, as shown in Figure 10.16:

Figure 10.16 – Creating a deployment pipeline

Click on it to create a deployment pipeline, specify a name and optional description for it, as shown in Figure 10.17, and then click on the Next button:

Figure 10.17 – Specifying a name and providing an optional description for the deployment pipeline

On the next screen of the wizard, you specify deployment stages, as shown in Figure 10.18. Typically, you will have Development, Test, and Production; however, if you need to, you can add additional stages or remove...

Managing database code for a Fabric data warehouse

Putting items in a Fabric workspace in source control is important for being able to deploy changes between workspaces, but what about code within your data warehouse? A warehouse itself is of no use if changes to the table definitions, stored procedures, and views don’t move as well. A Fabric data warehouse is integrated with several industry-standard tools and will be adding more native support for database objects in the future.

Among the ways to interact with and manage database code are the following:

• Data built tool (dbt): This is an open source framework that creates dbt projects that can then be deployed to different Fabric data warehouses
• SqlPackage: This is a command-line utility for automating database deployment tasks, including extraction of schema, scripting schema updates, and publishing project updates to databases
• SQL Database Projects: This extension gives a project-based approach to development...

Summary

Almost every organization nowadays follows DevOps processes to offer a collaborative platform for its developers to work together and for its ALM. In this chapter, you learned how you can leverage native integration of Git with Fabric for CI and build deployment pipelines for CD as part of your ALM – for delivering new changes, features, or bug fixes quicker to your end users. You now know how to move your code items from a lower environment (development) to a higher environment (test and/or production).

In the next chapter, we will discuss a wide range of AI experiences, called Copilots, in Fabric that help accelerate each step of the analytics journey.