Chapter 4: Understanding Azure DevOps Pipelines

When adopting Azure DevOps in your organization, one of the main important decisions you must make is how to define the pipeline of your development process. A pipeline is a company-defined model that describes the steps and actions that a code base must support, from building to the final release phase. It's a key part of any DevOps architecture.

In this chapter, we'll learn how to define and use pipelines with Azure DevOps for building code.

We will cover the following topics:

• Implementing a CI/CD process
• An overview of Azure Pipelines
• Creating and using build agents
• Overview of the YAML format
• Creating a CI/CD pipeline with Azure DevOps
• Retention of builds
• Multi-stage pipeline
• Build pipeline with GitHub repositories
• Using container jobs in Azure Pipelines
• Let's get started!

Technical requirements

To follow this chapter, you need to have the following:

• A valid organization in Azure DevOps
• An Azure subscription where you can create an Azure VM or a local machine on one of these environments so that you can install the build agent software
• Visual Studio or Visual Studio Code as your development environment
• Access to the following GitHub repository for cloning the project: https://github.com/Microsoft/PartsUnlimited

Implementing a CI/CD process

When adopting DevOps in a company, implementing the right DevOps tools with the right DevOps processes is essential and crucial. One of the fundamental flows in a DevOps implementation is the continuous integration (CI) and continuous delivery (CD) process, which can help developers build, test, and distribute a code base in a quicker, structured, and safer way.

CI is a software engineering practice where developers in a team integrate code modifications in a central repository a few times in a day. When a code modification integrated into a particular branch (normally with a pull request, as explained in the previous chapter), a new build is triggered in order to check the code and detect integration bugs quickly. Also, automatic tests (if available) are executed during this phase to check for breakages.

CD is the process that comes after the CI process. In this process, the output of the CI phase is packaged and delivered to the production stage...

Overview of Azure Pipelines

Azure Pipelines is a cloud service offered by the Azure platform that allows you to automate the building, testing, and releasing phases of your development life cycle (CI/CD). Azure Pipelines works with any language or platform, it's integrated in Azure DevOps, and you can build your code on Windows, Linux, or macOS machines.

Azure Pipelines is free for public projects, while for private projects, you have up to 1,800 minutes' (30 hours) worth of pipelines for free each month. More information about pricing can be found here:

https://azure.microsoft.com/en-us/pricing/details/devops/azure-devops-services/

Some important feature of Azure Pipelines can be summarized as follows:

• It's platform and language independent, which means you can build code on every platform using the code base you want.
• It can be integrated with different types of repositories (Azure Repos, GitHub, GitHub Enterprise, BitBucket, and so on).
• Lots...

Understanding build agents

To build and deploy your code using Azure Pipelines, you need at least one agent. An agent is a service that runs the jobs defined in your pipeline. The execution of these jobs can occur directly on the agent's host machine or in containers.

When defining agents for your pipeline, you have essentially two types of possible agents:

• Microsoft-hosted agents: This is a service totally managed by Microsoft and it's cleared on every execution of the pipeline (on each pipeline execution, you have a fresh new environment).
• Self-hosted agents: This is a service that you need to set up and manage by yourself. This can be a custom virtual machine on Azure or a custom on-premise machine inside your infrastructure. In a self-hosted agent, you can install all the software you need for your builds, and this is persisted on every pipeline execution. A self-hosted agent can be on Windows, Linux, macOS, or in a Docker container.

Microsoft-hosted...

Overview of the YAML language

YAML, an acronym for YAML Ain't Markup Language, is a human-readable scripting language used for data serialization and normally used for handling configurations definitions for applications. It can be considered a superset of JSON.

YAML uses indentation for handling the structure of the object's definitions, and it's insensitive to quotation marks and braces. It's simply a data representation language and is not used for executing commands.

With Azure DevOps, YAML is extremely important because it allows you to define a pipeline by using a script definition instead of a graphical interface (that cannot be ported between projects).

The official YAML website can be found here:

http://yaml.org/

A YAML structure is based on key-value elements:

Key: Value # This is a comment

In the following sections, we'll learn how to define objects in YAML.

Scalars

As an example, the following are scalar variables...

Creating a build pipeline with Azure DevOps

Having a build pipeline in place is a fundamental step if you want to implement continuous integration for your code (having your code automatically built and tested on every commit).

The prerequisite to creating a build pipeline with Azure DevOps is obviously to have some code stored inside a repository.

To create a build pipeline with Azure DevOps, you need to go to the Pipelines hub and select the Pipelines action:

Figure 4.15 – Build pipeline creation

From here, you can create a new build pipeline by selecting the New pipeline button. When pressed, you will see the following screen, which asks you for a code repository:

Figure 4.16 – Selecting a repository

This screen is extremely important. From here, you can start creating a build pipeline in two possible ways (described previously):

1. Using a YAML file to create your pipeline definition. This is what happens...

Retention of builds

When you run a pipeline, Azure DevOps logs each step's execution and stores the final artifacts and tests for each run.

Azure DevOps has a default retention policy for pipeline execution of 30 days. You can change these default values by going to Project settings | Pipelines | Settings:

Figure 4.39 – Pipeline retention policy

You can also use the Copy files task to store your build and artifacts data in external storage so that you can preserve them for longer than what's specified in the retention policy:

Figure 4.40 – Copy files task

The YAML definition for this task is as follows:

- task: CopyFiles@2
  displayName: 'Copy files to shared network'
  inputs:
    SourceFolder: '$(Build.SourcesDirectory)'
    Contents: '**'
    TargetFolder...

Multi-stage pipeline

As we explained previously, you can organize the jobs in your pipeline into stages. Stages are logical boundaries inside a pipeline flow (units of works that you can assign to an agent) that allow you to isolate the work, pause the pipeline, and execute checks or other actions. By default, every pipeline is composed of one stage, but you can create more than one and arrange those stages into a dependency graph.

The basic YAML definition of a multi-stage pipeline is as follows:

stages:  
	- stage: Build  
	  jobs:  
	  - job: BuildJob  
	    steps:  
	    - script: echo Build!  
	- stage: Test  
	  jobs:  
	  - job: TestOne  
	    steps:  
	    - script: echo Test 1  
	  - job: TestTwo 
	    steps:&...

Building a pipeline with GitHub repositories

GitHub is one of the most popular platforms for source control management and often, it's quite common to have scenarios where the code is stored inside a GitHub repository and you want to use Azure DevOps for managing CI/CD.

By using Azure DevOps and the Azure Pipeline service, you can also create pipelines for a repository stored on GitHub, thus triggering a build pipeline on every commit in a branch inside the GitHub repository. We will do this by following these steps:

1. To use Azure Pipelines to build your GitHub repository, you need to add the Azure DevOps extension to your GitHub account. From your GitHub page, select the Marketplace link from the top bar and search for Azure Pipelines. Select the Azure Pipelines extension and click on Set up a plan, as shown in the following screenshot:

   Figure 4.44 – Azure Pipelines on GitHub – setup

2. Select the Free plan, click the Install it for free button, and then...

Using container jobs in Azure Pipelines

In this chapter, we saw that when you create a pipeline, you define jobs, and that when the pipeline is executed, these jobs runs on the host machine where the agent is installed.

If you're using Windows or Linux agents, you can also run a job inside a container (in an isolated way from the host). To run a job inside a container, you need to have Docker installed on the agent and your pipeline must have permission to access the Docker daemon. If you're using Microsoft-hosted agents, running jobs in containers is actually supported on the windows-2019 and ubuntu-16.04 pool images.

As an example, this is a YAML definition for using a container job in a Windows pipeline:

pool:
  vmImage: 'windows-2019'
container: mcr.microsoft.com/windows/servercore:ltsc2019
steps:
- script: date /t
  displayName: Gets the current date
- script: dir  
  workingDirectory: $(Agent.BuildiDirectory...

Summary

In this chapter, we provided an overview of the Azure Pipelines service and we saw how to implement a CI/CD process by using Azure DevOps. We also saw how to create a pipeline for code hosted in a repository by using the graphical interface and by using YAML, as well as how to use and create build agents. We then looked at how to create a build pipeline by using the classic editor and by using a YAML definition. We also saw an example of a multi-stage pipeline and how to use Azure DevOps pipelines to build code inside a GitHub repository, before looking at how to use parallel tasks in a build pipeline to improve build performance. Finally, we learned how to create a build agent on Azure Container Instances and how to use a container's jobs.

In the next chapter, we'll learn how to execute quality tests for our code base in a build pipeline.