Chapter 6: Hosting Your Own Azure Pipeline Agent

In the previous two chapters, we looked at setting up continuous integration through Azure Pipelines while using Microsoft-hosted agents. In this chapter, we'll be building a self-hosted agent and updating the pipeline to use our own agent, rather than using the Microsoft-hosted one.

We will first look at the types of pipeline agents available and then dive into the technical specifications of setting up the agent pools. We will also look at how you can use VM scale sets for large-scale Azure DevOps projects.

We'll be covering the following topics:

• Azure pipeline agent overview
• Understanding the types of agents in Azure Pipelines
• Planning and setting up your own pipeline agent in Azure
• Updating your Azure pipeline to use your self-hosted agent
• Using containers as your self-hosted agents
• Planning for scale – using Azure VM scale sets as self-hosted agents

Technical requirements

To follow this chapter, you need to have an active Azure DevOps organization and an Azure subscription to create a VM.

Getting the project pre-requisites ready: This section requires you to have the PartsUnlimited project ready in your own DevOps organization. If you are continuing from the previous chapter, Chapter 5, Running Quality Tests in a Build Pipeline, you should have it already.

If you do not have the project ready in your DevOps org, you can import it using Azure DevOps Demo Generator – https://azuredevopsdemogenerator.azurewebsites.net/:

1. Log in to the Azure DevOps Demo Generator website.
2. Enter a project name and select your DevOps organization.
3. Click on Choose Template and find PartsUnlimited.
4. Once you're ready, click Create Project:

   Figure 6.1 – Creating a sample DevOps project

   It will take a couple of minutes for the project to be imported; you can monitor the progress using the progress bar displayed...

Azure pipeline agent overview

An Azure pipeline agent is the component responsible for executing the tasks defined in the pipeline definition. This agent typically runs inside a VM or container and includes the pre-requisites required for the pipeline to run successfully.

In most cases, you'll need to have an agent in order to run the pipeline. As your project size and the number of developers grows, you will need to have more agents to support the scale.

Each execution of a pipeline initiates a job on one of the agents, and one agent can only run one job at a time. Azure pipeline agents can be hosted in the cloud or on-premises in one of the following compute infrastructures:

• Server or client host (physical or virtual)
• Containers
• Azure VM scale sets (preview)

Azure pipeline agents are grouped into agent pools. You can create as many agent pools as you require.

Important note

Azure Pipelines supports running basic tasks, such as invoking the...

Understanding the types of agents in Azure Pipelines

Azure Pipelines offers two types of agents:

• Microsoft-hosted agents
• Self-hosted agents

Let's look at them in detail.

Microsoft-hosted agents

Microsoft-hosted agents are fully managed VMs, deployed and managed by Microsoft. You can choose to use a Microsoft-hosted agent with no additional pre-requisites or configurations. Microsoft-hosted agents are the simplest and are available at no additional cost.

Every time you execute a pipeline, you get a new VM for running the job, and it's discarded after one use.

Self-hosted agents

Self-hosted agents are servers owned by you, running in any cloud platform or data center owned by you. Self-hosted agents are preferred due to various reasons, including security, scalability, and performance.

You can configure your self-hosted agent to have the dependencies pre-installed, which will help you decrease the time for your pipeline execution.

...

Planning and setting up your self-hosted Azure pipeline agent

In order to use a self-hosted agent with Azure Pipelines, you will need to set up a machine and configure it for your pipeline requirements. Typically, you would choose an OS version best suited for your project, considering the framework, libraries, and build tools compatibility.

For the purpose of this demonstration, we'll be setting up a VM in Azure and will configure it to use a self-hosted agent. You can choose to host your agent server in any cloud or on-premises environment.

Choosing the right OS/image for the agent VM

The first decision you take while setting up the VM is choosing the OS/image for the server depending on your target deployment. If you are deploying in an on-premises environment, you may just select one of the supported OS versions (such as Windows Server 2016) and install the necessary software. In the case of cloud deployments, you have multiple options provided in the form of images...

Updating your Azure pipeline to use self-hosted agents

In this section, we'll take the Azure pipeline scenario covered in the last chapters (PartsUnlimited) and modify it to use our newly created self-hosted agent. This will enable us to use our self-hosted agent to run the pipelines, rather than using Microsoft-provided agents.

Preparing your self-hosted agent to build the Parts Unlimited project

Before we can start using the self-hosted agent, we must prepare it to support building our sample project, PartsUnlimited. The PartsUnlimited project is built using Visual Studio leveraging .NET Framework, Azure development tools and .NET Core, Node.js, and so on. In order to use our self-hosted agent for building the solution, we must install the required dependencies prior to running the pipeline jobs:

1. Log in to your self-hosted agent VM.
2. Download the Visual Studio build tools with this link: https://visualstudio.microsoft.com/thank-you-downloading-visual-studio...

Using containers as self-hosted agents

Azure Pipelines supports using Docker containers as the compute target for running pipeline jobs. You can use both Windows containers (Windows Server Core/Nano Server) and Linux containers (Ubuntu) to host your agents.

In order to connect the container to your Azure DevOps organization, you'll need to pass a few environment variables, such as the agent pool name, personal access token, and so on.

Setting up Windows containers as Azure pipeline agents

In order to use Windows containers as Azure pipeline agents, you need to build the container image first and then run it with your Azure DevOps organization environment variables. Let's look at the process.

Building the container image

Follow these steps to build the container image:

1. Launch Command Prompt and run the following commands:

   mkdir C:\dockeragent
   cd C:\dockeragent

2. Create a new file named Dockerfile (no extension) and update it with the following content...

Summary

In this chapter, we looked at using Microsoft-hosted agents and self-hosted agents to run your Azure pipeline jobs. We dug deep into the process of setting up a self-hosted agent and updated our pipelines to use the self-hosted agent.

We also looked at how you can use Docker containers, Azure container instances, and Azure VM scale sets as your Azure pipeline agents. With this chapter, you should be able to plan and implement the appropriate pipeline agent solution for your projects

In the next chapter, we'll learn about Artifacts in Azure DevOps.