Chapter 11: Minimizing Downtime with Rolling Deployments

Ansible is well suited to the task of upgrading or deploying applications in a live service environment. Of course, application deployments and upgrades can be approached with a variety of different strategies. The best approach depends on the application itself, the capabilities of the infrastructure the application runs on, and any promised service-level agreements (SLAs) with the users of the application. Whatever the approach, it is vital that the application deployment or upgrade is controlled, predictable, and repeatable in order to ensure that users experience a stable service while automated deployments occur in the background. The last thing anyone wants is an outage caused by unexpected behavior from their automation tool; an automation tool should be trustworthy and not an additional risk factor.

Although there is a myriad of choices, some deployment strategies are more common than others, and in this chapter, we...

Technical requirements

To follow the examples presented in this chapter, you will need a Linux machine running Ansible 4.3 or a newer version. Almost any flavor of Linux should do—for those interested in specifics, all the code presented in this chapter was tested on Ubuntu Server 20.04 Long-Term Support (LTS) unless stated otherwise, and on Ansible 4.3. The example code that accompanies this chapter can be downloaded from GitHub at https://github.com/PacktPublishing/Mastering-Ansible-Fourth-Edition/tree/main/Chapter11.

Check out the following video to see the code in action: https://bit.ly/3lZ6Y9W

In-place upgrades

The first type of deployment that we'll cover is in-place upgrades. This style of deployment operates on an infrastructure that already exists, in order to upgrade the existing application. This model is a traditional model that was used when the creation of new infrastructure was a costly endeavor, in terms of both time and money.

A general design pattern to minimize the downtime during this type of upgrade is to deploy the application across multiple hosts, behind a load balancer. The load balancer will act as a gateway between users of the application and the servers that run the application. Requests for the application will come to the load balancer, and, depending on the configuration, the load balancer will decide which backend server to direct the requests to.

To perform a rolling in-place upgrade of an application deployed with this pattern, each server (or a small subset of the servers) will be disabled at the load balancer, upgraded...

Expanding and contracting

An alternative to the in-place upgrade strategy is the expand and contract strategy. This strategy has become popular of late, thanks to the self-service nature of on-demand infrastructures, such as cloud computing or virtualization pools. The ability to create new servers on-demand from a large pool of available resources means that every deployment of an application can happen on brand new systems. This strategy avoids a host of issues, such as a buildup of cruft on long-running systems, such as the following:

• Configuration files that are no longer managed by Ansible being left behind
• Runaway processes consuming resources in the background
• Changes being made to the server manually by human beings without updating the Ansible playbooks

Starting fresh each time also removes the differences between initial deployment and an upgrade. The same code path can be used, reducing the risk of surprises when upgrading an application...

Failing fast

When performing an upgrade of an application, it may be desirable to fully stop the deployment at any sign of an error. A partially upgraded system with mixed versions may not work at all, so continuing with part of the infrastructure while leaving the failed systems behind can lead to big problems. Fortunately, Ansible provides a mechanism to decide when to reach a fatal-error scenario.

By default, when Ansible is running through a playbook and encounters an error, it will remove the failed host from the list of play hosts and continue with the tasks or plays. Ansible will stop executing either when all the requested hosts for a play have failed or when all the plays have been completed. To change this behavior, there are a couple of play controls that can be employed. Those controls are any_errors_fatal, max_fail_percentage, and force_handlers, and these are discussed next.

The any_errors_fatal option

This setting instructs Ansible to consider the...

Minimizing disruptions

During deployment, there are often tasks that can be considered disruptive or destructive. These tasks may include restarting services, performing database migrations, and so on. Disruptive tasks should be clustered together to minimize the overall impact on an application, while destructive tasks should only be performed once. The next two subsections explore how you can meet both these targets using Ansible.

Delaying a disruption

Restarting services for a new configuration or code version is a very common requirement. When viewed in isolation, a single service can be restarted whenever the code and configuration for the application have changed, without concern for the overall distributed system health. Typically, a distributed system will have roles for each part of the system, and each role will essentially operate in isolation on the hosts targeted to perform those roles. When deploying an application for the first time, there is...

Serializing single tasks

Certain applications that run multiple copies of a service may not react well to all of those services being restarted at once. Typically, when upgrading this type of application, a serial play is used. However, if the application is of a large enough scale, serializing the entire play may be wildly inefficient. A different approach can be used, which is to serialize only the sensitive tasks (often the handlers to restart services).

To serialize a specific handler task, we can make use of a built-in variable, play_hosts. This variable holds a list of hosts that should be used for a given task as a part of the play. It is kept up to date with hosts that have failed or are unreachable. Using this variable, we can construct a loop to iterate over each host that could potentially run a handler task. Instead of using the item value in the module arguments, we'll use the item value in a when conditional...

Summary

Deployment and upgrade strategies are a matter of taste. Each strategy comes with distinct advantages and disadvantages. Ansible does not declare an opinion about which is better, and therefore it is well suited to perform deployments and upgrades regardless of the strategy. Ansible provides features and design patterns that facilitate a variety of styles with ease. Understanding the nature of each strategy and how Ansible can be tuned for that strategy will empower you to decide on and design deployments for each of your applications. Task controls and built-in variables provide methods to efficiently upgrade large-scale applications while treating specific tasks carefully.

In this chapter, you learned how to use Ansible to perform in-place upgrades and some different methodologies for these, including techniques such as expanding and contracting an environment. You learned about failing fast to ensure that playbooks don't cause extensive damage if an early part of...

Questions

1. What is a valid strategy for minimizing disruption when in-place upgrades are performed?

   a) Use the serial mode to alter how many hosts Ansible performs the upgrade on at one time.

   b) Use the limit parameter to alter how many hosts Ansible performs the upgrade on at one time.

   c) Have lots of small inventories, each with just a few hosts in.

   d) Revoke access to the hosts by Ansible.

2. What is a key benefit of expanding and contracting as an upgrade strategy?

   a) Reduced cloud operating costs.

   b) It fits well with a development-operations (DevOps) culture.

   c) All hosts are newly built for each application deployment or upgrade, reducing the possibility of stale libraries and configurations.

   d) It provides flexibility in your approach to upgrades.

3. Why would you want to fail fast?

   a) So that you know about your playbook errors as soon as possible.

   b) So that you minimize the damage or disruption caused by a failed play.

   c) So that you can debug your code.

   d) So that you can be...