Chapter 10: Updating Software in the Field

In previous chapters, we discussed various ways to build the software for a Linux device and also how to create system images for various types of mass storage. When you go into production, you just need to copy the system image to the flash memory, and it is ready to be deployed. Now, I want to consider the life of the device beyond the first shipment.

As we move into the era of the Internet of Things, the devices that we create are very likely to be connected together by the internet. At the same time, software is becoming exponentially more complex. More software means more bugs. Connection to the internet means those bugs can be exploited from afar. Consequentially, we have a common requirement to be able to update software in the field. Software updates bring more advantages than fixing bugs, however. They open the door to adding value to existing hardware by improving system performance over time or enabling features.

In this chapter...

Technical requirements

To follow along with the examples, make sure you have the following:

• A Linux-based host system with a minimum of 60 GB of available disk space
• Yocto 3.1 (Dunfell) LTS release
• Etcher for Linux
• A microSD card reader and card
• A Raspberry Pi 4
• A 5V 3A USB-C power supply
• A Wi-Fi router

Yocto is required for the Using Mender for local updates and Using Mender for OTA updates sections of this chapter.

You should have already built the 3.1 (Dunfell) LTS release of Yocto for Chapter 6, Selecting a Build System. If you have not, then refer to the Compatible Linux Distribution and Build Host Packages sections of the Yocto Project Quick Build guide (https://www.yoctoproject.org/docs/current/brief-yoctoprojectqs/brief-yoctoprojectqs.html) before building Yocto on your Linux host according to the instructions from Chapter 6.

All of the code for this chapter can be found in the Chapter10 folder of the book's GitHub repository...

From where do updates originate?

There are many approaches to software updates. Broadly, I characterize them as
the following:

• Local updates, often performed by a technician who carries the update on a
  portable medium such as a USB flash drive or an SD card and has to access each system individually
• Remote updates, where the update is initiated by the user or a technician locally, but it is downloaded from a remote server
• Over-the-air (OTA) updates, where the update is pushed and managed entirely remotely, without any need for local input

I will begin by describing several approaches to software updates, and then I will show an example using Mender (https://mender.io).

What to update

Embedded Linux devices are very diverse in their design and implementation. However, they all have these basic components:

• Bootloader
• Kernel
• Root filesystem
• System applications
• Device-specific data

Some components are harder to update than others, as summarized in this diagram:

Figure 10.1 – Components of an update

Let's look at each component in turn.

Bootloader

The bootloader is the first piece of code to run when the processor is powered up. The way the processor locates the bootloader is very device-specific, but in most cases, there is only one such location, and so there can only be one bootloader. If there is no backup, updating the bootloader is risky: what happens if the system powers down midway? Consequently, most update solutions leave the bootloader alone. This is not a big problem, because the bootloader only runs for a short time at power-on and is not normally a great source...

The basics of software updates

Updating software seems, at first sight, to be a simple task: you just need to overwrite some files with new copies. But then your engineer's training kicks in as you begin to realize all the things that could go wrong. What if the power goes down during the update? What if a bug, not seen while testing the update, renders a percentage of the devices unbootable? What if a third party sends a fake update that enlists your device as part of a botnet? At the very least, the software update mechanism must be:

• Robust, so that an update does not render the device unusable
• Fail-safe, so that there is a fallback mode if all else fails
• Secure, to prevent the device from being hijacked by people installing
  unauthorized updates

In other words, we need a system that is not susceptible to Murphy's law, which states that if something can go wrong, then it will go wrong, eventually. Some of these problems are non-trivial, however. Deploying...

Types of update mechanism

In this section, I will describe three approaches to applying software updates: symmetric, or A/B, image update; asymmetric image update, also known as recovery mode update; and finally, atomic file update.

Symmetric image update

In this scheme, there are two copies of the operating system, each comprising the Linux kernel, root filesystem, and system applications. They are labeled as A and B in the following diagram:

Figure 10.2 – symmetric image update

Symmetric image updates work as follows:

1. The bootloader has a flag that indicates which image it should load. Initially, the flag is set to A, so the bootloader loads OS image A.
2. To install an update, the updater application, which is part of the operating system, overwrites OS image B.
3. When complete, the updater changes the boot flag to B and reboots.
4. Now the bootloader will load the new operating system.
5. When a further update is installed...

OTA updates

Updating over-the-air (OTA) means having the ability to push software to a device or group of devices via a network, usually without any end user interaction with the device. For this to happen, we need a central server to control the update process and a protocol for downloading the update to the update client. In a typical implementation, the client polls the update server from time to time to check if there are any updates pending. The polling interval needs to be long enough that the poll traffic does not take a significant portion of the network bandwidth, but short enough that the updates can be delivered in a timely fashion. An interval of tens of minutes to several hours is often a good compromise. The poll messages from the device contain some sort of unique identifier, such as a serial number or MAC address, and the current software version. From this, the update server can see if an update is needed. The poll messages may also contain other status information...

Using Mender for local updates

So much for the theory. In the next two sections of this chapter, I want to demonstrate how the principles I have talked about so far work in practice. For these examples, I will use Mender. Mender uses a symmetric A/B image update mechanism, with a fallback in the event of a failed update. It can operate in standalone mode for local updates, or in managed mode for OTA updates. I will begin with standalone mode.

Mender is written and supported by mender.io (https://mender.io). There is much more information about the software in the documentation section of the website. I will not delve deeply into the configuration of the software here since my aim is to illustrate the principles of software updates. Let's begin with the Mender client.

Building the Mender client

The Mender client is available as a Yocto meta layer. These examples use the Dunfell release of the Yocto Project, which is the same one that we used in Chapter 6, Selecting a Build...

Installing an update

Now we want to make a change to the root filesystem and then install it as an update:

1. Open another shell and put yourself back in the working build directory:

   $ source poky/oe-init-build-env build-mender-qemu

2. Make a copy of the image we just built. This will be the live image that we are going to update:

   $ cd tmp/deploy/images/vexpress-qemu
   $ cp core-image-full-cmdline-vexpress-qemu-grub.uefiimg \
   core-image-live-vexpress-qemu-grub.uefiimg 
   $ cd -

   If we don't do this, the QEMU script will just load the latest image generated by BitBake, including updates, which defeats the object of the demonstration.

3. Next, change the hostname of the target, which will be easy to see when it is installed. To do this, edit conf/local.conf and add this line:

   hostname_pn-base-files = "vexpress-qemu-release2"

4. Now we can build the image in the same way as before:

   $ bitbake core-image-full-cmdline

   This time we are not interested in the .uefiimg file, which...

Using Mender for OTA updates

Once again, we will be using the Mender client on the device, but this time operating it in managed mode, and in addition, we will be configuring a server to deploy the update so that no local interaction is needed. Mender provides an open source server for this. For documentation on how to set up this demo server, see https://docs.mender.io/2.4/getting-started/on-premise-installation.

The installation requires Docker Engine version 19.03 or later to be installed. Refer to the Docker website at https://docs.docker.com/engine/installation. It also requires Docker Compose version 1.25 or later as described here: https://docs.docker.com/compose/install/.

To verify which versions of Docker and Docker Compose you have on your system, use these commands:

$ docker --version
Docker version 19.03.8, build afacb8b7f0
$ docker-compose --version
docker-compose version 1.25.0, build unknown

The Mender server also requires a command-line JSON parser called...

Using balena for local updates

Balena uses Docker containers to deploy software updates. Devices run balenaOS, a Yocto-based Linux distribution that comes with balenaEngine, balena's Docker-compatible container engine. OTA updates occur automatically by way of releases pushed from balenaCloud, a hosted service for managing fleets of devices. Balena can also operate in local mode so that updates originate from a server running on your local host machine rather than the cloud. We will stick to local mode for the following exercises.

Balena is written and supported by balena.io (https://balena.io). There is much more information about the software in the Reference section of the online Docs at balena.io. We won't dig into how balena works since our goal is to deploy and automatically update software on a small fleet of devices for fast development.

Balena provides prebuilt balenaOS images for popular dev boards such as the Raspberry Pi 4 and BeagleBone Black. Downloading...

Summary

Being able to update the software on devices in the field is at the very least a useful attribute, and if the device is connected to the internet, it becomes an absolute must. And yet, all too often it is a feature that is left until the last part of a project, on the assumption that it is not a hard problem to solve. In this chapter, I hope that I have illustrated the problems that are associated with designing an effective and robust update mechanism, and also that there are several open source options readily available. You do not have to reinvent the wheel anymore.

The approach used most often, and also the one with the most real-world testing, is the symmetric image (A/B) update, or its cousin, the asymmetric (recovery) image update. Here, you have the choice of SWUpdate, RAUC, Mender, and fwup. A more recent innovation is the atomic file update, in the form of OSTree. This has good characteristics in reducing the amount of data that needs to be downloaded and the amount...