Chapter 8: Yocto Under the Hood

In this chapter, we'll dive deeper into Yocto, embedded Linux's premier build system. We will begin with a tour of Yocto's architecture, taking you through the entire build workflow step by step. Next, we'll look at Yocto's multi-layer approach and why it is a good idea to separate metadata into different layers. As more and more BitBake layers stack up inside your projects, problems will inevitably arise. We will examine a number of ways to debug Yocto build failures, including task logs, devshell, and dependency graphs.

After taking apart the build system, we'll revisit the topic of BitBake from the previous chapter. This time around, we'll cover more of the basic syntax and semantics so that you can write your own recipes from scratch. We'll look at real-world examples of BitBake shell and Python code from actual recipe, include, and configuration files so that you know what to expect when you begin to venture...

Technical requirements

To follow along with the examples in this chapter, make sure you have the following:

• A Linux-based system host system with a minimum of 60 GB available disk space
• Yocto 3.1 (Dunfell) LTS release

You should have already built the 3.1 (Dunfell) LTS release of Yocto in Chapter 6,
Selecting a Build System. If you have not, then please refer to the Compatible Linux Distribution and Build Host Packages sections of the Yocto Project Quick Build guide (https://www.yoctoproject.org/docs/current/brief-yoctoprojectqs/brief-yoctoprojectqs.html) before building Yocto on your Linux host according to the instructions in Chapter 6.

Decomposing Yocto's architecture and workflow

Yocto is a complex beast. Taking it apart is the first step toward understanding it. The architecture of a build system can be organized in terms of its workflow. Yocto gets its workflow from the OpenEmbbedded project, which it is based on. Source materials feed into the system as inputs by way of metadata in the form of BitBake recipes. The build system uses this metadata to fetch, configure, and compile the source code into binary package feeds. These individual output packages are assembled inside a staging area before the finished Linux image and SDK are generated, complete with a manifest that includes a license for each package that's on board:

Figure 8.1 – OpenEmbedded architecture workflow

Here are the seven steps of Yocto's build system workflow, as shown in the
preceding diagram:

1. Define layers for policy, machine, and software metadata.
2. Fetch sources from the source...

Separating metadata into layers

Yocto metadata is organized around the following concepts:

• distro: OS features, including choice of C library, init system, and
  window manager
• machine: CPU architecture, kernel, drivers, and bootloader
• recipe: Application binaries and/or scripts
• image: Development, manufacturing, or production

These concepts map directly to actual byproducts of the build system, thus offering us guidance when designing our projects. We could rush to assemble everything inside a single layer, but that would likely result in a project that is inflexible and unmaintainable. Hardware inevitably gets revised, and one successful consumer device quickly multiplies into a series of products. For these reasons, it is better to adopt a multi-layered approach early on so that we end up with software components that we can easily modify, swap out, and reuse.

At a minimum, you should create individual distribution, BSP, and application layers for every...

Troubleshooting build failures

In the two preceding chapters, we learned how to build bootable images for QEMU, our Nova board, and the Raspberry Pi 4. But what happens when things go wrong? In this section, we will cover a number of useful debugging techniques that should make the prospect of wrangling Yocto build failures less intimidating.

To execute the commands in the subsequent exercises, you need to activate a BitBake environment, as follows:

1. First, navigate one level above the directory where you cloned Yocto.
2. Next, set up your BitBake work environment:

   $ source poky/oe-init-build-env build-rpi

This sets a bunch of environment variables and puts you back in the build-rpi directory that we created in the previous chapter.

Isolating errors

So, your build failed, but where did it fail? You have an error message, but what does it mean and where did it come from? Do not despair. The first step in debugging is reproducing the bug. Once you can reproduce...

Understanding BitBake syntax and semantics

BitBake is a task runner. It is similar to GNU make in that respect, except that it operates on recipes instead of makefiles. The metadata in these recipes defines tasks in shell and Python. BitBake itself is written in Python. The OpenEmbedded project that Yocto is based on consists of BitBake and a large collection of recipes for building embedded Linux distributions. BitBake's power lies in its ability to run tasks in parallel while still satisfying inter-task dependencies. Its layered and inheritance-based approach to metadata enables Yocto to scale in ways Buildroot-based build systems simply cannot.

In Chapter 6, Selecting a Build System, we learned about the five types of BitBake metadata files; that is, .bb, .bbappend, .inc, .bbclass, and .conf. We also wrote BitBake recipes for building a basic helloworld program and nova-image image. Now, we will look more closely at the contents of BitBake metadata files. We know that tasks...

Summary

Even though you can build just about anything with Yocto, it's not always easy to tell what the build system is doing or how. There is hope for us, though. There are command-line tools to help us find where something came from and how to change it. There are task logs we can read from and write to. There is also devshell, which we can use to configure and compile individual things from the command line. And if we divide our projects into multiple layers from the outset, we are likely to get much more mileage out of the work
we do.

BitBake's mix of shell and Python supports some powerful language constructs, such as inheritance, overrides, and conditional variable selection. That's both good and bad. It's good in the sense that layers and recipes are completely composable and customizable. It's bad in the sense that metadata in different recipe files and different layers can interact in strange and unexpected ways. Combine those powerful language...

Further reading

The following resources contain more information about the topics that were introduced in this chapter:

• Yocto Projects Overview and Concepts Manual, by Scott Rifenbark: https://www.yoctoproject.org/docs/latest/overview-manual/overview-manual.html
• What I Wish I'd Known, Yocto Project: https://www.yoctoproject.org/docs/what-i-wish-id-known
• BitBake User Manual, by Richard Purdie, Chris Larson, and Phil Blundell: https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html
• Embedded Linux Projects Using Yocto Project Cookbook, by Alex Gonzalez