Chapter 4. Grasping the BitBake Tool
We now start our journey to understand how the Yocto Project's engine works behind the scenes. In the preceding chapters, we were introduced to the usual Yocto Project workflow to create and emulate images.
In this chapter, we will understand the metadata concept and how recipes depend on each other, and are used by Poky. We will understand how BitBake downloads every needed source code package and how these packages are stored in the directory used to build. This chapter also lists the most common tasks used to generate packages, and determines how packages fit into generated images.
Understanding the BitBake tool
The BitBake task scheduler started as a fork from Portage, which is the package management system used in the Gentoo distribution. However, nowadays the two projects have diverged a lot due to the different usage focusses. The Yocto Project and the OpenEmbedded Project are the most known and intensive users of BitBake, which remains a separated and independent project with its own development cycle and mailing list (<bitbake-devel@lists.openembedded.org>).
As presented in Chapter 1, Meeting the Yocto Project, BitBake is a task scheduler that parses Python and the shell script mixed code. The code parsed generates and runs tasks that may have a complex dependency chain, which is scheduled to allow a parallel execution and maximize the use of computational resources. BitBake can be understood as a tool similar to GNU Make in some aspects.
In this chapter, we cover the main aspects of BitBake tool. However, for more in-depth details about the tool, please...
The metadata used by BitBake can be classified into three major areas:
Configuration (the .conf files)
Classes (the .bbclass files)
Recipes (the .bb and .bbappend files)
The configuration files define the global content, which is used to provide information and configure how the recipes will work. One common example of the configuration file is the machine file that has a list of settings, which describes the board hardware.
The classes are used by the whole system and can be inherited by recipes, according to their needs or by default, as in this case with classes used to define the system behavior and provide the base methods. For example, kernel.bbclass helps the process of build, install, and package of the Linux kernel, independent of version and vendor.
Tip
The recipes and classes are written in a mix of Python and shell scripting code.
The classes and recipes describe the tasks to be run and provide the needed information to allow BitBake to generate the needed task chain...
As previously said, there are three metadata groups—configuration, class, and recipe.
The first parsed metadata in BitBake is configuration metadata, identified by the .conf file extension. This metadata is global, and therefore, affects all recipes and tasks which are executed.
BitBake first searches the current working directory for an optional build/conf/bblayers.conf configuration file, and it is expected to contain a BBLAYERS variable that is a space-delimited list of layer directories. For each directory in this list, a build/conf/layer.conf file is searched for and parsed with the LAYERDIR variable being set to the directory where the layer was found. This process automatically sets up BBPATH and other variables for a given build directory for the user.
Tip
The order of the listed layers in the BBLAYERS variable is followed by BitBake when parsing the metadata. In case your layer needs to be parsed first, be sure to have it listed in the right order in BBLAYERS.
BitBake...
In order to accomplish the dependency, the recipes must declare what they need to have available during the build process. BitBake ensures that the build-time dependencies are satisfied before starting the recipe build. This is easier to understand if we think about an application that uses a library. So, this library must be built and its headers made available for use, before the application itself can be built. The DEPENDS variable is used in a recipe to inform BitBake about the build-time dependency.
When an application depends on something to run, this is called a runtime dependency. This is common for shared data among applications (for example, icons), which is used only when running the application but not used during its build process or when an application calls another during its execution. The runtime dependencies can be expressed using the RDEPENDS variable in a recipe.
With the recipe dependencies chain, BitBake can sort all the recipes in a feasible order for the...
Preferring and providing recipes
The dependency relation between recipes is core to BitBake and the build tool as a whole. It is defined inside each recipe file, with a variable which describes on what a recipe depends (DEPENDS) and what a recipe provides to the system (PROVIDES). These two variables together build the dependency graph used by BitBake during the dependency resolution.
So, if a recipe foo_1.0.bb depends on bar, BitBake lists all recipes providing bar. The bar dependency can be satisfied by the following:
A dependency can be satisfied by several recipes (for example, two or more recipes have PROVIDES += "bar"). In this case, we must inform BitBake the provider to use.
The virtual/kernel provider is a clear example where this mechanism is used. The virtual/ namespace is the convention adopted when we have a set of commonly overridden providers.
All recipes that require...
One of the main features supported by BitBake is source code fetching. This support has been designed to be as modular and as flexible as possible. The mechanism used by BitBake to fetch the source code is internally called as fetcher backend. There are several fetcher backends supported, which can be configured to align the user requirements and optimize source code fetching.
When the Poky source code is downloaded, what is actually copied is the metadata and the BitBake tool. All other source code is fetched on demand. Every Linux-based system includes the Linux kernel and several other utilities that form the root filesystem, such as openssh. The OpenSSH source code is available from its upstream website as a tar.gz file hosted on a FTP server; the Linux kernel release may be fetched from a Git repository.
BitBake offers support for many different fetcher modules that allow retrieval of tarball files and a number of other protocols such as Git, Subversion, Bazaar...
Understanding BitBake's tasks
BitBake uses execution units, which are in essence a set of clustered instructions that run in sequence. These units are known as tasks. There are many tasks being scheduled, executed, and checked by BitBake during every recipe build, provided by classes to form the framework that we use to build a recipe. Some are important to be understood as we often use, extend, implement, or replace them ourselves when writing a recipe.
Run the following command:
BitBake runs a set of scheduled tasks. When we wish to run a specific task, we can use the following command:
To list the tasks defined for a recipe, we can use the following command:
We will briefly describe each of these here:
Generating a root filesystem image
One of the most common uses of Poky is the rootfs image generation (rootfs). The rootfs image should be seen as a ready-to-use root filesystem for a target. The image can be made up of one filesystem or may include other artifacts to be available during its generation as the Linux kernel, device tree, and bootloader binaries and other filesystems. The process to generate the image is composed of several steps, and its most common usages are the following:
Generate the rootfs directory.
Create the required files.
Wrap the final filesystem accordingly to the specific requirements (it may be a disk file with several partitions and contents).
Finally, compress it, if applicable.
All these steps are performed by subtasks of do_rootfs.
The rootfs is basically a directory with the desired packages installed (the package generation is covered in Chapter 6, Assimilating Packaging Support), with the needed tweaks applied just afterwards. The tweaks make minor adjustments...
In this chapter, we learned the metadata concept, how recipes depend on each other, and how Poky deals with dependencies. We also got a better view of the behind-the-scenes tasks done by BitBake to download all the required source code, to store it in the build directory used to build and generate packages, as well as and how these packages fit into generated images.
In the next chapter, we will see the contents of the build directory after complete image generation and how BitBake uses it in the baking process, including the contents of the temporary build directory and its generated files.
