Chapter 3. The Software Layer

In this chapter, we will cover the following recipes:

• Exploring an image's contents
• Adding a new software layer
• Selecting a specific package version and provider
• Adding supported packages
• Adding new packages
• Adding data, scripts, or configuration files
• Managing users and groups
• Using the sysvinit initialization system
• Using the systemd initialization system
• Installing package installation scripts
• Reducing the Linux kernel image size
• Reducing the root filesystem image size
• Memory-based root filesystems
• Securing the root filesystem
• Releasing software
• Analyzing your system for compliance
• Working with open source and proprietary code

Introduction

With hardware-specific changes on their way, the next step is customizing the target root filesystem, the software that runs under the Linux kernel, also called the Linux user space.

The usual approach to this is to start with one of the available core images and both optimize and customize it as per the needs of your embedded project. Usually, the images chosen as a starting point are either core-image-base for a headless system, or core-image-sato for a graphical system, but any of them will do.

This chapter will show you how to add a software layer to contain those changes, and will explain some of the common customizations made, such as size optimization. It will also show you how to add new packages to your root filesystem, including licensing considerations.

Exploring an image's contents

We have already seen how to use the build history feature to obtain a list of packages and files included in our image. In this recipe, we will explain how the root filesystem is built so that we are able to track its components.

Getting ready

When packages are built, they are placed inside the working directory of your project, usually tmp/work, and classified according to their architecture. For example, on a wandboard build, we find the following directories:

• all-poky-linux: This is used for architecture-independent packages
• cortexa9hf-neon-poky-linux-gnueabi: This is used for cortexa9, hard floating point packages
• cortexa9hf-neon-mx6qdl-poky-linux-gnueabi: This is used for cortexa9, hard floating point packages that are specific to the i.MX6 architecture
• wandboard-poky-linux-gnueabi: This is used for machine-specific packages, in this case wandboard
• x86_64-linux: This is used for the packages that form the host sysroot
• x86_64-nativesdk-pokysdk-linux: This is used...

Adding a new software layer

Root filesystem customization involves adding or modifying content to the base image. Metadata for this content goes into one or more software layers, depending on the amount of customization needed.

A typical embedded project will have just one software layer containing all non-hardware-specific customizations. But it is also possible to have extra layers for graphical frameworks or system-wide elements.

Getting ready

Before starting work on a new layer, it is a good practice to check whether someone else provides a similar layer. Also, if you are trying to integrate an open source project, check whether a layer with it already exists. There is an index of available layers at http://layers.openembedded.org/.

How to do it...

A new software layer can be created in the same way we did for a BSP layer in the previous chapter. We can, for example, create a new meta-custom layer using the yocto-layer command as we learned in the Creating a custom BSP layer recipe in Chapter...

Selecting a specific package version and provider

Our layers can provide recipes for different versions of the same package. For example, the meta-freescale layer contains several different types of Linux kernel recipes:

• linux-imx: This corresponds to the NXP's i.MX BSP kernel source fetched from http://git.freescale.com/git/cgit.cgi/imx/linux-imx.git/
• linux-fslc-imx: This is a fork from NXP's i.MX BSP kernel tree with a stability release and community patches applied, fetched from https://github.com/Freescale/linux-fslc
• linux-fslc: This is a mainline Linux kernel fork with community patches, fetched from https://github.com/Freescale/linux-fslc
• linux-qoriq: This is a kernel with NXP's QorIQ PowerPC-based platform support fetched from https://github.com/qoriq-open-source/linux.git

As we mentioned before, all recipes provide the package name (for example, linux-imx or linux-fslc) by default, but all Linux recipes must also provide the virtual/kernel virtual package. The build system will resolve...

Adding supported packages

It is common to want to add new packages to an image that already has an available recipe in one of the included Yocto layers.

When the target image desired is very different from the supplied core images, it is recommended to define a new image rather than to customize an existing one.

This recipe will show how to customize an existing image by adding supported packages to it, but also to create a completely new image recipe if needed.

Getting ready

To discover whether a package we require is included in our configured layers, and what specific versions are supported, we can use bitbake-layers from our build directory as we saw previously:

$ bitbake-layers show-recipes htophtop: meta-oe              1.0.3

Alternatively, we can also use BitBake as follows:

$ bitbake -s | grep htophtop                                                :1.0.3-r0

Or we can use the find Linux command in our sources directory:

$ find /opt/yocto/fsl-community-bsp/sources -type f -name "htop*.bb"...

Adding new packages

We have seen how to customize our image so that we can add supported packages to it. When we can't find an existing recipe, or we need to integrate some new software we have developed, we will need to create a new Yocto recipe.

Getting ready

There are some questions we need to ask ourselves before starting to write a new recipe:

• Where is the source code stored?
• Is it source-controlled or released as a tarball?
• What is the source code license?
• What build system is it using?
• Does it need configuration?
• Can we cross-compile it as it is, or does it need to be patched?
• What files need to be deployed to the root filesystem, and where do they go?
• Are there any system changes that need to happen, such as new users orinit scripts?
• Are there any dependencies that need to be installed into sysroot beforehand?

Once we know the answers to these questions, we are ready to start writing our recipe.

How to do it...

It is best to start from a blank template such as the one that follows rather than...

Adding data, scripts, or configuration files

All recipes inherit the base class with the default set of tasks to run. After inheriting the base class, a recipe knows how to do things such as fetching and compiling.

As most recipes are meant to install some sort of executable, the base class knows how to build it. But sometimes all we want is to install data, scripts, or configuration files into the filesystem.

If the data or configuration is related to an application, the most logical thing to do is to package it together with the application's recipe itself, and if we think it is better to be installed separately, we could even split it into its own package.

But other times, the data or configuration is unrelated to an application; maybe it applies to the whole system or we just want to provide a separate recipe for it. Or we could even want to install some interpreted language scripts that don't need to be compiled.

How to do it...

In those cases, our recipe should inherit the allarch class...

Managing users and groups

It is also common to need to add or modify users and groups to our filesystem. This recipe explains how it is done.

Getting ready

User information is stored in the /etc/passwd file, a text file that is used as a database for the system user's information. The passwd file is human-readable.

Each line in it corresponds to one user in the system, and it has the following format:

<username>:<password>:<uid>:<gid>:<comment>:<home directory>:<login shell> 

Let's see each of the parameters of this format:

• username: A unique string that identifies the user at login
• uid: User ID, a number that Linux uses to identify the user
• gid: Group ID, a number that Linux uses to identify the user's primary group
• comment: Comma-separated values that describe the account, typically the user's contact details
• home directory: Path to the user's home directory
• login shell: Shell that is started for interactive logins

The default passwd file is stored with the...

Using the sysvinit initialization manager

The initialization manager is an important part of the root filesystem. It is the first thing the kernel executes, and it has the responsibility to start the rest of the system, and manages system shutdown too.

The initialization manager is usually fixed by the Linux distribution. The Poky distribution allows you to select between initialization managers, the most used being System V init (sysvinit) and systemd. The poky-tiny distribution has no initialization manager and uses BusyBox init to directly read the inittab file and launch programs.

The UNIX System V operating system introduced an initialization system that was the basis for Linux's initialization. Several recent attempts have been made to replace it, and systemd has finally managed to get enough traction in between Linux distributions to be considered the successor to sysvinit.

This recipe will introduce the sysvinit initialization manager.

Getting ready

sysvinit is the default initialization...

Using the systemd initialization manager

As an alternative to sysvinit, you can configure your project to use systemd as an initialization manager. systemd not only performs system initialization, it also manages system processes, and packs many more features than sysvinit.

The systemd initialization manager is replacing sysvinit and other initialization managers in most Linux distributions. However, the adoption of systemd in major Linux distributions has not lacked controversy. Many see systemd as an overly complex system with a complicated architecture that goes against the Unix philosophy of simple, short, clear, and extensible modular software, easily maintained and repurposed.

systemd is also big in size compared with sysvinit, although it can be scaled down and customized for embedded projects. Some of its features are tightly integrated with kernel features and updating it may force a kernel update, which is not always feasible for embedded devices.

It does however feature parallel...

Installing package installation scripts

The supported package formats, .rpm, .ipk, and .deb, support the addition of installation scripts that can be run at different times during a package installation process. In this recipe, we will see how to install them.

Getting ready

There are different types of installation script:

• Pre-installation scripts (pkg_preinst): These are called before the package is unpacked
• Post-installation scripts (pkg_postinst): These are called after the package is unpacked, and dependencies will be configured
• Pre-removal scripts (pkg_prerm): These are called with installed or at least partially installed packages
• Post-removal scripts (pkg_postrm): These are called after the package's files have been removed or replaced

How to do it...

An example snippet showing the installation of a pre-installation script in a recipe is as follows:

pkg_preinst_${PN} () { 
    # Shell commands 
} 

All installation scripts work in the same way, with the exception that post-installation scripts...

Reducing the Linux kernel image size

Before or in parallel with the root filesystem customization, embedded projects usually require an image size optimization that will reduce the boot time and memory usage.

Smaller images mean less storage space, less transmission time, and less programming time, which saves money both in manufacturing and field updates.

By default, the compressed Linux kernel image (zImage) for the Wandboard is around 6.2 MB. This recipe will show how we can reduce that.

How to do it...

An example of a minimal kernel configuration for a Wandboard that is able to boot from a microSD card root filesystem is the arch/arm/configs/wandboard-custom-minimal_defconfig file:

CONFIG_KERNEL_XZ=y                                                               
# CONFIG_SWAP is not set                                                         
CONFIG_NO_HZ=y                                                                   
CONFIG_HIGH_RES_TIMERS=y                                         ...

Reducing the root filesystem size

Images with reduced sizes are used alongside production images for tasks such as rescue systems and manufacturing test processes. They are also ideal to be built as initramfs images, that is, images that the Linux kernel mounts from memory, and can even be bundled into a single Linux kernel image binary.

By default, the core-image-minimal size for the Wandboard unpacked root filesystem tarball is around 13 MB, and core-image-sato is around 244 MB. This recipe will explore methods to reduce the size of the root filesystem.

How to do it...

An example of a small image that does not include the core image class and can be used as the base for a bottom-up root filesystem image with reduced size, recipes-core/images/image-small.bb, is shown next:

DESCRIPTION = "Minimal console-only image." 
 
IMAGE_INSTALL= "\
        base-files \  
        base-passwd \ 
        busybox \  
        sysvinit \  
        initscripts \  
" 
 
IMAGE_LINGUAS = " " 
 
LICENSE = "MIT" ...

Memory-based root filesystems

Early user space or initial memory images, known as initramfs images, are typically used as early user spaces with the task of mounting complex filesystems, such as for example when an encrypted partition needs to be set up. For that use case, initramfs will prepare the final root filesystem and then perform a switch_root, effectively replacing the old filesystem with the new and running a new init from the new filesystem. initramfs needs to contain all tools needed to mount the final root filesystem.

Other uses for initramfs include rescue and manufacturing systems.

This recipe will show how to configure the Yocto build system to build initramfs images.

Getting ready

When the Linux kernel mounts a file system from physical storage, it keeps a cache for quick access. A memory filesystem is just this kernel cache without a backing store.

An initramfs is a compressed cpio format file that the kernel extracts into a memory filesystem and uses as the root filesystem...

Securing the root filesystem

In the age of internet-connected devices and the Internet of Things, security in embedded devices is an important consideration in the design of an embedded system.

In this recipe, we will discuss two of the frameworks most frequently used to enhance the security of the root filesystem and show how to integrate them into your Yocto Project-based product.

Getting ready

Linux implements file-level security. Each file has a set of access attributes for user, group and others, and they can be assigned read, write, or execute permissions, as well as some special modes.

Filesystem access is hence based on the system users and groups. When you list files with the ls -l command, you see the file attributes with the following format:

-rwxr-xr-x

The first letter is the file type, followed by triads of user, group, and other permissions. The file can be read, written, and executed by the user that owns it, but only read and executed by a user in the same group or by a different...

Releasing software

When releasing a product based on the Yocto project, we have to consider that we are building on top of a multitude of different open source projects, each with different licensing requirements.

At the minimum, your embedded product will contain a bootloader (probably u-boot), the Linux kernel, and a root filesystem with one or more applications. Both u-boot and the Linux kernel are licensed under the General Public License version 2 (GPLv2), and the root filesystem could contain a variety of programs with different licenses.

All open source licenses allow you to sell a commercial product with a mixture of proprietary and open licenses as long as they are independent and the product complies with all the open source licenses. We will discuss open source and proprietary cohabiting in the Working with open source and proprietary code recipe later on.

When developing open source applications, it's important to give careful thought to the license the software will be released...

Analyzing your system for compliance

The Yocto build system makes it easy to provide auditing information to our legal advisers. This recipe will explain how.

How to do it...

Under tmp/deploy/licenses, we find a directory list of packages (including their corresponding licenses) and an image folder with a package and license manifest.

For the example reduced image provided before, image-small, we have the following:

tmp/deploy/licenses/image-small-initramfs-wandboard-<timestamp>/package.manifest 
base-files 
base-passwd 
busybox 
busybox-syslog 
busybox-udhcpc 
initscripts 
initscripts-functions 
libc6 
run-postinsts 
sysvinit 
sysvinit-inittab 
sysvinit-pidof 
update-alternatives-opkg 
update-rc.d 

And the corresponding tmp/deploy/licenses/image-small-initramfs-wandboard-<timestamp>/license.manifest file excerpt is as follows:

PACKAGE NAME: base-files                                                         
PACKAGE VERSION: 3.0.14                                                  ...

Working with open source and proprietary code

It is common for an embedded product to be built upon an open source system like the one built by Yocto, and to include proprietary software that adds value and specializes the product. This proprietary part usually contains intellectual property and needs to be protected, and it's important to understand how it can coexist with open source code.

This recipe will discuss some examples of open source packages commonly found on embedded products and will briefly explain how to use proprietary software with them.

How to do it...

Open source licenses can be broadly divided into two categories based on whether they are:

• Permissive: These are similar to Internet Software Consortium (ISC), MIT, and BSD licenses. They have few requirements attached to them and they just require us to preserve copyright and limited warranty notices.
• Restrictive: These are similar to the GPL, and bind us to not only distribute the source code and modifications, either with...