For Linux to be run on any device hardware, you will require some hardware-dependent components that are able to abstract the work for hardware-independent ones. The boot loader, kernel, and toolchain contain hardware-dependent components that make the performance of work easier for all the other components. For example, a BusyBox developer will only concentrate on developing the required functionalities for his application, and will not concentrate on hardware compatibility. All these hardware-dependent components offer support for a large variety of hardware architectures for both 32 and 64 bits. For example, the U-Boot implementation is the easiest to take as an example when it comes to source code inspection. From this, we can easily visualize how support for a new device can be added.

We will now try to do some of the little exercises presented previously, but before moving further, I must present the computer configuration on which I will continue to do the exercises, to make sure that that you face as few problems as possible. I am working on an Ubuntu 14.04 and have downloaded the 64-bit image available on the Ubuntu website at http://www.ubuntu.com/download/desktop

Information relevant to the Linux operation running on your computer can be gathered using this command:

uname –srmpio

The preceding command generates this output:

Linux 3.13.0-36-generic x86_64 x86_64 x86_64 GNU/Linux

The next command to gather the information relevant to the Linux operation is as follows:

cat /etc/lsb-release

The preceding command generates this output:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

Now, moving on to exercises, the first one requires you fetch the git repository sources for the U-Boot package:

sudo apt-get install git-core
git clone http://git.denx.de/u-boot.git

After the sources are available on your machine, you can try to take a look inside the board directory; here, a number of development board manufacturers will be present. Let's take a look at board/atmel/sama5d3_xplained, board/faraday/a320evb, board/freescale/imx, and board/freescale/b4860qds. By observing each of these directories, a pattern can be visualized. Almost all of the boards contain a Kconfig file, inspired mainly from kernel sources because they present the configuration dependencies in a clearer manner. A maintainers file offers a list with the contributors to a particular board support. The base Makefile file takes from the higher-level makefiles the necessary object files, which are obtained after a board-specific support is built. The difference is with board/freescale/imx which only offers a list of configuration data that will be later used by the high-level makefiles.

At the kernel level, the hardware-dependent support is added inside the arch file. Here, for each specific architecture besides Makefile and Kconfig, various numbers of subdirectories could also be added. These offer support for different aspects of a kernel, such as the boot, kernel, memory management, or specific applications.

By cloning the kernel sources, the preceding information can be easily visualized by using this code:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Some of the directories that can be visualized are arch/arc and arch/metag.

From the toolchain point of view, the hardware-dependent component is represented by the GNU C Library, which is, in turn, usually represented by glibc. This provides the system call interface that connects to the kernel architecture-dependent code and further provides the communication mechanism between these two entities to user applications. System calls are presented inside the sysdeps directory of the glibc sources if the glibc sources are cloned, as follows:

git clone http://sourceware.org/git/glibc.git

The preceding information can be verified using two methods: the first one involves opening the sysdeps/arm directory, for example, or by reading the ChangeLog.old-ports-arm library. Although it's old and has nonexistent links, such as ports directory, which disappeared from the newer versions of the repository, the latter can still be used as a reference point.

These packages are also very easily accessible using the Yocto Project's poky repository. As mentioned at https://www.yoctoproject.org/about:

Most of the interaction anyone has with the Yocto Project is done through the Poky build system, which is one of its core components that offers the features and functionalities needed to generate fully customizable Linux software stacks. The first step needed to ensure interaction with the repository sources would be to clone them:

git clone -b dizzy http://git.yoctoproject.org/git/poky

After the sources are present on your computer, a set of recipes and configuration files need to be inspected. The first location that can be inspected is the U-Boot recipe, available at meta/recipes-bsp/u-boot/u-boot_2013.07.bb. It contains the instructions necessary to build the U-Boot package for the corresponding selected machine. The next place to inspect is in the recipes available in the kernel. Here, the work is sparse and more package versions are available. It also provides some bbappends for available recipes, such as meta/recipes-kernel/linux/linux-yocto_3.14.bb and meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend. This constitutes a good example for one of the kernel package versions available when starting a new build using BitBake.

Toolchain construction is a big and important step for host generated packages. To do this, a set of packages are necessary, such as gcc, binutils, glibc library, and kernel headers, which play an important role. The recipes corresponding to this package are available inside the meta/recipes-devtools/gcc/, meta/recipes-devtools/binutils, and meta/recipes-core/glibc paths. In all the available locations, a multitude of recipes can be found, each one with a specific purpose. This information will be detailed in the next chapter.

The configurations and options for the selection of one package version in favor of another is mainly added inside the machine configuration. One such example is the Freescale MPC8315E-rdb low-power model supported by Yocto 1.6, and its machine configuration is available inside the meta-yocto-bsp/conf/machine/mpc8315e-rdb.conf file.

of work easier for all the other components. For example, a BusyBox developer will only concentrate on developing the required functionalities for his application, and will not concentrate on hardware compatibility. All these hardware-dependent components offer support for a large variety of hardware architectures for both 32 and 64 bits. For example, the U-Boot implementation is the easiest to take as an example when it comes to source code inspection. From this, we can easily visualize how support for a new device can be added.

We will now try to do some of the little exercises presented previously, but before moving further, I must present the computer configuration on which I will continue to do the exercises, to make sure that that you face as few problems as possible. I am working on an Ubuntu 14.04 and have downloaded the 64-bit image available on the Ubuntu website at http://www.ubuntu.com/download/desktop

Information relevant to the Linux operation running on your computer can be gathered using this command:

uname –srmpio

The preceding command generates this output:

Linux 3.13.0-36-generic x86_64 x86_64 x86_64 GNU/Linux

The next command to gather the information relevant to the Linux operation is as follows:

cat /etc/lsb-release

The preceding command generates this output:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

Now, moving on to exercises, the first one requires you fetch the git repository sources for the U-Boot package:

sudo apt-get install git-core
git clone http://git.denx.de/u-boot.git

After the sources are available on your machine, you can try to take a look inside the board directory; here, a number of development board manufacturers will be present. Let's take a look at board/atmel/sama5d3_xplained, board/faraday/a320evb, board/freescale/imx, and board/freescale/b4860qds. By observing each of these directories, a pattern can be visualized. Almost all of the boards contain a Kconfig file, inspired mainly from kernel sources because they present the configuration dependencies in a clearer manner. A maintainers file offers a list with the contributors to a particular board support. The base Makefile file takes from the higher-level makefiles the necessary object files, which are obtained after a board-specific support is built. The difference is with board/freescale/imx which only offers a list of configuration data that will be later used by the high-level makefiles.

At the kernel level, the hardware-dependent support is added inside the arch file. Here, for each specific architecture besides Makefile and Kconfig, various numbers of subdirectories could also be added. These offer support for different aspects of a kernel, such as the boot, kernel, memory management, or specific applications.

By cloning the kernel sources, the preceding information can be easily visualized by using this code:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Some of the directories that can be visualized are arch/arc and arch/metag.

From the toolchain point of view, the hardware-dependent component is represented by the GNU C Library, which is, in turn, usually represented by glibc. This provides the system call interface that connects to the kernel architecture-dependent code and further provides the communication mechanism between these two entities to user applications. System calls are presented inside the sysdeps directory of the glibc sources if the glibc sources are cloned, as follows:

git clone http://sourceware.org/git/glibc.git

The preceding information can be verified using two methods: the first one involves opening the sysdeps/arm directory, for example, or by reading the ChangeLog.old-ports-arm library. Although it's old and has nonexistent links, such as ports directory, which disappeared from the newer versions of the repository, the latter can still be used as a reference point.

These packages are also very easily accessible using the Yocto Project's poky repository. As mentioned at https://www.yoctoproject.org/about:

Most of the interaction anyone has with the Yocto Project is done through the Poky build system, which is one of its core components that offers the features and functionalities needed to generate fully customizable Linux software stacks. The first step needed to ensure interaction with the repository sources would be to clone them:

git clone -b dizzy http://git.yoctoproject.org/git/poky

After the sources are present on your computer, a set of recipes and configuration files need to be inspected. The first location that can be inspected is the U-Boot recipe, available at meta/recipes-bsp/u-boot/u-boot_2013.07.bb. It contains the instructions necessary to build the U-Boot package for the corresponding selected machine. The next place to inspect is in the recipes available in the kernel. Here, the work is sparse and more package versions are available. It also provides some bbappends for available recipes, such as meta/recipes-kernel/linux/linux-yocto_3.14.bb and meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend. This constitutes a good example for one of the kernel package versions available when starting a new build using BitBake.

Toolchain construction is a big and important step for host generated packages. To do this, a set of packages are necessary, such as gcc, binutils, glibc library, and kernel headers, which play an important role. The recipes corresponding to this package are available inside the meta/recipes-devtools/gcc/, meta/recipes-devtools/binutils, and meta/recipes-core/glibc paths. In all the available locations, a multitude of recipes can be found, each one with a specific purpose. This information will be detailed in the next chapter.

The configurations and options for the selection of one package version in favor of another is mainly added inside the machine configuration. One such example is the Freescale MPC8315E-rdb low-power model supported by Yocto 1.6, and its machine configuration is available inside the meta-yocto-bsp/conf/machine/mpc8315e-rdb.conf file.

on to exercises, the first one requires you fetch the git repository sources for the U-Boot package:

sudo apt-get install git-core
git clone http://git.denx.de/u-boot.git

After the sources are available on your machine, you can try to take a look inside the board directory; here, a number of development board manufacturers will be present. Let's take a look at board/atmel/sama5d3_xplained, board/faraday/a320evb, board/freescale/imx, and board/freescale/b4860qds. By observing each of these directories, a pattern can be visualized. Almost all of the boards contain a Kconfig file, inspired mainly from kernel sources because they present the configuration dependencies in a clearer manner. A maintainers file offers a list with the contributors to a particular board support. The base Makefile file takes from the higher-level makefiles the necessary object files, which are obtained after a board-specific support is built. The difference is with board/freescale/imx which only offers a list of configuration data that will be later used by the high-level makefiles.

At the kernel level, the hardware-dependent support is added inside the arch file. Here, for each specific architecture besides Makefile and Kconfig, various numbers of subdirectories could also be added. These offer support for different aspects of a kernel, such as the boot, kernel, memory management, or specific applications.

By cloning the kernel sources, the preceding information can be easily visualized by using this code:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Some of the directories that can be visualized are arch/arc and arch/metag.

From the toolchain point of view, the hardware-dependent component is represented by the GNU C Library, which is, in turn, usually represented by glibc. This provides the system call interface that connects to the kernel architecture-dependent code and further provides the communication mechanism between these two entities to user applications. System calls are presented inside the sysdeps directory of the glibc sources if the glibc sources are cloned, as follows:

git clone http://sourceware.org/git/glibc.git

The preceding information can be verified using two methods: the first one involves opening the sysdeps/arm directory, for example, or by reading the ChangeLog.old-ports-arm library. Although it's old and has nonexistent links, such as ports directory, which disappeared from the newer versions of the repository, the latter can still be used as a reference point.

These packages are also very easily accessible using the Yocto Project's poky repository. As mentioned at https://www.yoctoproject.org/about:

Most of the interaction anyone has with the Yocto Project is done through the Poky build system, which is one of its core components that offers the features and functionalities needed to generate fully customizable Linux software stacks. The first step needed to ensure interaction with the repository sources would be to clone them:

git clone -b dizzy http://git.yoctoproject.org/git/poky

After the sources are present on your computer, a set of recipes and configuration files need to be inspected. The first location that can be inspected is the U-Boot recipe, available at meta/recipes-bsp/u-boot/u-boot_2013.07.bb. It contains the instructions necessary to build the U-Boot package for the corresponding selected machine. The next place to inspect is in the recipes available in the kernel. Here, the work is sparse and more package versions are available. It also provides some bbappends for available recipes, such as meta/recipes-kernel/linux/linux-yocto_3.14.bb and meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend. This constitutes a good example for one of the kernel package versions available when starting a new build using BitBake.

Toolchain construction is a big and important step for host generated packages. To do this, a set of packages are necessary, such as gcc, binutils, glibc library, and kernel headers, which play an important role. The recipes corresponding to this package are available inside the meta/recipes-devtools/gcc/, meta/recipes-devtools/binutils, and meta/recipes-core/glibc paths. In all the available locations, a multitude of recipes can be found, each one with a specific purpose. This information will be detailed in the next chapter.

The configurations and options for the selection of one package version in favor of another is mainly added inside the machine configuration. One such example is the Freescale MPC8315E-rdb low-power model supported by Yocto 1.6, and its machine configuration is available inside the meta-yocto-bsp/conf/machine/mpc8315e-rdb.conf file.

Buildroot as a tool tries to simplify the ways in which a Linux system is generated using a cross-compiler. Buildroot is able to generate a bootloader, kernel image, root filesystem, and even a cross-compiler. It can generate each one of these components, although in an independent way, and because of this, its main usage has been restricted to a cross-compiled toolchain that generates a corresponding and custom root filesystem. It is mainly used in embedded devices and very rarely for x86 architectures; its main focus being architectures, such as ARM, PowerPC, or MIPS. As with every tool presented in this book, it is designed to run on Linux, and certain packages are expected to be present on the host system for their proper usage. There are a couple of mandatory packages and some optional ones as well.

There is a list of mandatory packages that contain the certain packages, and are described inside the Buildroot manual available at http://buildroot.org/downloads/manual/manual.html. These packages are as follows:

Beside these mandatory packages, there are also a number of optional packages. They are very useful for the following:

Buildroot releases are made available to the open source community at http://buildroot.org/downloads/ every three months, specifically in February, May, August, and November, and the release name has the buildroot-yyyy-mm format. For people interested in giving Buildroot a try, the manual described in the previous section should be the starting point for installing and configuration. Developers interested in taking a look at the Buildroot source code can refer to http://git.buildroot.net/buildroot/.

Next, there will be presented a new set of tools that brought their contribution to this field and place on a lower support level the Buildroot project. I believe that a quick review of the strengths and weaknesses of these tools would be required. We will start with Scratchbox and, taking into consideration that it is already deprecated, there is not much to say about it; it's being mentioned purely for historical reasons. Next on the line is LTIB, which constituted the standard for Freescale hardware until the adoption of Yocto. It is well supported by Freescale in terms of Board Support Packages (BSPs) and contains a large database of components. On the other hand, it is quite old and it was switched with Yocto. It does not contain the support of new distributions, it is not used by many hardware providers, and, in a short period of time, it could very well become as deprecated as Scratchbox. Buildroot is the last of them and is easy to use, having a Makefile base format and an active community behind it. However, it is limited to smaller and simpler images or devices, and it is not aware of partial builds or packages.

The next tools to be introduced are very closely related and, in fact, have the same inspiration and common ancestor, the OpenEmbedded project. All three projects are linked by the common engine called Bitbake and are inspired by the Gentoo Portage build tool. OpenEmbedded was first developed in 2001 when the Sharp Corporation launched the ARM-based PDA, and SL-5000 Zaurus, which run Lineo, an embedded Linux distribution. After the introduction of Sharp Zaurus, it did not take long for Chris Larson to initiate the OpenZaurus Project, which was meant to be a replacement for SharpROM, based on Buildroot. After this, people started to contribute many more software packages, and even the support of new devices, and, eventually, the system started to show its limitations. In 2003, discussions were initiated around a new build system that could offer a generic build environment and incorporate the usage models requested by the open source community; this was the system to be used for embedded Linux distributions. These discussions started showing results in 2003, and what has emerged today is the Openembedded project. It had packages ported from OpenZaurus by people, such as Chris Larson, Michael Lauer, and Holger Schurig, according to the capabilities of the new build system.

The Yocto Project is the next evolutionary stage of the same project and has the Poky build system as its core piece, which was created by Richard Purdie. The project started as a stabilized branch of the OpenEmbedded project and only included a subset of the numerous recipes available on OpenEmbedded; it also had a limited set of devices and support of architectures. Over time, it became much more than this: it changed into a software development platform that incorporated a fakeroot replacement, an Eclipse plug-in, and QEMU-based images. Both the Yocto Project and OpenEmbedded now coordinate around a core set of metadata called OpenEmbedded-Core (OE-Core).

The Yocto Project is sponsored by the Linux Foundation, and offers a starting point for developers of Linux embedded systems who are interested in developing a customized distribution for embedded products in a hardware-agnostic environment. The Poky build system represents one of its core components and is also quite complex. At the center of all this lies Bitbake, the engine that powers everything, the tool that processes metadata, downloads corresponding source codes, resolves dependencies, and stores all the necessary libraries and executables inside the build directory accordingly. Poky combines the best from OpenEmbedded with the idea of layering additional software components that could be added or removed from a build environment configuration, depending on the needs of the developer.

Poky is build system that is developed with the idea of keeping simplicity in mind. By default, the configuration for a test build requires very little interaction from the user. Based on the clone made in one of the previous exercises, we can do a new exercise to emphasize this idea:

cd poky
source oe-init-build-env ../build-test
bitbake core-image-minimal

As shown in this example, it is easy to obtain a Linux image that can be later used for testing inside a QEMU environment. There are a number of images footprints available that will vary from a shell-accessible minimal image to an LSB compliant image with GNOME Mobile user interface support. Of course, that these base images can be imported in new ones for added functionalities. The layered structure that Poky has is a great advantage because it adds the possibility to extend functionalities and to contain the impact of errors. Layers could be used for all sort of functionalities, from adding support for a new hardware platform to extending the support for tools, and from a new software stack to extended image features. The sky is the limit here because almost any recipe can be combined with another.

All this is possible because of the Bitbake engine, which, after the environment setup and the tests for minimal systems requirements are met, based on the configuration files and input received, identifies the interdependencies between tasks, the execution order of tasks, generates a fully functional cross-compilation environment, and starts building the necessary native and target-specific packages tasks exactly as they were defined by the developer. Here is an example with a list of the available tasks for a package:

The metadata modularization is based on two ideas—the first one refers to the possibility of prioritizing the structure of layers, and the second refers to the possibility of not having the need for duplicate work when a recipe needs changes. The layers are overlapping. The most general layer is meta, and all the other layers are usually stacked over it, such as meta-yocto with Yocto-specific recipes, machine specific board support packages, and other optional layers, depending on the requirements and needs of developers. The customization of recipes should be done using bbappend situated in an upper layer. This method is preferred to ensure that the duplication of recipes does not happen, and it also helps to support newer and older versions of them.

An example of the organization of layers is found in the previous example that specified the list of the available tasks for a package. If a user is interested in identifying the layers used by the test build setup in the previous exercise that specified the list of the available tasks for a package, the bblayers.conf file is a good source of inspiration. If cat is done on this file, the following output will be visible:

# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "6"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /home/alex/workspace/book/poky/meta \
  /home/alex/workspace/book/poky/meta-yocto \
  /home/alex/workspace/book/poky/meta-yocto-bsp \
  "
BBLAYERS_NON_REMOVABLE ?= " \
  /home/alex/workspace/book/poky/meta \
  /home/alex/workspace/book/poky/meta-yocto \
  "

The complete command for doing this is:

cat build-test/conf/bblayers.conf

Here is a visual mode for the layered structure of a more generic build directory:

Yocto as a project offers another important feature: the possibility of having an image regenerated in the same way, no matter what factors change on your host machine. This is a very important feature, taking into consideration not only that, in the development process, changes to a number of tools, such as autotools, cross-compiler, Makefile, perl, bison, pkgconfig, and so on, could occur, but also the fact that parameters could change in the interaction process with regards to a repository. Simply cloning one of the repository branches and applying corresponding patches may not solve all the problems. The solution that the Yocto Project has to these problems is quite simple. By defining parameters prior to any of the steps of the installation as variables and configuration parameters inside recipes, and by making sure that the configuration process is also automated, will minimize the risks of manual interaction are minimized. This process makes sure that image generation is always done as it was the first time.

Since the development tools on the host machine are prone to change, Yocto usually compiles the necessary tools for the development process of packages and images, and only after their build process is finished, the Bitbake build engine starts building the requested packages. This isolation from the developer's machine helps the development process by guaranteeing the fact that updates from the host machine do not influence or affect the processes of generating the embedded Linux distribution.

Another critical point that was elegantly solved by the Yocto Project is represented by the way that the toolchain handles the inclusion of headers and libraries; because this could bring later on not only compilation but also execution errors that are very hard to predict. Yocto resolves these problems by moving all the headers and libraries inside the corresponding sysroots directory, and by using the sysroot option, the build process makes sure that no contamination is done with the native components. An example will emphasize this information better:

ls -l build-test/tmp/sysroots/
total 12K
drwxr-xr-x 8 alex alex 4,0K sep 28 04:17 qemux86/
drwxr-xr-x 5 alex alex 4,0K sep 28 00:48 qemux86-tcbootstrap/
drwxr-xr-x 9 alex alex 4,0K sep 28 04:21 x86_64-linux/

ls -l build-test/tmp/sysroots/qemux86/ 
total 24K
drwxr-xr-x 2 alex alex 4,0K sep 28 01:52 etc/
drwxr-xr-x 5 alex alex 4,0K sep 28 04:15 lib/
drwxr-xr-x 6 alex alex 4,0K sep 28 03:51 pkgdata/
drwxr-xr-x 2 alex alex 4,0K sep 28 04:17 sysroot-providers/
drwxr-xr-x 7 alex alex 4,0K sep 28 04:16 usr/
drwxr-xr-x 3 alex alex 4,0K sep 28 01:52 var/

The Yocto project contributes to making reliable embedded Linux development and because of its dimensions, it is used for lots of things, ranging from board support packages by hardware companies to new software solutions by software development companies. Yocto is not a perfect tool and it has certain drawbacks:

There are also other things that bother developers, such as ptest integration and SDK sysroot's lack of extensibility, but a part of them are solved by the big community behind the project, and until the project shows its limitations, a new one will still need to wait to take its place. Until this happens, Yocto is the framework to use to develop custom embedded Linux distribution or products based in Linux.

a tool tries to simplify the ways in which a Linux system is generated using a cross-compiler. Buildroot is able to generate a bootloader, kernel image, root filesystem, and even a cross-compiler. It can generate each one of these components, although in an independent way, and because of this, its main usage has been restricted to a cross-compiled toolchain that generates a corresponding and custom root filesystem. It is mainly used in embedded devices and very rarely for x86 architectures; its main focus being architectures, such as ARM, PowerPC, or MIPS. As with every tool presented in this book, it is designed to run on Linux, and certain packages are expected to be present on the host system for their proper usage. There are a couple of mandatory packages and some optional ones as well.

There is a list of mandatory packages that contain the certain packages, and are described inside the Buildroot manual available at http://buildroot.org/downloads/manual/manual.html. These packages are as follows:

Beside these mandatory packages, there are also a number of optional packages. They are very useful for the following:

Buildroot releases are made available to the open source community at http://buildroot.org/downloads/ every three months, specifically in February, May, August, and November, and the release name has the buildroot-yyyy-mm format. For people interested in giving Buildroot a try, the manual described in the previous section should be the starting point for installing and configuration. Developers interested in taking a look at the Buildroot source code can refer to http://git.buildroot.net/buildroot/.

Next, there will be presented a new set of tools that brought their contribution to this field and place on a lower support level the Buildroot project. I believe that a quick review of the strengths and weaknesses of these tools would be required. We will start with Scratchbox and, taking into consideration that it is already deprecated, there is not much to say about it; it's being mentioned purely for historical reasons. Next on the line is LTIB, which constituted the standard for Freescale hardware until the adoption of Yocto. It is well supported by Freescale in terms of Board Support Packages (BSPs) and contains a large database of components. On the other hand, it is quite old and it was switched with Yocto. It does not contain the support of new distributions, it is not used by many hardware providers, and, in a short period of time, it could very well become as deprecated as Scratchbox. Buildroot is the last of them and is easy to use, having a Makefile base format and an active community behind it. However, it is limited to smaller and simpler images or devices, and it is not aware of partial builds or packages.

The next tools to be introduced are very closely related and, in fact, have the same inspiration and common ancestor, the OpenEmbedded project. All three projects are linked by the common engine called Bitbake and are inspired by the Gentoo Portage build tool. OpenEmbedded was first developed in 2001 when the Sharp Corporation launched the ARM-based PDA, and SL-5000 Zaurus, which run Lineo, an embedded Linux distribution. After the introduction of Sharp Zaurus, it did not take long for Chris Larson to initiate the OpenZaurus Project, which was meant to be a replacement for SharpROM, based on Buildroot. After this, people started to contribute many more software packages, and even the support of new devices, and, eventually, the system started to show its limitations. In 2003, discussions were initiated around a new build system that could offer a generic build environment and incorporate the usage models requested by the open source community; this was the system to be used for embedded Linux distributions. These discussions started showing results in 2003, and what has emerged today is the Openembedded project. It had packages ported from OpenZaurus by people, such as Chris Larson, Michael Lauer, and Holger Schurig, according to the capabilities of the new build system.

The Yocto Project is the next evolutionary stage of the same project and has the Poky build system as its core piece, which was created by Richard Purdie. The project started as a stabilized branch of the OpenEmbedded project and only included a subset of the numerous recipes available on OpenEmbedded; it also had a limited set of devices and support of architectures. Over time, it became much more than this: it changed into a software development platform that incorporated a fakeroot replacement, an Eclipse plug-in, and QEMU-based images. Both the Yocto Project and OpenEmbedded now coordinate around a core set of metadata called OpenEmbedded-Core (OE-Core).

The Yocto Project is sponsored by the Linux Foundation, and offers a starting point for developers of Linux embedded systems who are interested in developing a customized distribution for embedded products in a hardware-agnostic environment. The Poky build system represents one of its core components and is also quite complex. At the center of all this lies Bitbake, the engine that powers everything, the tool that processes metadata, downloads corresponding source codes, resolves dependencies, and stores all the necessary libraries and executables inside the build directory accordingly. Poky combines the best from OpenEmbedded with the idea of layering additional software components that could be added or removed from a build environment configuration, depending on the needs of the developer.

Poky is build system that is developed with the idea of keeping simplicity in mind. By default, the configuration for a test build requires very little interaction from the user. Based on the clone made in one of the previous exercises, we can do a new exercise to emphasize this idea:

cd poky
source oe-init-build-env ../build-test
bitbake core-image-minimal

As shown in this example, it is easy to obtain a Linux image that can be later used for testing inside a QEMU environment. There are a number of images footprints available that will vary from a shell-accessible minimal image to an LSB compliant image with GNOME Mobile user interface support. Of course, that these base images can be imported in new ones for added functionalities. The layered structure that Poky has is a great advantage because it adds the possibility to extend functionalities and to contain the impact of errors. Layers could be used for all sort of functionalities, from adding support for a new hardware platform to extending the support for tools, and from a new software stack to extended image features. The sky is the limit here because almost any recipe can be combined with another.

All this is possible because of the Bitbake engine, which, after the environment setup and the tests for minimal systems requirements are met, based on the configuration files and input received, identifies the interdependencies between tasks, the execution order of tasks, generates a fully functional cross-compilation environment, and starts building the necessary native and target-specific packages tasks exactly as they were defined by the developer. Here is an example with a list of the available tasks for a package:

The metadata modularization is based on two ideas—the first one refers to the possibility of prioritizing the structure of layers, and the second refers to the possibility of not having the need for duplicate work when a recipe needs changes. The layers are overlapping. The most general layer is meta, and all the other layers are usually stacked over it, such as meta-yocto with Yocto-specific recipes, machine specific board support packages, and other optional layers, depending on the requirements and needs of developers. The customization of recipes should be done using bbappend situated in an upper layer. This method is preferred to ensure that the duplication of recipes does not happen, and it also helps to support newer and older versions of them.

An example of the organization of layers is found in the previous example that specified the list of the available tasks for a package. If a user is interested in identifying the layers used by the test build setup in the previous exercise that specified the list of the available tasks for a package, the bblayers.conf file is a good source of inspiration. If cat is done on this file, the following output will be visible:

# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "6"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /home/alex/workspace/book/poky/meta \
  /home/alex/workspace/book/poky/meta-yocto \
  /home/alex/workspace/book/poky/meta-yocto-bsp \
  "
BBLAYERS_NON_REMOVABLE ?= " \
  /home/alex/workspace/book/poky/meta \
  /home/alex/workspace/book/poky/meta-yocto \
  "

The complete command for doing this is:

cat build-test/conf/bblayers.conf

Here is a visual mode for the layered structure of a more generic build directory:

Yocto as a project offers another important feature: the possibility of having an image regenerated in the same way, no matter what factors change on your host machine. This is a very important feature, taking into consideration not only that, in the development process, changes to a number of tools, such as autotools, cross-compiler, Makefile, perl, bison, pkgconfig, and so on, could occur, but also the fact that parameters could change in the interaction process with regards to a repository. Simply cloning one of the repository branches and applying corresponding patches may not solve all the problems. The solution that the Yocto Project has to these problems is quite simple. By defining parameters prior to any of the steps of the installation as variables and configuration parameters inside recipes, and by making sure that the configuration process is also automated, will minimize the risks of manual interaction are minimized. This process makes sure that image generation is always done as it was the first time.

Since the development tools on the host machine are prone to change, Yocto usually compiles the necessary tools for the development process of packages and images, and only after their build process is finished, the Bitbake build engine starts building the requested packages. This isolation from the developer's machine helps the development process by guaranteeing the fact that updates from the host machine do not influence or affect the processes of generating the embedded Linux distribution.

Another critical point that was elegantly solved by the Yocto Project is represented by the way that the toolchain handles the inclusion of headers and libraries; because this could bring later on not only compilation but also execution errors that are very hard to predict. Yocto resolves these problems by moving all the headers and libraries inside the corresponding sysroots directory, and by using the sysroot option, the build process makes sure that no contamination is done with the native components. An example will emphasize this information better:

ls -l build-test/tmp/sysroots/
total 12K
drwxr-xr-x 8 alex alex 4,0K sep 28 04:17 qemux86/
drwxr-xr-x 5 alex alex 4,0K sep 28 00:48 qemux86-tcbootstrap/
drwxr-xr-x 9 alex alex 4,0K sep 28 04:21 x86_64-linux/

ls -l build-test/tmp/sysroots/qemux86/ 
total 24K
drwxr-xr-x 2 alex alex 4,0K sep 28 01:52 etc/
drwxr-xr-x 5 alex alex 4,0K sep 28 04:15 lib/
drwxr-xr-x 6 alex alex 4,0K sep 28 03:51 pkgdata/
drwxr-xr-x 2 alex alex 4,0K sep 28 04:17 sysroot-providers/
drwxr-xr-x 7 alex alex 4,0K sep 28 04:16 usr/
drwxr-xr-x 3 alex alex 4,0K sep 28 01:52 var/

The Yocto project contributes to making reliable embedded Linux development and because of its dimensions, it is used for lots of things, ranging from board support packages by hardware companies to new software solutions by software development companies. Yocto is not a perfect tool and it has certain drawbacks:

There are also other things that bother developers, such as ptest integration and SDK sysroot's lack of extensibility, but a part of them are solved by the big community behind the project, and until the project shows its limitations, a new one will still need to wait to take its place. Until this happens, Yocto is the framework to use to develop custom embedded Linux distribution or products based in Linux.

to be introduced are very closely related and, in fact, have the same inspiration and common ancestor, the OpenEmbedded project. All three projects are linked by the common engine called Bitbake and are inspired by the Gentoo Portage build tool. OpenEmbedded was first developed in 2001 when the Sharp Corporation launched the ARM-based PDA, and SL-5000 Zaurus, which run Lineo, an embedded Linux distribution. After the introduction of Sharp Zaurus, it did not take long for Chris Larson to initiate the OpenZaurus Project, which was meant to be a replacement for SharpROM, based on Buildroot. After this, people started to contribute many more software packages, and even the support of new devices, and, eventually, the system started to show its limitations. In 2003, discussions were initiated around a new build system that could offer a generic build environment and incorporate the usage models requested by the open source community; this was the system to be used for embedded Linux distributions. These discussions started showing results in 2003, and what has emerged today is the Openembedded project. It had packages ported from OpenZaurus by people, such as Chris Larson, Michael Lauer, and Holger Schurig, according to the capabilities of the new build system.

The Yocto Project is the next evolutionary stage of the same project and has the Poky build system as its core piece, which was created by Richard Purdie. The project started as a stabilized branch of the OpenEmbedded project and only included a subset of the numerous recipes available on OpenEmbedded; it also had a limited set of devices and support of architectures. Over time, it became much more than this: it changed into a software development platform that incorporated a fakeroot replacement, an Eclipse plug-in, and QEMU-based images. Both the Yocto Project and OpenEmbedded now coordinate around a core set of metadata called OpenEmbedded-Core (OE-Core).

The Yocto Project is sponsored by the Linux Foundation, and offers a starting point for developers of Linux embedded systems who are interested in developing a customized distribution for embedded products in a hardware-agnostic environment. The Poky build system represents one of its core components and is also quite complex. At the center of all this lies Bitbake, the engine that powers everything, the tool that processes metadata, downloads corresponding source codes, resolves dependencies, and stores all the necessary libraries and executables inside the build directory accordingly. Poky combines the best from OpenEmbedded with the idea of layering additional software components that could be added or removed from a build environment configuration, depending on the needs of the developer.

Poky is build system that is developed with the idea of keeping simplicity in mind. By default, the configuration for a test build requires very little interaction from the user. Based on the clone made in one of the previous exercises, we can do a new exercise to emphasize this idea:

cd poky
source oe-init-build-env ../build-test
bitbake core-image-minimal

As shown in this example, it is easy to obtain a Linux image that can be later used for testing inside a QEMU environment. There are a number of images footprints available that will vary from a shell-accessible minimal image to an LSB compliant image with GNOME Mobile user interface support. Of course, that these base images can be imported in new ones for added functionalities. The layered structure that Poky has is a great advantage because it adds the possibility to extend functionalities and to contain the impact of errors. Layers could be used for all sort of functionalities, from adding support for a new hardware platform to extending the support for tools, and from a new software stack to extended image features. The sky is the limit here because almost any recipe can be combined with another.

All this is possible because of the Bitbake engine, which, after the environment setup and the tests for minimal systems requirements are met, based on the configuration files and input received, identifies the interdependencies between tasks, the execution order of tasks, generates a fully functional cross-compilation environment, and starts building the necessary native and target-specific packages tasks exactly as they were defined by the developer. Here is an example with a list of the available tasks for a package:

The metadata modularization is based on two ideas—the first one refers to the possibility of prioritizing the structure of layers, and the second refers to the possibility of not having the need for duplicate work when a recipe needs changes. The layers are overlapping. The most general layer is meta, and all the other layers are usually stacked over it, such as meta-yocto with Yocto-specific recipes, machine specific board support packages, and other optional layers, depending on the requirements and needs of developers. The customization of recipes should be done using bbappend situated in an upper layer. This method is preferred to ensure that the duplication of recipes does not happen, and it also helps to support newer and older versions of them.

An example of the organization of layers is found in the previous example that specified the list of the available tasks for a package. If a user is interested in identifying the layers used by the test build setup in the previous exercise that specified the list of the available tasks for a package, the bblayers.conf file is a good source of inspiration. If cat is done on this file, the following output will be visible:

# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "6"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /home/alex/workspace/book/poky/meta \
  /home/alex/workspace/book/poky/meta-yocto \
  /home/alex/workspace/book/poky/meta-yocto-bsp \
  "
BBLAYERS_NON_REMOVABLE ?= " \
  /home/alex/workspace/book/poky/meta \
  /home/alex/workspace/book/poky/meta-yocto \
  "

The complete command for doing this is:

cat build-test/conf/bblayers.conf

Here is a visual mode for the layered structure of a more generic build directory:

Yocto as a project offers another important feature: the possibility of having an image regenerated in the same way, no matter what factors change on your host machine. This is a very important feature, taking into consideration not only that, in the development process, changes to a number of tools, such as autotools, cross-compiler, Makefile, perl, bison, pkgconfig, and so on, could occur, but also the fact that parameters could change in the interaction process with regards to a repository. Simply cloning one of the repository branches and applying corresponding patches may not solve all the problems. The solution that the Yocto Project has to these problems is quite simple. By defining parameters prior to any of the steps of the installation as variables and configuration parameters inside recipes, and by making sure that the configuration process is also automated, will minimize the risks of manual interaction are minimized. This process makes sure that image generation is always done as it was the first time.

Since the development tools on the host machine are prone to change, Yocto usually compiles the necessary tools for the development process of packages and images, and only after their build process is finished, the Bitbake build engine starts building the requested packages. This isolation from the developer's machine helps the development process by guaranteeing the fact that updates from the host machine do not influence or affect the processes of generating the embedded Linux distribution.

Another critical point that was elegantly solved by the Yocto Project is represented by the way that the toolchain handles the inclusion of headers and libraries; because this could bring later on not only compilation but also execution errors that are very hard to predict. Yocto resolves these problems by moving all the headers and libraries inside the corresponding sysroots directory, and by using the sysroot option, the build process makes sure that no contamination is done with the native components. An example will emphasize this information better:

ls -l build-test/tmp/sysroots/
total 12K
drwxr-xr-x 8 alex alex 4,0K sep 28 04:17 qemux86/
drwxr-xr-x 5 alex alex 4,0K sep 28 00:48 qemux86-tcbootstrap/
drwxr-xr-x 9 alex alex 4,0K sep 28 04:21 x86_64-linux/

ls -l build-test/tmp/sysroots/qemux86/ 
total 24K
drwxr-xr-x 2 alex alex 4,0K sep 28 01:52 etc/
drwxr-xr-x 5 alex alex 4,0K sep 28 04:15 lib/
drwxr-xr-x 6 alex alex 4,0K sep 28 03:51 pkgdata/
drwxr-xr-x 2 alex alex 4,0K sep 28 04:17 sysroot-providers/
drwxr-xr-x 7 alex alex 4,0K sep 28 04:16 usr/
drwxr-xr-x 3 alex alex 4,0K sep 28 01:52 var/

The Yocto project contributes to making reliable embedded Linux development and because of its dimensions, it is used for lots of things, ranging from board support packages by hardware companies to new software solutions by software development companies. Yocto is not a perfect tool and it has certain drawbacks:

There are also other things that bother developers, such as ptest integration and SDK sysroot's lack of extensibility, but a part of them are solved by the big community behind the project, and until the project shows its limitations, a new one will still need to wait to take its place. Until this happens, Yocto is the framework to use to develop custom embedded Linux distribution or products based in Linux.

To make sure that you write robust code, a number of guidelines should be mentioned. The first one refers to the fact that limitations should not be used for any data structure, including files, file names, lines, and symbols, and especially arbitrary limitations. All data structures should be dynamically allocated. One of the reasons for this is represented by the fact that most Unix utilities silently truncate long lines; GNU utilities do not do these kind of things.

Utilities that are used to read files should avoid dropping null characters or nonprinting characters. The exception here is when these utilities, that are intended for interfacing with certain types of printers or terminals, are unable to handle the previously mentioned characters. The advice that I'd give in this case would be to try and make programs work with a UTF-8 character set, or other sequences of bytes used to represent multibyte characters.

Make sure that you check system calls for error return values; the exception here is when a developer wishes to ignore the errors. It would be a good idea to include the system error text from strerror, perror, or equivalent error handling functions, in error messages that result from a crashed on system call, adding the name of the source code file, and also the name of the utility. This is done to make sure that the error message is easy to read and understand by anyone involved in the interaction with the source code or the program.

Check the return value for malloc or realloc to verify if they've returned zero. In case realloc is used in order to make a block smaller in systems that approximate block dimensions to powers of 2, realloc may have a different behavior and get a different block. In Unix, when realloc has a bug, it destroys the storage block for a zero return value. For GNU, this bug does not occur, and when it fails, the original block remains unchanged. If you want to run the same program on Unix and do not want to lose data, you could check if the bug was resolved on the Unix system or use the malloc GNU.

The content of the block that was freed is not accessible to alter or for any other interactions from the user. This can be done before calling free.

When a malloc command fails in a noninteractive program, we face a fatal error. In case the same situation is repeated, but, this time, an interactive program is involved, it would be better to abort the command and return to the read loop. This offers the possibility to free up virtual memory, kill other processes, and retry the command.

To decode arguments, the getopt_long option can be used.

When writing static storage during program execution, use C code for its initialization. However, for data that will not be changed, reserve C initialized declarations.

Try to keep away from low-level interfaces to unknown Unix data structures - this could happen when the data structure do not work in a compatible fashion. For example, to find all the files inside a directory, a developer could use the readdir function, or any high-level interface available function, since these do not have compatibility problems.

For signal handling, use the BSD variant of signal and the POSIX sigaction function. The USG signal interface is not the best alternative in this case. Using POSIX signal functions is nowadays considered the easiest way to develop a portable program. However, the use of one function over another is completely up to the developer.

For error checks that identify impossible situations, just abort the program, since there is no need to print any messages. These type of checks bear witness to the existence of bugs. To fix these bugs, a developer will have to inspect the available source code and even start a debugger. The best approach to solve this problem would be to describe the bugs and problems using comments inside the source code. The relevant information could be found inside variables after examining them accordingly with a debugger.

Do not use a count of the encountered errors in a program as an exit status. This practice is not the best, mostly because the values for an exit status are limited to 8 bits only, and an execution of the executable might have more than 255 errors. For example, if you try to return exit status 256 for a process, the parent process will see a status of zero and consider that the program finished successfully.

If temporary files are created, checking that the TMPDIR environment variable would be a good idea. If the variable is defined, it would be wise to use the /tmp directory instead. The use of temporary files should be done with caution because there is the possibility of security breaches occurring when creating them in world-writable directories. For C language, this can be avoided by creating temporary files in the following manner:

fd = open (filename, O_WRONLY | O_CREAT | O_EXCL, 0600);

This can also be done using the mkstemps function, which is made available by Gnulib.

For a bash environment, use the noclobber environment variable, or the set -C short version, to avoid the previously mentioned problem. Furthermore, the mktemp available utility is altogether a better solution for making a temporary file a shell environment; this utility is available in the GNU Coreutils package.

After the introduction of the packages that comprise a toolchain, this section will introduce the steps needed to obtain a custom toolchain. The toolchain that will be generated will contain the same sources as the ones available inside the Poky dizzy branch. Here, I am referring to the gcc version 4.9, binutils version 2.24, and glibc version 2.20. For Ubuntu systems, there are also shortcuts available. A generic toolchain can be installed using the available package manager, and there are also alternatives, such as downloading custom toolchains available inside Board Support Packages, or even from third parties, including CodeSourcery and Linaro. More information on toolchains can be found at http://elinux.org/Toolchains. The architecture that will be used as a demo is an ARM architecture.

The toolchain build process has eight steps. I will only outline the activities required for each one of them, but I must mention that they are all automatized inside the Yocto Project recipes. Inside the Yocto Project section, the toolchain is generated without notice. For interaction with the generated toolchain, the simplest task would be to call meta-ide-support, but this will be presented in the appropriate section as follows:

After these steps are performed, a toolchain will be available for the developer to use. The same strategy and build procedure steps is followed inside the Yocto Project.

should be dynamically allocated. One of the reasons for this is represented by the fact that most Unix utilities silently truncate long lines; GNU utilities do not do these kind of things.

Utilities that are used to read files should avoid dropping null characters or nonprinting characters. The exception here is when these utilities, that are intended for interfacing with certain types of printers or terminals, are unable to handle the previously mentioned characters. The advice that I'd give in this case would be to try and make programs work with a UTF-8 character set, or other sequences of bytes used to represent multibyte characters.

Make sure that you check system calls for error return values; the exception here is when a developer wishes to ignore the errors. It would be a good idea to include the system error text from strerror, perror, or equivalent error handling functions, in error messages that result from a crashed on system call, adding the name of the source code file, and also the name of the utility. This is done to make sure that the error message is easy to read and understand by anyone involved in the interaction with the source code or the program.

Check the return value for malloc or realloc to verify if they've returned zero. In case realloc is used in order to make a block smaller in systems that approximate block dimensions to powers of 2, realloc may have a different behavior and get a different block. In Unix, when realloc has a bug, it destroys the storage block for a zero return value. For GNU, this bug does not occur, and when it fails, the original block remains unchanged. If you want to run the same program on Unix and do not want to lose data, you could check if the bug was resolved on the Unix system or use the malloc GNU.

The content of the block that was freed is not accessible to alter or for any other interactions from the user. This can be done before calling free.

When a malloc command fails in a noninteractive program, we face a fatal error. In case the same situation is repeated, but, this time, an interactive program is involved, it would be better to abort the command and return to the read loop. This offers the possibility to free up virtual memory, kill other processes, and retry the command.

To decode arguments, the getopt_long option can be used.

When writing static storage during program execution, use C code for its initialization. However, for data that will not be changed, reserve C initialized declarations.

Try to keep away from low-level interfaces to unknown Unix data structures - this could happen when the data structure do not work in a compatible fashion. For example, to find all the files inside a directory, a developer could use the readdir function, or any high-level interface available function, since these do not have compatibility problems.

For signal handling, use the BSD variant of signal and the POSIX sigaction function. The USG signal interface is not the best alternative in this case. Using POSIX signal functions is nowadays considered the easiest way to develop a portable program. However, the use of one function over another is completely up to the developer.

For error checks that identify impossible situations, just abort the program, since there is no need to print any messages. These type of checks bear witness to the existence of bugs. To fix these bugs, a developer will have to inspect the available source code and even start a debugger. The best approach to solve this problem would be to describe the bugs and problems using comments inside the source code. The relevant information could be found inside variables after examining them accordingly with a debugger.

Do not use a count of the encountered errors in a program as an exit status. This practice is not the best, mostly because the values for an exit status are limited to 8 bits only, and an execution of the executable might have more than 255 errors. For example, if you try to return exit status 256 for a process, the parent process will see a status of zero and consider that the program finished successfully.

If temporary files are created, checking that the TMPDIR environment variable would be a good idea. If the variable is defined, it would be wise to use the /tmp directory instead. The use of temporary files should be done with caution because there is the possibility of security breaches occurring when creating them in world-writable directories. For C language, this can be avoided by creating temporary files in the following manner:

fd = open (filename, O_WRONLY | O_CREAT | O_EXCL, 0600);

This can also be done using the mkstemps function, which is made available by Gnulib.

For a bash environment, use the noclobber environment variable, or the set -C short version, to avoid the previously mentioned problem. Furthermore, the mktemp available utility is altogether a better solution for making a temporary file a shell environment; this utility is available in the GNU Coreutils package.

After the introduction of the packages that comprise a toolchain, this section will introduce the steps needed to obtain a custom toolchain. The toolchain that will be generated will contain the same sources as the ones available inside the Poky dizzy branch. Here, I am referring to the gcc version 4.9, binutils version 2.24, and glibc version 2.20. For Ubuntu systems, there are also shortcuts available. A generic toolchain can be installed using the available package manager, and there are also alternatives, such as downloading custom toolchains available inside Board Support Packages, or even from third parties, including CodeSourcery and Linaro. More information on toolchains can be found at http://elinux.org/Toolchains. The architecture that will be used as a demo is an ARM architecture.

The toolchain build process has eight steps. I will only outline the activities required for each one of them, but I must mention that they are all automatized inside the Yocto Project recipes. Inside the Yocto Project section, the toolchain is generated without notice. For interaction with the generated toolchain, the simplest task would be to call meta-ide-support, but this will be presented in the appropriate section as follows:

After these steps are performed, a toolchain will be available for the developer to use. The same strategy and build procedure steps is followed inside the Yocto Project.

steps needed to obtain a custom toolchain. The toolchain that will be generated will contain the same sources as the ones available inside the Poky dizzy branch. Here, I am referring to the gcc version 4.9, binutils version 2.24, and glibc version 2.20. For Ubuntu systems, there are also shortcuts available. A generic toolchain can be installed using the available package manager, and there are also alternatives, such as downloading custom toolchains available inside Board Support Packages, or even from third parties, including CodeSourcery and Linaro. More information on toolchains can be found at http://elinux.org/Toolchains. The architecture that will be used as a demo is an ARM architecture.

The toolchain build process has eight steps. I will only outline the activities required for each one of them, but I must mention that they are all automatized inside the Yocto Project recipes. Inside the Yocto Project section, the toolchain is generated without notice. For interaction with the generated toolchain, the simplest task would be to call meta-ide-support, but this will be presented in the appropriate section as follows:

After these steps are performed, a toolchain will be available for the developer to use. The same strategy and build procedure steps is followed inside the Yocto Project.

In an industrial environment, interaction with the U-Boot is mainly done through the Ethernet interface. Not only does an Ethernet interface enable the faster transfer of operating system images, but it is also less prone to errors than a serial connection.

One of the most important features available inside a bootloader is related to the support for Dynamic Host Control Protocol (DHCP), Trivial File Transfer Protocol (TFTP), and even Bootstrap Protocol (BOOTP). BOOTP and DHCP enable an Ethernet connection to configure itself and acquire an IP address from a specialized server. TFTP enables the download of files through a TFTP server. The messages passed between a target device and the DHCP/BOOTP servers are represented in the following image in a more generic manner. Initially, the hardware platform sends a broadcast message that arrives at all the DHCP/BOOTP servers available. Each server sends back its offer, which also contains an IP address, and the client accepts the one that suits its purposes the best and declines the other ones.

After the target device has finished communication with DHCP/BOOTP, it remains with a configuration that is specific to the target and contains information, such as the hostname, target IP and hardware Ethernet address (MAC address), netmask, tftp server IP address and even a TFTP filename. This information is bound to the Ethernet port and is used later in the booting process.

To boot images, U-Boot offers a number of capabilities that refer to the support of storage subsystems. These options include the RAM boot, MMC boot, NAND boot, NFS boot and so on. The support for these options is not always easy and could imply both hardware and software complexity.

I've mentioned previously that U-Boot is one of the most used and known bootloaders available. This is also due to the fact that its architecture enables the porting of new development platforms and processors in a very easy manner. At the same time, there are a huge number of development platforms available that could be used as references. The first thing that any developer who is interested in porting a new platform should do is to inspect the board and arch directories to establish their baselines, and, at the same time, also identify their similarities with other CPUs and available boards.

The board.cfg file is the starting point to register a new platform. Here, the following information should be added as a table line:

To port a machine similar to SAMA5D3 Xplained, one of the directories that could be consulted is the arch directory. It contains files, such as board.c, with information related to the initialization process for boards and SOCs. The most notable processes are board_init_r(), which does the setup and probing for board and peripherals after its relocation in the RAM, board_init_f(), which identifies the stack size and reserved address before its relocation in the RAM, and init_sequence[], which is called inside the board_init_f for the setup of peripherals. Other important files inside the same locations are the bootm.c and interrupts.c files. The former has the main responsibility of the boot from memory of the operating system, and the latter is responsible for implementation of generic interrupts.

The board directory also has some interesting files and functions that need to be mentioned here, such as the board/atmel/sama5d3_xplained/sama5d3_xplained.c file. It contains functions, such as board_init(), dram_init(), board_eth_init(), board_mmc_init, spl_board_ init(), and mem_init() that are used for initialization, and some of them called by the arch/arm/lib/board.c file.

Here are some other relevant directories:

More information about the SAMA5D3 Xplained board can be found by inspecting the corresponding doc directory and README files, such as README.at91, README.at91-soc, README.atmel_mci, README.atmel_pmecc, README.ARM-memory-map, and so on.

For people interested in committing to the changes they made while porting a new development board, CPU, or SOC to U-Boot, a few rules should be followed. All of these are related to the git interaction and help you to ensure the proper maintenance of your branches.

The first thing that a developer should do is to track the upstream branch that corresponds to a local branch. Another piece of advice would be to forget about git merge and instead use git rebase. Keeping in contact with the upstream repository can be done using the git fetch command. To work with patches, some general rules need to be followed, and patches need to have only one logical change, which can be any one of these:

Let's take a look at following diagram, which illustrates the git rebase operation:

As shown in both the preceding and following diagram, the git rebase operation has recreated the work from one branch onto another. Every commit from one branch is made available on the succeeding one, just after the last commit from it.

The git merge operation, on the other hand, is a new commit that has two parents: the branch from which it was ported, and the new branch on which it was merged. In fact, it gathers a series of commits into one branch with a different commit ID, which is why they are difficult to manage.

More information related to git interactions can be found at http://git-scm.com/documentation or http://www.denx.de/wiki/U-Boot/Patches.

Almost always when porting a new feature in U-Boot, debugging is involved. For a U-Boot debugger, there are two different situations that can occur:

In the next few lines, the second situation will be considered: the baseline enabling a debugging session for U-Boot. To make sure that debugging is possible, the elf file needs to be executed. Also, it cannot be manipulated directly because the linking address will be relocated. For this, a few tricks should be used:

An early development platform can be set up after relocation takes place and the proper breakpoint should be set after the relocation has ended. A symbol needs to be reloaded for U-Boot because the relocation will move the linking address. For all of these tasks, a gdb script is indicated as gdb gdb-script.sh:

#!/bin/sh
${CROSS_COMPILE}-gdb u-boot –command=gdb-command-script.txt

vim gdb-command-script.txt
target remote ${ip}:${port}
load
set symbol-reloading
# break the process before calling board_init_r() function
b start.S:79
c
…
# the symbol-file need to be align to the address available after relocation
add-symbol-file u-boot ${addr}
# break the process at board_init_r() function for single stepping b board.c:494

environment, interaction with the U-Boot is mainly done through the Ethernet interface. Not only does an Ethernet interface enable the faster transfer of operating system images, but it is also less prone to errors than a serial connection.

One of the most important features available inside a bootloader is related to the support for Dynamic Host Control Protocol (DHCP), Trivial File Transfer Protocol (TFTP), and even Bootstrap Protocol (BOOTP). BOOTP and DHCP enable an Ethernet connection to configure itself and acquire an IP address from a specialized server. TFTP enables the download of files through a TFTP server. The messages passed between a target device and the DHCP/BOOTP servers are represented in the following image in a more generic manner. Initially, the hardware platform sends a broadcast message that arrives at all the DHCP/BOOTP servers available. Each server sends back its offer, which also contains an IP address, and the client accepts the one that suits its purposes the best and declines the other ones.

After the target device has finished communication with DHCP/BOOTP, it remains with a configuration that is specific to the target and contains information, such as the hostname, target IP and hardware Ethernet address (MAC address), netmask, tftp server IP address and even a TFTP filename. This information is bound to the Ethernet port and is used later in the booting process.

To boot images, U-Boot offers a number of capabilities that refer to the support of storage subsystems. These options include the RAM boot, MMC boot, NAND boot, NFS boot and so on. The support for these options is not always easy and could imply both hardware and software complexity.

I've mentioned previously that U-Boot is one of the most used and known bootloaders available. This is also due to the fact that its architecture enables the porting of new development platforms and processors in a very easy manner. At the same time, there are a huge number of development platforms available that could be used as references. The first thing that any developer who is interested in porting a new platform should do is to inspect the board and arch directories to establish their baselines, and, at the same time, also identify their similarities with other CPUs and available boards.

The board.cfg file is the starting point to register a new platform. Here, the following information should be added as a table line:

To port a machine similar to SAMA5D3 Xplained, one of the directories that could be consulted is the arch directory. It contains files, such as board.c, with information related to the initialization process for boards and SOCs. The most notable processes are board_init_r(), which does the setup and probing for board and peripherals after its relocation in the RAM, board_init_f(), which identifies the stack size and reserved address before its relocation in the RAM, and init_sequence[], which is called inside the board_init_f for the setup of peripherals. Other important files inside the same locations are the bootm.c and interrupts.c files. The former has the main responsibility of the boot from memory of the operating system, and the latter is responsible for implementation of generic interrupts.

The board directory also has some interesting files and functions that need to be mentioned here, such as the board/atmel/sama5d3_xplained/sama5d3_xplained.c file. It contains functions, such as board_init(), dram_init(), board_eth_init(), board_mmc_init, spl_board_ init(), and mem_init() that are used for initialization, and some of them called by the arch/arm/lib/board.c file.

Here are some other relevant directories:

More information about the SAMA5D3 Xplained board can be found by inspecting the corresponding doc directory and README files, such as README.at91, README.at91-soc, README.atmel_mci, README.atmel_pmecc, README.ARM-memory-map, and so on.

For people interested in committing to the changes they made while porting a new development board, CPU, or SOC to U-Boot, a few rules should be followed. All of these are related to the git interaction and help you to ensure the proper maintenance of your branches.

The first thing that a developer should do is to track the upstream branch that corresponds to a local branch. Another piece of advice would be to forget about git merge and instead use git rebase. Keeping in contact with the upstream repository can be done using the git fetch command. To work with patches, some general rules need to be followed, and patches need to have only one logical change, which can be any one of these:

Let's take a look at following diagram, which illustrates the git rebase operation:

As shown in both the preceding and following diagram, the git rebase operation has recreated the work from one branch onto another. Every commit from one branch is made available on the succeeding one, just after the last commit from it.

The git merge operation, on the other hand, is a new commit that has two parents: the branch from which it was ported, and the new branch on which it was merged. In fact, it gathers a series of commits into one branch with a different commit ID, which is why they are difficult to manage.

More information related to git interactions can be found at http://git-scm.com/documentation or http://www.denx.de/wiki/U-Boot/Patches.

Almost always when porting a new feature in U-Boot, debugging is involved. For a U-Boot debugger, there are two different situations that can occur:

In the next few lines, the second situation will be considered: the baseline enabling a debugging session for U-Boot. To make sure that debugging is possible, the elf file needs to be executed. Also, it cannot be manipulated directly because the linking address will be relocated. For this, a few tricks should be used:

An early development platform can be set up after relocation takes place and the proper breakpoint should be set after the relocation has ended. A symbol needs to be reloaded for U-Boot because the relocation will move the linking address. For all of these tasks, a gdb script is indicated as gdb gdb-script.sh:

#!/bin/sh
${CROSS_COMPILE}-gdb u-boot –command=gdb-command-script.txt

vim gdb-command-script.txt
target remote ${ip}:${port}
load
set symbol-reloading
# break the process before calling board_init_r() function
b start.S:79
c
…
# the symbol-file need to be align to the address available after relocation
add-symbol-file u-boot ${addr}
# break the process at board_init_r() function for single stepping b board.c:494

previously that U-Boot is one of the most used and known bootloaders available. This is also due to the fact that its architecture enables the porting of new development platforms and processors in a very easy manner. At the same time, there are a huge number of development platforms available that could be used as references. The first thing that any developer who is interested in porting a new platform should do is to inspect the board and arch directories to establish their baselines, and, at the same time, also identify their similarities with other CPUs and available boards.

The board.cfg file is the starting point to register a new platform. Here, the following information should be added as a table line:

To port a machine similar to SAMA5D3 Xplained, one of the directories that could be consulted is the arch directory. It contains files, such as board.c, with information related to the initialization process for boards and SOCs. The most notable processes are board_init_r(), which does the setup and probing for board and peripherals after its relocation in the RAM, board_init_f(), which identifies the stack size and reserved address before its relocation in the RAM, and init_sequence[], which is called inside the board_init_f for the setup of peripherals. Other important files inside the same locations are the bootm.c and interrupts.c files. The former has the main responsibility of the boot from memory of the operating system, and the latter is responsible for implementation of generic interrupts.

The board directory also has some interesting files and functions that need to be mentioned here, such as the board/atmel/sama5d3_xplained/sama5d3_xplained.c file. It contains functions, such as board_init(), dram_init(), board_eth_init(), board_mmc_init, spl_board_ init(), and mem_init() that are used for initialization, and some of them called by the arch/arm/lib/board.c file.

Here are some other relevant directories:

More information about the SAMA5D3 Xplained board can be found by inspecting the corresponding doc directory and README files, such as README.at91, README.at91-soc, README.atmel_mci, README.atmel_pmecc, README.ARM-memory-map, and so on.

For people interested in committing to the changes they made while porting a new development board, CPU, or SOC to U-Boot, a few rules should be followed. All of these are related to the git interaction and help you to ensure the proper maintenance of your branches.

The first thing that a developer should do is to track the upstream branch that corresponds to a local branch. Another piece of advice would be to forget about git merge and instead use git rebase. Keeping in contact with the upstream repository can be done using the git fetch command. To work with patches, some general rules need to be followed, and patches need to have only one logical change, which can be any one of these:

Let's take a look at following diagram, which illustrates the git rebase operation:

As shown in both the preceding and following diagram, the git rebase operation has recreated the work from one branch onto another. Every commit from one branch is made available on the succeeding one, just after the last commit from it.

The git merge operation, on the other hand, is a new commit that has two parents: the branch from which it was ported, and the new branch on which it was merged. In fact, it gathers a series of commits into one branch with a different commit ID, which is why they are difficult to manage.

More information related to git interactions can be found at http://git-scm.com/documentation or http://www.denx.de/wiki/U-Boot/Patches.

Almost always when porting a new feature in U-Boot, debugging is involved. For a U-Boot debugger, there are two different situations that can occur:

In the next few lines, the second situation will be considered: the baseline enabling a debugging session for U-Boot. To make sure that debugging is possible, the elf file needs to be executed. Also, it cannot be manipulated directly because the linking address will be relocated. For this, a few tricks should be used:

An early development platform can be set up after relocation takes place and the proper breakpoint should be set after the relocation has ended. A symbol needs to be reloaded for U-Boot because the relocation will move the linking address. For all of these tasks, a gdb script is indicated as gdb gdb-script.sh:

#!/bin/sh
${CROSS_COMPILE}-gdb u-boot –command=gdb-command-script.txt

vim gdb-command-script.txt
target remote ${ip}:${port}
load
set symbol-reloading
# break the process before calling board_init_r() function
b start.S:79
c
…
# the symbol-file need to be align to the address available after relocation
add-symbol-file u-boot ${addr}
# break the process at board_init_r() function for single stepping b board.c:494

When discussing the memory in Linux, we can refer to it as the physical and virtual memory. Compartments of the RAM memory are used for the containment of the Linux kernel variables and data structures, the rest of the memory being used for dynamic allocations, as described here:

The physical memory defines algorithms and data structures that are able to maintain the memory, and it is done at the page level relatively independently by the virtual memory. Here, each physical page has a struct page descriptor associated with it that is used to incorporate information about the physical page. Each page has a struct page descriptor defined. Some of the fields of this structure are as follows:

The zones of the physical memory have been previously. The physical memory is split up into multiple nodes that have a common physical address space and a fast local memory access. The smallest of them is ZONE_DMA between 0 to 16Mb. The next is ZONE_NORMAL, which is the LowMem area between 16Mb to 896Mb, and the largest one is ZONE_HIGHMEM, which is between 900Mb to 4GB/64Gb. This information can be visible both in the preceding and following images:

The virtual memory is used both in the user space and the kernel space. The allocation for a memory zone implies the allocation of a physical page as well as the allocation of an address space area; this is done both in the page table and in the internal structures available inside the operating system. The usage of the page table differs from one architecture type to another. For the Complex instruction set computing (CISC) architecture, the page table is used by the processor, but on a Reduced instruction set computing (RISC) architecture, the page table is used by the core for a page lookup and translation lookaside buffer (TLB) add operations. Each zone descriptor is used for zone mapping. It specifies whether the zone is mapped for usage by a file if the zone is read-only, copy-on-write, and so on. The address space descriptor is used by the operating system to maintain high-level information.

The memory allocation is different between the user space and kernel space context because the kernel space memory allocation is not able to allocate memory in an easy manner. This difference is mostly due to the fact that error management in the kernel context is not easily done, or at least not in the same key as the user space context. This is one of the problems that will be presented in this section along with the solutions because it helps readers understand how memory management is done in the context of the Linux kernel.

The methods used by the kernel for memory handling is the first subject that will be discussed here. This is done to make sure that you understand the methods used by the kernel to obtain memory. Although the smallest addressable unit of a processor is a byte, the Memory Management Unit (MMU), the unit responsible for virtual to physical translation the smallest addressable unit is the page. A page's size varies from one architecture to another. It is responsible for maintaining the system's page tables. Most of 32-bit architectures use 4KB pages, whereas the 64-bit ones usually have 8KB pages. For the Atmel SAMA5D3-Xplained board, the definition of the struct page structure is as follows:

struct page {
        unsigned long 	flags;
        atomic_t        _count;
        atomic_t        _mapcount;
        struct address_space *mapping;
        void        *virtual;
        unsigned long 	debug_flags;
        void        *shadow;
        int        _last_nid;

};

This is one of the most important fields of the page structure. The flags field, for example, represents the status of the page; this holds information, such as whether the page is dirty or not, locked, or in another valid state. The values that are associated with this flag are defined inside the include/linux/page-flags-layout.h header file. The virtual field represents the virtual address associated with the page, count represents the count value for the page that is usually accessible indirectly through the page_count() function. All the other fields can be accessed inside the include/linux/mm_types.h header file.

The kernel divides the hardware into various zone of memory, mostly because there are pages in the physical memory that are not accessible for a number of the tasks. For example, there are hardware devices that can perform DMA. These actions are done by interacting with only a zone of the physical memory, simply called ZONE_DMA. It is accessible between 0-16 Mb for x86 architectures.

There are four main memory zones available and other two less notable ones that are defined inside the kernel sources in the include/linux/mmzone.h header file. The zone mapping is also architecture-dependent for the Atmel SAMA5D3-Xplained board. We have the following zones defined:

enum zone_type {
#ifdef CONFIG_ZONE_DMA
        /*
         * ZONE_DMA is used when there are devices that are not able
         * to do DMA to all of addressable memory (ZONE_NORMAL). Then we
         * carve out the portion of memory that is needed for these devices.
         * The range is arch specific.
         *
         * Some examples
         *
         * Architecture         Limit
         * ---------------------------
         * parisc, ia64, sparc  <4G
         * s390                 <2G
         * arm                  Various
         * alpha                Unlimited or 0-16MB.
         *
         * i386, x86_64 and multiple other arches
         *                      <16M.
         */
        ZONE_DMA,
#endif
#ifdef CONFIG_ZONE_DMA32
        /*
         * x86_64 needs two ZONE_DMAs because it supports devices that are
         * only able to do DMA to the lower 16M but also 32 bit devices that
         * can only do DMA areas below 4G.
         */
        ZONE_DMA32,
#endif
        /*
         * Normal addressable memory is in ZONE_NORMAL. DMA operations can be
         * performed on pages in ZONE_NORMAL if the DMA devices support
         * transfers to all addressable memory.
         */
        ZONE_NORMAL,
#ifdef CONFIG_HIGHMEM
        /*
         * A memory area that is only addressable by the kernel through
         * mapping portions into its own address space. This is for example
         * used by i386 to allow the kernel to address the memory beyond
         * 900MB. The kernel will set up special mappings (page
         * table entries on i386) for each page that the kernel needs to
         * access.
         */
        ZONE_HIGHMEM,
#endif
        ZONE_MOVABLE,
        __MAX_NR_ZONES
};

There are allocations that require interaction with more than one zone. One such example is a normal allocation that is able to use either ZONE_DMA or ZONE_NORMAL. ZONE_NORMAL is preferred because it does not interfere with direct memory accesses, though when the memory is at full usage, the kernel might use other available zones besides the ones that it uses in normal scenarios. The kernel that is available is a struct zone structure that defines each zone's relevant information. For the Atmel SAMA5D3-Xplained board, this structure is as shown here:

struct zone {
        unsigned long 	watermark[NR_WMARK];
        unsigned long 	percpu_drift_mark;
        unsigned long 	lowmem_reserve[MAX_NR_ZONES];
        unsigned long 	dirty_balance_reserve;
        struct per_cpu_pageset __percpu *pageset;
        spinlock_t        lock;
        int        all_unreclaimable;
        struct free_area        free_area[MAX_ORDER];
        unsigned int            compact_considered;
        unsigned int            compact_defer_shift;
        int                     compact_order_failed;
        spinlock_t              lru_lock;
        struct lruvec           lruvec;
        unsigned long         pages_scanned;
        unsigned long         flags;
        unsigned int        inactive_ratio;
        wait_queue_head_t       * wait_table;
        unsigned long         wait_table_hash_nr_entries;
        unsigned long         wait_table_bits;
        struct pglist_data    *zone_pgdat;
        unsigned long         zone_start_pfn;
        unsigned long         spanned_pages;
        unsigned long         present_pages;
        unsigned long         managed_pages;
        const char              *name;
};

As you can see, the zone that defines the structure is an impressive one. Some of the most interesting fields are represented by the watermark variable, which contain the high, medium, and low watermarks for the defined zone. The present_pages attribute represents the available pages within the zone. The name field represents the name of the zone, and others, such as the lock field, a spin lock that shields the zone structure for simultaneous access. All the other fields that can be identified inside the corresponding include/linux/mmzone.h header file for the Atmel SAMA5D3 Xplained board.

With this information available, we can move ahead and find out how the kernel implements memory allocation. All the available functions that are necessary for memory allocation and memory interaction in general, are inside the linux/gfp.h header file. Some of these functions are:

struct page * alloc_pages(gfp_t gfp_mask, unsigned int order)

This function is used to allocate physical pages in a continuous location. At the end, the return value is represented by the pointer of the first page structure if the allocation is successful, or NULL if errors occur:

void * page_address(struct page *page)

This function is used to get the logical address for a corresponding memory page:

unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order)

This one is similar to the alloc_pages() function, but the difference is that the return variable is offered in the struct page * alloc_page(gfp_t gfp_mask) return argument:

unsigned long __get_free_page(gfp_t gfp_mask)
struct page * alloc_page(gfp_t gfp_mask)

The preceding two functions are wrappers over similar ones, the difference is that this function returns only one page information. The order for this function has the zero value:

unsigned long get_zeroed_page(unsigned int gfp_mask)

The preceding function does what the name suggests. It returns the page full of zero values. The difference between this function and the __get_free_page() function is that after being released, the page is filled with zero values:

void __free_pages(struct page *page, unsigned int order)
void free_pages(unsigned long addr, unsigned int order)
void free_page(unsigned long addr)

The preceding functions are used for freeing the given allocated pages. The passing of the pages should be done with care because the kernel is not able to check the information it is provided.

A process, as presented previously, is a fundamental unit in a Linux operating system and at the same time, is a form of abstraction. It is, in fact, a program in execution, but a program by itself is not a process. It needs to be in an active state and have associated resources. A process is able to become a parent by using the fork() function, which spawns a child process. Both parent and child processes reside in separate address spaces, but both of them have the same content. The exec() family of function is the one that is able to execute a different program, create an address space, and load it inside that address space.

When fork() is used, the resources that the parent process has are reproduced for the child. This function is implemented in a very interesting manner; it uses the clone() system call that, at it's base, contains the copy_process() function. This functions does the following:

Threads in Linux are very similar to processes. They are viewed as processes that share various resources, such as memory address space, open files, and so on. The creation of threads is similar to a normal task, the exception being the clone() function, which passes flags that mention shared resources. For example, the clone function calls for a thread, which is clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND, 0), while for the normal fork looks similar to clone(SIGCHLD, 0).

The notion of kernel threads appeared as a solution to problems involving tasks running in the background of the kernel context. The kernel thread does not have an address space and is only available inside the kernel context. It has the same properties as a normal process, but is only used for special tasks, such as ksoftirqd, flush, and so on.

At the end of the execution, the process need to be terminated so that the resources can be freed, and the parent of the executing process needs to be notified about this. The method that is most used to terminate a process is done by calling the exit() system call. A number of steps are needed for this process:

After the preceding steps are performed, the object associated with this task is freed and it becomes unrunnable. Its memory exists solely as information for its parent. After its parent announces that this information is of no use to it, this memory is freed for the system to use.

The process scheduler decides which resources are allocated for a runnable process. It is a piece of software that is responsible for multitasking, resource allocation to various processes, and decides how to best set the resources and processor time. it also decides which processes should run next.

The first design of the Linux scheduler was very simplistic. It was not able to scale properly when the number of processes increased, so from the 2.5 kernel version, a new scheduler was developed. It is called O(1) scheduler and offers a constant time algorithm for time slice calculation and a run queue that is defined on a per-processor basis. Although it is perfect for large servers, it is not the best solution for a normal desktop system. From the 2.6 kernel version, improvements have been made to the O(1) scheduler, such as the fair scheduling concept that later materialized from the kernel version 2.6.23 into the Completely Fair Scheduler (CFS), which became the defacto scheduler.

The CFC has a simple idea behind. It behaves as if we have a perfect multitasking processor where each process gets 1/n slice of the processor's time and this time slice is an incredibly small. The n value represents the number of running processes. Con Kolivas is the Australian programmer that contributed to the fair scheduling implementation, also known as Rotating Staircase Deadline Scheduler (RSDL). Its implementation required a red-black tree for the priorities of self-balancing and also a time slice that is calculated at the nanosecond level. Similarly to the O(1) scheduler, CFS applies the notion of weight, which implies that some processes wait more than others. This is based on the weighed fair queuing algorithm.

A process scheduler constitutes one of the most important components of the Linux kernel because it defines the user interaction with the operating system in general. The Linux kernel CFS is the scheduler that appeals to developers and users because it offers scalability and performance with the most reasonable approach.

For processes to interact with a system, an interface should be provided to give the user space application the possibility of interacting with hardware and other processes.System calls. These are used as an interface between the hardware and the user space. They are also used to ensure stability, security, and abstraction, in general. These are common layers that constitute an entry point into the kernel alongside traps and exceptions, as described here:

The interaction with most of the system calls that are available inside the Linux system is done using the C library. They are able to define a number of arguments and return a value that reveals whether they were successful or not. A value of zero usually means that the execution ended with success, and in case errors appear, an error code will be available inside the errno variable. When a system call is done, the following steps are followed:

A system call has a syscall number associated with it, which is a unique number used as a reference for the system call that cannot be changed (there is no possibility of implementing a system call). A symbolic constant for each system call number is available in the <sys/syscall.h> header file. To check the existence of a system call, sys_ni_syscall() is used, which returns the ENOSYS error for an invalid system call.

The Linux operating system is able to support a large variety of filesystem options. This is done due to the existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

discussing the memory in Linux, we can refer to it as the physical and virtual memory. Compartments of the RAM memory are used for the containment of the Linux kernel variables and data structures, the rest of the memory being used for dynamic allocations, as described here:

The physical memory defines algorithms and data structures that are able to maintain the memory, and it is done at the page level relatively independently by the virtual memory. Here, each physical page has a struct page descriptor associated with it that is used to incorporate information about the physical page. Each page has a struct page descriptor defined. Some of the fields of this structure are as follows:

The zones of the physical memory have been previously. The physical memory is split up into multiple nodes that have a common physical address space and a fast local memory access. The smallest of them is ZONE_DMA between 0 to 16Mb. The next is ZONE_NORMAL, which is the LowMem area between 16Mb to 896Mb, and the largest one is ZONE_HIGHMEM, which is between 900Mb to 4GB/64Gb. This information can be visible both in the preceding and following images:

The virtual memory is used both in the user space and the kernel space. The allocation for a memory zone implies the allocation of a physical page as well as the allocation of an address space area; this is done both in the page table and in the internal structures available inside the operating system. The usage of the page table differs from one architecture type to another. For the Complex instruction set computing (CISC) architecture, the page table is used by the processor, but on a Reduced instruction set computing (RISC) architecture, the page table is used by the core for a page lookup and translation lookaside buffer (TLB) add operations. Each zone descriptor is used for zone mapping. It specifies whether the zone is mapped for usage by a file if the zone is read-only, copy-on-write, and so on. The address space descriptor is used by the operating system to maintain high-level information.

The memory allocation is different between the user space and kernel space context because the kernel space memory allocation is not able to allocate memory in an easy manner. This difference is mostly due to the fact that error management in the kernel context is not easily done, or at least not in the same key as the user space context. This is one of the problems that will be presented in this section along with the solutions because it helps readers understand how memory management is done in the context of the Linux kernel.

The methods used by the kernel for memory handling is the first subject that will be discussed here. This is done to make sure that you understand the methods used by the kernel to obtain memory. Although the smallest addressable unit of a processor is a byte, the Memory Management Unit (MMU), the unit responsible for virtual to physical translation the smallest addressable unit is the page. A page's size varies from one architecture to another. It is responsible for maintaining the system's page tables. Most of 32-bit architectures use 4KB pages, whereas the 64-bit ones usually have 8KB pages. For the Atmel SAMA5D3-Xplained board, the definition of the struct page structure is as follows:

struct page {
        unsigned long 	flags;
        atomic_t        _count;
        atomic_t        _mapcount;
        struct address_space *mapping;
        void        *virtual;
        unsigned long 	debug_flags;
        void        *shadow;
        int        _last_nid;

};

This is one of the most important fields of the page structure. The flags field, for example, represents the status of the page; this holds information, such as whether the page is dirty or not, locked, or in another valid state. The values that are associated with this flag are defined inside the include/linux/page-flags-layout.h header file. The virtual field represents the virtual address associated with the page, count represents the count value for the page that is usually accessible indirectly through the page_count() function. All the other fields can be accessed inside the include/linux/mm_types.h header file.

The kernel divides the hardware into various zone of memory, mostly because there are pages in the physical memory that are not accessible for a number of the tasks. For example, there are hardware devices that can perform DMA. These actions are done by interacting with only a zone of the physical memory, simply called ZONE_DMA. It is accessible between 0-16 Mb for x86 architectures.

There are four main memory zones available and other two less notable ones that are defined inside the kernel sources in the include/linux/mmzone.h header file. The zone mapping is also architecture-dependent for the Atmel SAMA5D3-Xplained board. We have the following zones defined:

enum zone_type {
#ifdef CONFIG_ZONE_DMA
        /*
         * ZONE_DMA is used when there are devices that are not able
         * to do DMA to all of addressable memory (ZONE_NORMAL). Then we
         * carve out the portion of memory that is needed for these devices.
         * The range is arch specific.
         *
         * Some examples
         *
         * Architecture         Limit
         * ---------------------------
         * parisc, ia64, sparc  <4G
         * s390                 <2G
         * arm                  Various
         * alpha                Unlimited or 0-16MB.
         *
         * i386, x86_64 and multiple other arches
         *                      <16M.
         */
        ZONE_DMA,
#endif
#ifdef CONFIG_ZONE_DMA32
        /*
         * x86_64 needs two ZONE_DMAs because it supports devices that are
         * only able to do DMA to the lower 16M but also 32 bit devices that
         * can only do DMA areas below 4G.
         */
        ZONE_DMA32,
#endif
        /*
         * Normal addressable memory is in ZONE_NORMAL. DMA operations can be
         * performed on pages in ZONE_NORMAL if the DMA devices support
         * transfers to all addressable memory.
         */
        ZONE_NORMAL,
#ifdef CONFIG_HIGHMEM
        /*
         * A memory area that is only addressable by the kernel through
         * mapping portions into its own address space. This is for example
         * used by i386 to allow the kernel to address the memory beyond
         * 900MB. The kernel will set up special mappings (page
         * table entries on i386) for each page that the kernel needs to
         * access.
         */
        ZONE_HIGHMEM,
#endif
        ZONE_MOVABLE,
        __MAX_NR_ZONES
};

There are allocations that require interaction with more than one zone. One such example is a normal allocation that is able to use either ZONE_DMA or ZONE_NORMAL. ZONE_NORMAL is preferred because it does not interfere with direct memory accesses, though when the memory is at full usage, the kernel might use other available zones besides the ones that it uses in normal scenarios. The kernel that is available is a struct zone structure that defines each zone's relevant information. For the Atmel SAMA5D3-Xplained board, this structure is as shown here:

struct zone {
        unsigned long 	watermark[NR_WMARK];
        unsigned long 	percpu_drift_mark;
        unsigned long 	lowmem_reserve[MAX_NR_ZONES];
        unsigned long 	dirty_balance_reserve;
        struct per_cpu_pageset __percpu *pageset;
        spinlock_t        lock;
        int        all_unreclaimable;
        struct free_area        free_area[MAX_ORDER];
        unsigned int            compact_considered;
        unsigned int            compact_defer_shift;
        int                     compact_order_failed;
        spinlock_t              lru_lock;
        struct lruvec           lruvec;
        unsigned long         pages_scanned;
        unsigned long         flags;
        unsigned int        inactive_ratio;
        wait_queue_head_t       * wait_table;
        unsigned long         wait_table_hash_nr_entries;
        unsigned long         wait_table_bits;
        struct pglist_data    *zone_pgdat;
        unsigned long         zone_start_pfn;
        unsigned long         spanned_pages;
        unsigned long         present_pages;
        unsigned long         managed_pages;
        const char              *name;
};

As you can see, the zone that defines the structure is an impressive one. Some of the most interesting fields are represented by the watermark variable, which contain the high, medium, and low watermarks for the defined zone. The present_pages attribute represents the available pages within the zone. The name field represents the name of the zone, and others, such as the lock field, a spin lock that shields the zone structure for simultaneous access. All the other fields that can be identified inside the corresponding include/linux/mmzone.h header file for the Atmel SAMA5D3 Xplained board.

With this information available, we can move ahead and find out how the kernel implements memory allocation. All the available functions that are necessary for memory allocation and memory interaction in general, are inside the linux/gfp.h header file. Some of these functions are:

struct page * alloc_pages(gfp_t gfp_mask, unsigned int order)

This function is used to allocate physical pages in a continuous location. At the end, the return value is represented by the pointer of the first page structure if the allocation is successful, or NULL if errors occur:

void * page_address(struct page *page)

This function is used to get the logical address for a corresponding memory page:

unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order)

This one is similar to the alloc_pages() function, but the difference is that the return variable is offered in the struct page * alloc_page(gfp_t gfp_mask) return argument:

unsigned long __get_free_page(gfp_t gfp_mask)
struct page * alloc_page(gfp_t gfp_mask)

The preceding two functions are wrappers over similar ones, the difference is that this function returns only one page information. The order for this function has the zero value:

unsigned long get_zeroed_page(unsigned int gfp_mask)

The preceding function does what the name suggests. It returns the page full of zero values. The difference between this function and the __get_free_page() function is that after being released, the page is filled with zero values:

void __free_pages(struct page *page, unsigned int order)
void free_pages(unsigned long addr, unsigned int order)
void free_page(unsigned long addr)

The preceding functions are used for freeing the given allocated pages. The passing of the pages should be done with care because the kernel is not able to check the information it is provided.

A process, as presented previously, is a fundamental unit in a Linux operating system and at the same time, is a form of abstraction. It is, in fact, a program in execution, but a program by itself is not a process. It needs to be in an active state and have associated resources. A process is able to become a parent by using the fork() function, which spawns a child process. Both parent and child processes reside in separate address spaces, but both of them have the same content. The exec() family of function is the one that is able to execute a different program, create an address space, and load it inside that address space.

When fork() is used, the resources that the parent process has are reproduced for the child. This function is implemented in a very interesting manner; it uses the clone() system call that, at it's base, contains the copy_process() function. This functions does the following:

Threads in Linux are very similar to processes. They are viewed as processes that share various resources, such as memory address space, open files, and so on. The creation of threads is similar to a normal task, the exception being the clone() function, which passes flags that mention shared resources. For example, the clone function calls for a thread, which is clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND, 0), while for the normal fork looks similar to clone(SIGCHLD, 0).

The notion of kernel threads appeared as a solution to problems involving tasks running in the background of the kernel context. The kernel thread does not have an address space and is only available inside the kernel context. It has the same properties as a normal process, but is only used for special tasks, such as ksoftirqd, flush, and so on.

At the end of the execution, the process need to be terminated so that the resources can be freed, and the parent of the executing process needs to be notified about this. The method that is most used to terminate a process is done by calling the exit() system call. A number of steps are needed for this process:

After the preceding steps are performed, the object associated with this task is freed and it becomes unrunnable. Its memory exists solely as information for its parent. After its parent announces that this information is of no use to it, this memory is freed for the system to use.

The process scheduler decides which resources are allocated for a runnable process. It is a piece of software that is responsible for multitasking, resource allocation to various processes, and decides how to best set the resources and processor time. it also decides which processes should run next.

The first design of the Linux scheduler was very simplistic. It was not able to scale properly when the number of processes increased, so from the 2.5 kernel version, a new scheduler was developed. It is called O(1) scheduler and offers a constant time algorithm for time slice calculation and a run queue that is defined on a per-processor basis. Although it is perfect for large servers, it is not the best solution for a normal desktop system. From the 2.6 kernel version, improvements have been made to the O(1) scheduler, such as the fair scheduling concept that later materialized from the kernel version 2.6.23 into the Completely Fair Scheduler (CFS), which became the defacto scheduler.

The CFC has a simple idea behind. It behaves as if we have a perfect multitasking processor where each process gets 1/n slice of the processor's time and this time slice is an incredibly small. The n value represents the number of running processes. Con Kolivas is the Australian programmer that contributed to the fair scheduling implementation, also known as Rotating Staircase Deadline Scheduler (RSDL). Its implementation required a red-black tree for the priorities of self-balancing and also a time slice that is calculated at the nanosecond level. Similarly to the O(1) scheduler, CFS applies the notion of weight, which implies that some processes wait more than others. This is based on the weighed fair queuing algorithm.

A process scheduler constitutes one of the most important components of the Linux kernel because it defines the user interaction with the operating system in general. The Linux kernel CFS is the scheduler that appeals to developers and users because it offers scalability and performance with the most reasonable approach.

For processes to interact with a system, an interface should be provided to give the user space application the possibility of interacting with hardware and other processes.System calls. These are used as an interface between the hardware and the user space. They are also used to ensure stability, security, and abstraction, in general. These are common layers that constitute an entry point into the kernel alongside traps and exceptions, as described here:

The interaction with most of the system calls that are available inside the Linux system is done using the C library. They are able to define a number of arguments and return a value that reveals whether they were successful or not. A value of zero usually means that the execution ended with success, and in case errors appear, an error code will be available inside the errno variable. When a system call is done, the following steps are followed:

A system call has a syscall number associated with it, which is a unique number used as a reference for the system call that cannot be changed (there is no possibility of implementing a system call). A symbolic constant for each system call number is available in the <sys/syscall.h> header file. To check the existence of a system call, sys_ni_syscall() is used, which returns the ENOSYS error for an invalid system call.

The Linux operating system is able to support a large variety of filesystem options. This is done due to the existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

A process, as presented previously, is a fundamental unit in a Linux operating system and at the same time, is a form of abstraction. It is, in fact, a program in execution, but a program by itself is not a process. It needs to be in an active state and have associated resources. A process is able to become a parent by using the fork() function, which spawns a child process. Both parent and child processes reside in separate address spaces, but both of them have the same content. The exec() family of function is the one that is able to execute a different program, create an address space, and load it inside that address space.

When fork() is used, the resources that the parent process has are reproduced for the child. This function is implemented in a very interesting manner; it uses the clone() system call that, at it's base, contains the copy_process() function. This functions does the following:

Threads in Linux are very similar to processes. They are viewed as processes that share various resources, such as memory address space, open files, and so on. The creation of threads is similar to a normal task, the exception being the clone() function, which passes flags that mention shared resources. For example, the clone function calls for a thread, which is clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND, 0), while for the normal fork looks similar to clone(SIGCHLD, 0).

The notion of kernel threads appeared as a solution to problems involving tasks running in the background of the kernel context. The kernel thread does not have an address space and is only available inside the kernel context. It has the same properties as a normal process, but is only used for special tasks, such as ksoftirqd, flush, and so on.

At the end of the execution, the process need to be terminated so that the resources can be freed, and the parent of the executing process needs to be notified about this. The method that is most used to terminate a process is done by calling the exit() system call. A number of steps are needed for this process:

After the preceding steps are performed, the object associated with this task is freed and it becomes unrunnable. Its memory exists solely as information for its parent. After its parent announces that this information is of no use to it, this memory is freed for the system to use.

The process scheduler decides which resources are allocated for a runnable process. It is a piece of software that is responsible for multitasking, resource allocation to various processes, and decides how to best set the resources and processor time. it also decides which processes should run next.

The first design of the Linux scheduler was very simplistic. It was not able to scale properly when the number of processes increased, so from the 2.5 kernel version, a new scheduler was developed. It is called O(1) scheduler and offers a constant time algorithm for time slice calculation and a run queue that is defined on a per-processor basis. Although it is perfect for large servers, it is not the best solution for a normal desktop system. From the 2.6 kernel version, improvements have been made to the O(1) scheduler, such as the fair scheduling concept that later materialized from the kernel version 2.6.23 into the Completely Fair Scheduler (CFS), which became the defacto scheduler.

The CFC has a simple idea behind. It behaves as if we have a perfect multitasking processor where each process gets 1/n slice of the processor's time and this time slice is an incredibly small. The n value represents the number of running processes. Con Kolivas is the Australian programmer that contributed to the fair scheduling implementation, also known as Rotating Staircase Deadline Scheduler (RSDL). Its implementation required a red-black tree for the priorities of self-balancing and also a time slice that is calculated at the nanosecond level. Similarly to the O(1) scheduler, CFS applies the notion of weight, which implies that some processes wait more than others. This is based on the weighed fair queuing algorithm.

A process scheduler constitutes one of the most important components of the Linux kernel because it defines the user interaction with the operating system in general. The Linux kernel CFS is the scheduler that appeals to developers and users because it offers scalability and performance with the most reasonable approach.

For processes to interact with a system, an interface should be provided to give the user space application the possibility of interacting with hardware and other processes.System calls. These are used as an interface between the hardware and the user space. They are also used to ensure stability, security, and abstraction, in general. These are common layers that constitute an entry point into the kernel alongside traps and exceptions, as described here:

The interaction with most of the system calls that are available inside the Linux system is done using the C library. They are able to define a number of arguments and return a value that reveals whether they were successful or not. A value of zero usually means that the execution ended with success, and in case errors appear, an error code will be available inside the errno variable. When a system call is done, the following steps are followed:

A system call has a syscall number associated with it, which is a unique number used as a reference for the system call that cannot be changed (there is no possibility of implementing a system call). A symbolic constant for each system call number is available in the <sys/syscall.h> header file. To check the existence of a system call, sys_ni_syscall() is used, which returns the ENOSYS error for an invalid system call.

The Linux operating system is able to support a large variety of filesystem options. This is done due to the existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

A process, as presented previously, is a fundamental unit in a Linux operating system and at the same time, is a form of abstraction. It is, in fact, a program in execution, but a program by itself is not a process. It needs to be in an active state and have associated resources. A process is able to become a parent by using the fork() function, which spawns a child process. Both parent and child processes reside in separate address spaces, but both of them have the same content. The exec() family of function is the one that is able to execute a different program, create an address space, and load it inside that address space.

When fork() is used, the resources that the parent process has are reproduced for the child. This function is implemented in a very interesting manner; it uses the clone() system call that, at it's base, contains the copy_process() function. This functions does the following:

Threads in Linux are very similar to processes. They are viewed as processes that share various resources, such as memory address space, open files, and so on. The creation of threads is similar to a normal task, the exception being the clone() function, which passes flags that mention shared resources. For example, the clone function calls for a thread, which is clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND, 0), while for the normal fork looks similar to clone(SIGCHLD, 0).

The notion of kernel threads appeared as a solution to problems involving tasks running in the background of the kernel context. The kernel thread does not have an address space and is only available inside the kernel context. It has the same properties as a normal process, but is only used for special tasks, such as ksoftirqd, flush, and so on.

At the end of the execution, the process need to be terminated so that the resources can be freed, and the parent of the executing process needs to be notified about this. The method that is most used to terminate a process is done by calling the exit() system call. A number of steps are needed for this process:

After the preceding steps are performed, the object associated with this task is freed and it becomes unrunnable. Its memory exists solely as information for its parent. After its parent announces that this information is of no use to it, this memory is freed for the system to use.

The process scheduler decides which resources are allocated for a runnable process. It is a piece of software that is responsible for multitasking, resource allocation to various processes, and decides how to best set the resources and processor time. it also decides which processes should run next.

The first design of the Linux scheduler was very simplistic. It was not able to scale properly when the number of processes increased, so from the 2.5 kernel version, a new scheduler was developed. It is called O(1) scheduler and offers a constant time algorithm for time slice calculation and a run queue that is defined on a per-processor basis. Although it is perfect for large servers, it is not the best solution for a normal desktop system. From the 2.6 kernel version, improvements have been made to the O(1) scheduler, such as the fair scheduling concept that later materialized from the kernel version 2.6.23 into the Completely Fair Scheduler (CFS), which became the defacto scheduler.

The CFC has a simple idea behind. It behaves as if we have a perfect multitasking processor where each process gets 1/n slice of the processor's time and this time slice is an incredibly small. The n value represents the number of running processes. Con Kolivas is the Australian programmer that contributed to the fair scheduling implementation, also known as Rotating Staircase Deadline Scheduler (RSDL). Its implementation required a red-black tree for the priorities of self-balancing and also a time slice that is calculated at the nanosecond level. Similarly to the O(1) scheduler, CFS applies the notion of weight, which implies that some processes wait more than others. This is based on the weighed fair queuing algorithm.

A process scheduler constitutes one of the most important components of the Linux kernel because it defines the user interaction with the operating system in general. The Linux kernel CFS is the scheduler that appeals to developers and users because it offers scalability and performance with the most reasonable approach.

For processes to interact with a system, an interface should be provided to give the user space application the possibility of interacting with hardware and other processes.System calls. These are used as an interface between the hardware and the user space. They are also used to ensure stability, security, and abstraction, in general. These are common layers that constitute an entry point into the kernel alongside traps and exceptions, as described here:

The interaction with most of the system calls that are available inside the Linux system is done using the C library. They are able to define a number of arguments and return a value that reveals whether they were successful or not. A value of zero usually means that the execution ended with success, and in case errors appear, an error code will be available inside the errno variable. When a system call is done, the following steps are followed:

A system call has a syscall number associated with it, which is a unique number used as a reference for the system call that cannot be changed (there is no possibility of implementing a system call). A symbolic constant for each system call number is available in the <sys/syscall.h> header file. To check the existence of a system call, sys_ni_syscall() is used, which returns the ENOSYS error for an invalid system call.

The Linux operating system is able to support a large variety of filesystem options. This is done due to the existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

presented previously, is a fundamental unit in a Linux operating system and at the same time, is a form of abstraction. It is, in fact, a program in execution, but a program by itself is not a process. It needs to be in an active state and have associated resources. A process is able to become a parent by using the fork() function, which spawns a child process. Both parent and child processes reside in separate address spaces, but both of them have the same content. The exec() family of function is the one that is able to execute a different program, create an address space, and load it inside that address space.

When fork() is used, the resources that the parent process has are reproduced for the child. This function is implemented in a very interesting manner; it uses the clone() system call that, at it's base, contains the copy_process() function. This functions does the following:

Threads in Linux are very similar to processes. They are viewed as processes that share various resources, such as memory address space, open files, and so on. The creation of threads is similar to a normal task, the exception being the clone() function, which passes flags that mention shared resources. For example, the clone function calls for a thread, which is clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND, 0), while for the normal fork looks similar to clone(SIGCHLD, 0).

The notion of kernel threads appeared as a solution to problems involving tasks running in the background of the kernel context. The kernel thread does not have an address space and is only available inside the kernel context. It has the same properties as a normal process, but is only used for special tasks, such as ksoftirqd, flush, and so on.

At the end of the execution, the process need to be terminated so that the resources can be freed, and the parent of the executing process needs to be notified about this. The method that is most used to terminate a process is done by calling the exit() system call. A number of steps are needed for this process:

After the preceding steps are performed, the object associated with this task is freed and it becomes unrunnable. Its memory exists solely as information for its parent. After its parent announces that this information is of no use to it, this memory is freed for the system to use.

The process scheduler decides which resources are allocated for a runnable process. It is a piece of software that is responsible for multitasking, resource allocation to various processes, and decides how to best set the resources and processor time. it also decides which processes should run next.

The first design of the Linux scheduler was very simplistic. It was not able to scale properly when the number of processes increased, so from the 2.5 kernel version, a new scheduler was developed. It is called O(1) scheduler and offers a constant time algorithm for time slice calculation and a run queue that is defined on a per-processor basis. Although it is perfect for large servers, it is not the best solution for a normal desktop system. From the 2.6 kernel version, improvements have been made to the O(1) scheduler, such as the fair scheduling concept that later materialized from the kernel version 2.6.23 into the Completely Fair Scheduler (CFS), which became the defacto scheduler.

The CFC has a simple idea behind. It behaves as if we have a perfect multitasking processor where each process gets 1/n slice of the processor's time and this time slice is an incredibly small. The n value represents the number of running processes. Con Kolivas is the Australian programmer that contributed to the fair scheduling implementation, also known as Rotating Staircase Deadline Scheduler (RSDL). Its implementation required a red-black tree for the priorities of self-balancing and also a time slice that is calculated at the nanosecond level. Similarly to the O(1) scheduler, CFS applies the notion of weight, which implies that some processes wait more than others. This is based on the weighed fair queuing algorithm.

A process scheduler constitutes one of the most important components of the Linux kernel because it defines the user interaction with the operating system in general. The Linux kernel CFS is the scheduler that appeals to developers and users because it offers scalability and performance with the most reasonable approach.

For processes to interact with a system, an interface should be provided to give the user space application the possibility of interacting with hardware and other processes.System calls. These are used as an interface between the hardware and the user space. They are also used to ensure stability, security, and abstraction, in general. These are common layers that constitute an entry point into the kernel alongside traps and exceptions, as described here:

The interaction with most of the system calls that are available inside the Linux system is done using the C library. They are able to define a number of arguments and return a value that reveals whether they were successful or not. A value of zero usually means that the execution ended with success, and in case errors appear, an error code will be available inside the errno variable. When a system call is done, the following steps are followed:

A system call has a syscall number associated with it, which is a unique number used as a reference for the system call that cannot be changed (there is no possibility of implementing a system call). A symbolic constant for each system call number is available in the <sys/syscall.h> header file. To check the existence of a system call, sys_ni_syscall() is used, which returns the ENOSYS error for an invalid system call.

The Linux operating system is able to support a large variety of filesystem options. This is done due to the existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

decides which resources are allocated for a runnable process. It is a piece of software that is responsible for multitasking, resource allocation to various processes, and decides how to best set the resources and processor time. it also decides which processes should run next.

The first design of the Linux scheduler was very simplistic. It was not able to scale properly when the number of processes increased, so from the 2.5 kernel version, a new scheduler was developed. It is called O(1) scheduler and offers a constant time algorithm for time slice calculation and a run queue that is defined on a per-processor basis. Although it is perfect for large servers, it is not the best solution for a normal desktop system. From the 2.6 kernel version, improvements have been made to the O(1) scheduler, such as the fair scheduling concept that later materialized from the kernel version 2.6.23 into the Completely Fair Scheduler (CFS), which became the defacto scheduler.

The CFC has a simple idea behind. It behaves as if we have a perfect multitasking processor where each process gets 1/n slice of the processor's time and this time slice is an incredibly small. The n value represents the number of running processes. Con Kolivas is the Australian programmer that contributed to the fair scheduling implementation, also known as Rotating Staircase Deadline Scheduler (RSDL). Its implementation required a red-black tree for the priorities of self-balancing and also a time slice that is calculated at the nanosecond level. Similarly to the O(1) scheduler, CFS applies the notion of weight, which implies that some processes wait more than others. This is based on the weighed fair queuing algorithm.

A process scheduler constitutes one of the most important components of the Linux kernel because it defines the user interaction with the operating system in general. The Linux kernel CFS is the scheduler that appeals to developers and users because it offers scalability and performance with the most reasonable approach.

For processes to interact with a system, an interface should be provided to give the user space application the possibility of interacting with hardware and other processes.System calls. These are used as an interface between the hardware and the user space. They are also used to ensure stability, security, and abstraction, in general. These are common layers that constitute an entry point into the kernel alongside traps and exceptions, as described here:

The interaction with most of the system calls that are available inside the Linux system is done using the C library. They are able to define a number of arguments and return a value that reveals whether they were successful or not. A value of zero usually means that the execution ended with success, and in case errors appear, an error code will be available inside the errno variable. When a system call is done, the following steps are followed:

A system call has a syscall number associated with it, which is a unique number used as a reference for the system call that cannot be changed (there is no possibility of implementing a system call). A symbolic constant for each system call number is available in the <sys/syscall.h> header file. To check the existence of a system call, sys_ni_syscall() is used, which returns the ENOSYS error for an invalid system call.

The Linux operating system is able to support a large variety of filesystem options. This is done due to the existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

to interact with a system, an interface should be provided to give the user space application the possibility of interacting with hardware and other processes.System calls. These are used as an interface between the hardware and the user space. They are also used to ensure stability, security, and abstraction, in general. These are common layers that constitute an entry point into the kernel alongside traps and exceptions, as described here:

The interaction with most of the system calls that are available inside the Linux system is done using the C library. They are able to define a number of arguments and return a value that reveals whether they were successful or not. A value of zero usually means that the execution ended with success, and in case errors appear, an error code will be available inside the errno variable. When a system call is done, the following steps are followed:

A system call has a syscall number associated with it, which is a unique number used as a reference for the system call that cannot be changed (there is no possibility of implementing a system call). A symbolic constant for each system call number is available in the <sys/syscall.h> header file. To check the existence of a system call, sys_ni_syscall() is used, which returns the ENOSYS error for an invalid system call.

The Linux operating system is able to support a large variety of filesystem options. This is done due to the existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

existence of Virtual File System (VFS), which is able to provide a common interface for a large number of filesystem types and handle the systems calls relevant to them.

The filesystem types supported by the VFS can be put in these three categories:

The virtual filesystem system call implementation is very well summarized in this image:

In the preceding image, it can be seen how easily the copy is handled from one filesystem type to another. It only uses the basic open(), close(), read(), write() functions available for all the other filesystem interaction. However, all of them implement the specific functionality underneath for the chosen filesystem. For example, the open() system calls sys_open()and it takes the same arguments as open() and returns the same result. The difference between sys_open() and open() is that sys_open() is a more permissive function.

All the other three system calls have corresponding sys_read(), sys_write(), and sys_close() functions that are called internally.

The first mechanism that will be discussed regarding bottom half interrupt handling is represented by softirqs. They are rarely used but can be found on the Linux kernel source code inside the kernel/softirq.c file. When it comes to implementation, they are statically allocated at the compile step. They are created when an entry is added in the include/linux/interrupt.h header file and the system information they provide is available inside the /proc/softirqs file. Although not used too often, they can be executed after exceptions, interrupts, system calls, and when the ksoftirkd daemon is run by the scheduler.

Next on the list are tasklets. Although they are built on top of softirqs, they are more commonly used for bottom half interrupt handling. Here are some of the reasons why this is done:

Tasklets have a struct tasklet_struct structure available. These are also available inside the include/linux/interrupt.h header file, and unlike softirqs, tasklets are non-reentrant.

Third on the list are work queues that represent a different form of doing the work allotted in comparison to previously presented mechanisms. The main differences are as follows:

Although they might have a latency that is slightly bigger the tasklets, the preceding qualities are really useful. The tasklets are built around the struct workqueue_struct structure, available inside the kernel/workqueue.c file.

The last and the newest addition to the bottom half mechanism options is represented by the kernel threads that are operated entirely in the kernel mode since they are created/destroyed by the kernel. They appeared during the 2.6.30 kernel release, and also have the same advantages as the work queues, along with some extra features, such as the possibility of having their own context. It is expected that eventually the kernel threads will replace the work queues and tasklets, since they are similar to the user space threads. A driver might want to request a threaded interrupt handler. All it needs to do in this case is to use request_threaded_irq() in a similar way to request_irq(). The request_threaded_irq() function offers the possibility of passing a handler and thread_fn to split the interrupt handling code into two parts. In addition to this, quick_check_handler is called to check if the interrupt was called from a device; if that is the case, it will need to call IRQ_WAKE_THREAD to wake up the handler thread and execute thread_fn.

The number of requests with which a kernel is dealing is likened to the number of requests a server has to receive. This situation can deal with race conditions, so a good synchronization method would be required. A number of policies are available for the way the kernel behaves by defining a kernel control path. Here is an example of a kernel control path:

The preceding image offers a clear picture as to why synchronization is necessary. For example, a race condition can appear when more than one kernel control path is interlinked. To protect these critical regions, a number of measures should be taken. Also, it should be taken into consideration that an interrupt handler cannot be interrupted and softirqs should not be interleaved.

A number of synchronization primitives have been born:

With the preceding methods, race condition situations try to be fixed. It is the job of the developer to identify and solve all the eventual synchronization problems that might appear.

by softirqs. They are rarely used but can be found on the Linux kernel source code inside the kernel/softirq.c file. When it comes to implementation, they are statically allocated at the compile step. They are created when an entry is added in the include/linux/interrupt.h header file and the system information they provide is available inside the /proc/softirqs file. Although not used too often, they can be executed after exceptions, interrupts, system calls, and when the ksoftirkd daemon is run by the scheduler.

Next on the list are tasklets. Although they are built on top of softirqs, they are more commonly used for bottom half interrupt handling. Here are some of the reasons why this is done:

Tasklets have a struct tasklet_struct structure available. These are also available inside the include/linux/interrupt.h header file, and unlike softirqs, tasklets are non-reentrant.

Third on the list are work queues that represent a different form of doing the work allotted in comparison to previously presented mechanisms. The main differences are as follows:

Although they might have a latency that is slightly bigger the tasklets, the preceding qualities are really useful. The tasklets are built around the struct workqueue_struct structure, available inside the kernel/workqueue.c file.

The last and the newest addition to the bottom half mechanism options is represented by the kernel threads that are operated entirely in the kernel mode since they are created/destroyed by the kernel. They appeared during the 2.6.30 kernel release, and also have the same advantages as the work queues, along with some extra features, such as the possibility of having their own context. It is expected that eventually the kernel threads will replace the work queues and tasklets, since they are similar to the user space threads. A driver might want to request a threaded interrupt handler. All it needs to do in this case is to use request_threaded_irq() in a similar way to request_irq(). The request_threaded_irq() function offers the possibility of passing a handler and thread_fn to split the interrupt handling code into two parts. In addition to this, quick_check_handler is called to check if the interrupt was called from a device; if that is the case, it will need to call IRQ_WAKE_THREAD to wake up the handler thread and execute thread_fn.

The number of requests with which a kernel is dealing is likened to the number of requests a server has to receive. This situation can deal with race conditions, so a good synchronization method would be required. A number of policies are available for the way the kernel behaves by defining a kernel control path. Here is an example of a kernel control path:

The preceding image offers a clear picture as to why synchronization is necessary. For example, a race condition can appear when more than one kernel control path is interlinked. To protect these critical regions, a number of measures should be taken. Also, it should be taken into consideration that an interrupt handler cannot be interrupted and softirqs should not be interleaved.

A number of synchronization primitives have been born:

With the preceding methods, race condition situations try to be fixed. It is the job of the developer to identify and solve all the eventual synchronization problems that might appear.

a server has to receive. This situation can deal with race conditions, so a good synchronization method would be required. A number of policies are available for the way the kernel behaves by defining a kernel control path. Here is an example of a kernel control path:

The preceding image offers a clear picture as to why synchronization is necessary. For example, a race condition can appear when more than one kernel control path is interlinked. To protect these critical regions, a number of measures should be taken. Also, it should be taken into consideration that an interrupt handler cannot be interrupted and softirqs should not be interleaved.

A number of synchronization primitives have been born:

With the preceding methods, race condition situations try to be fixed. It is the job of the developer to identify and solve all the eventual synchronization problems that might appear.

Linux kernel is a well known open source project. To make sure that developers know how to interact with it, information about how the git interaction is done with this project, and at the same time, some information about its development and release procedures will be presented. The project has evolved and its development processes and release procedures have evolved with it.

Before presenting the actual development process, a bit of history will be necessary. Until the 2.6 version of the Linux kernel project, one release was made every two or three years, and each of them was identified by even middle numbers, such as 1.0.x, 2.0.x, and 2.6.x. The development branches were instead defined using even numbers, such as 1.1.x, 2.1.x, and 2.5.x, and they were used to integrate various features and functionalities until a major release was prepared and ready to be shipped. All the minor releases had names, such as 2.6.32 and 2.2.23, and they were released between major release cycles.

This way of working was kept up until the 2.6.0 version when a large number of features were added inside the kernel during every minor release, and all of them were very well put together as to not cause the need for the branching out of a new development branch. This implied a faster pace of release with more features available. So, the following changes have appeared since the release of the 2.6.14 kernel:

This process worked great but the only problem was that the bug fixes were only released for the latest stable versions of the Linux kernel. People needed long term support versions and security updates for their older versions, general information about these versions that were long time supported, and so on.

This process changed in time and in July 2011, the 3.0 Linux kernel version appeared. It appeared with a couple of small changes designed to change the way the interaction was to be done to solve the previously mentioned requests. The changes were made to the numbering scheme, as follows:

Although it only removed one digit from the numbering scheme, this change was necessary because it marked the 20th anniversary of the Linux kernel.

Since a great number of patches and features are included in the Linux kernel everyday, it becomes difficult to keep track of all the changes, and the bigger picture in general. This changed over time because sites, such as http://kernelnewbies.org/LinuxChanges and http://lwn.net/, appeared to help developers keep in touch with the world of Linux kernel.

Besides these links, the git versioning control system can offer much needed information. Of course, this requires the existence of Linux kernel source clones to be available on the workstation. Some of the commands that offer a great deal of information are:

Of course, this is just a small list with helpful commands. All the other commands are available at http://git-scm.com/docs/.

The Linux kernel offers support for a large variety of CPU architectures. Each architecture and individual board have their own maintainers, and this information is available inside the MAINTAINERS file. Also, the difference between board porting is mostly given by the architecture, PowerPC being very different from ARM or x86. Since the development board that this book focuses on is an Atmel with an ARM Cortex-A5 core, this section will try to focus on ARM architecture.

The main focus in our case is the arch/arm directory, which contains sub directories such as, boot, common, configs, crypto, firmware, kernel, kvm, lib, mm, net, nwfpe, oprofile, tools, vfp, and xen. It also contains an important number of directories that are specific for different CPU families, such as the mach-* directories or the plat-* directories. The first mach-* category contains support for the CPU and several boards that use that CPU, and the second plat-* category contains platform-specific code. One example is plat-omap, which contains common code for both mach-omap1 and mach-omap2.

The development for the ARM architecture has suffered a great change since 2011. If until then ARM did not use a device tree, it was because it needed to keep a large portion of the code inside the mach-* specific directory, and for each board that had support inside the Linux kernel, a unique machine ID was associated and a machine structure was associates with each board that contained specific information and a set of callbacks. The boot loader passed this machine ID to a specific ARM registry and in this way, the kernel knew the board.

The increase in popularity of the ARM architecture came with the refactoring of the work and the introduction of the device tree that dramatically reduced the amount of code available inside the mach-* directories. If the SoC is supported by the Linux kernel, then adding support for a board is as simple as defining a device tree in the /arch/arm/boot/dts directory with an appropriate name. For example, for <soc-name>-<board-name>.d, include the relevant dtsi files if necessary. Make sure that you build the device tree blob (DTB) by including the device tree into arch/arm/boot/dts/Makefile and add the missing device drivers for board.

In the eventuality that the board does not have support inside the Linux kernel, the appropriate additions would be required inside the mach-* directory. Inside each mach-* directory, there are three types of files available:

For a given board, the proper configuration should be made first inside the arch/arm/mach-*/Kconfig file; for this, the machine ID should be identified for the board CPU. After the configuration is done, the compilation can begin, so for this, arch/arm/mach-*/Makefile should also be updated with the required files to ensure board support. Another step is represented by the machine structure that defines the board and the machine type number that needs to be defined in the board-<machine>.c file.

The machine structure uses two macros: MACHINE_START and MACHINE_END. Both are defined inside arch/arm/include/asm/march/arch.h and are used to define the machine_desc structure. The machine type number is available inside the arch/arm/tools/mach_types file. This file is used to generate the include/asm-arm/mach-types.h file for the board.

When the boot process starts in the first case, only the dtb is necessary to pass to the boot loader and loaded to initialize the Linux kernel, while in the second case, the machine type number needs to be loaded in the R1 register. In the early boot process, __lookup_machine_type looks for the machine_desc structure and loads it for the initialization of the board.

After this information has been presented to you, and if you are eager to contribute to the Linux kernel, then this section should be read next. If you want to really contribute to the Linux kernel project, then a few steps should be performed before starting this work. This is mostly related to documentation and investigation of the subject. No one wants to send a duplicate patch or replicate the work of someone else in vain, so a search on the Internet on the topic of your interest could save a lot of time. Other useful advice is that after you've familiarized yourself with the subject, avoid sending a workaround. Try to reach the problem and offer a solution. If not, report the problem and describe it thoroughly. If the solution is found, then make both the problem and solution available in the patch.

One of the most valuable things in the open source community is the help you can get from others. Share your question and issues, but do not forget to mention the solution also. Ask the questions in appropriate mailing lists and try to avoid the maintainers, if possible. They are usually very busy and have hundreds and thousands of e-mails to read and reply. Before asking for help, try to research the question you want to raise, it will help both when formulating it but also it could offer an answer. Use IRC, if available, for smaller questions and lastly, but most importantly, try to not overdo it.

When you are preparing for a patch, make sure that it is done on the corresponding branch, and also that you read the Documentation/BUG-HUNTING file first. Identify bug reports, if any, and make sure you link your patch to them. Do not hesitate to read the Documentation/SubmittingPatches guidelines before sending. Also, do not send your changes before testing them properly. Always sign your patches and make the first description line as suggestive as possible. When sending the patches, find appropriate mailing lists and maintainers and wait for the replies. Solve comments and resubmit them if this is needed, until the patch is considered acceptable.

a well known open source project. To make sure that developers know how to interact with it, information about how the git interaction is done with this project, and at the same time, some information about its development and release procedures will be presented. The project has evolved and its development processes and release procedures have evolved with it.

Before presenting the actual development process, a bit of history will be necessary. Until the 2.6 version of the Linux kernel project, one release was made every two or three years, and each of them was identified by even middle numbers, such as 1.0.x, 2.0.x, and 2.6.x. The development branches were instead defined using even numbers, such as 1.1.x, 2.1.x, and 2.5.x, and they were used to integrate various features and functionalities until a major release was prepared and ready to be shipped. All the minor releases had names, such as 2.6.32 and 2.2.23, and they were released between major release cycles.

This way of working was kept up until the 2.6.0 version when a large number of features were added inside the kernel during every minor release, and all of them were very well put together as to not cause the need for the branching out of a new development branch. This implied a faster pace of release with more features available. So, the following changes have appeared since the release of the 2.6.14 kernel:

This process worked great but the only problem was that the bug fixes were only released for the latest stable versions of the Linux kernel. People needed long term support versions and security updates for their older versions, general information about these versions that were long time supported, and so on.

This process changed in time and in July 2011, the 3.0 Linux kernel version appeared. It appeared with a couple of small changes designed to change the way the interaction was to be done to solve the previously mentioned requests. The changes were made to the numbering scheme, as follows:

Although it only removed one digit from the numbering scheme, this change was necessary because it marked the 20th anniversary of the Linux kernel.

Since a great number of patches and features are included in the Linux kernel everyday, it becomes difficult to keep track of all the changes, and the bigger picture in general. This changed over time because sites, such as http://kernelnewbies.org/LinuxChanges and http://lwn.net/, appeared to help developers keep in touch with the world of Linux kernel.

Besides these links, the git versioning control system can offer much needed information. Of course, this requires the existence of Linux kernel source clones to be available on the workstation. Some of the commands that offer a great deal of information are:

Of course, this is just a small list with helpful commands. All the other commands are available at http://git-scm.com/docs/.

The Linux kernel offers support for a large variety of CPU architectures. Each architecture and individual board have their own maintainers, and this information is available inside the MAINTAINERS file. Also, the difference between board porting is mostly given by the architecture, PowerPC being very different from ARM or x86. Since the development board that this book focuses on is an Atmel with an ARM Cortex-A5 core, this section will try to focus on ARM architecture.

The main focus in our case is the arch/arm directory, which contains sub directories such as, boot, common, configs, crypto, firmware, kernel, kvm, lib, mm, net, nwfpe, oprofile, tools, vfp, and xen. It also contains an important number of directories that are specific for different CPU families, such as the mach-* directories or the plat-* directories. The first mach-* category contains support for the CPU and several boards that use that CPU, and the second plat-* category contains platform-specific code. One example is plat-omap, which contains common code for both mach-omap1 and mach-omap2.

The development for the ARM architecture has suffered a great change since 2011. If until then ARM did not use a device tree, it was because it needed to keep a large portion of the code inside the mach-* specific directory, and for each board that had support inside the Linux kernel, a unique machine ID was associated and a machine structure was associates with each board that contained specific information and a set of callbacks. The boot loader passed this machine ID to a specific ARM registry and in this way, the kernel knew the board.

The increase in popularity of the ARM architecture came with the refactoring of the work and the introduction of the device tree that dramatically reduced the amount of code available inside the mach-* directories. If the SoC is supported by the Linux kernel, then adding support for a board is as simple as defining a device tree in the /arch/arm/boot/dts directory with an appropriate name. For example, for <soc-name>-<board-name>.d, include the relevant dtsi files if necessary. Make sure that you build the device tree blob (DTB) by including the device tree into arch/arm/boot/dts/Makefile and add the missing device drivers for board.

In the eventuality that the board does not have support inside the Linux kernel, the appropriate additions would be required inside the mach-* directory. Inside each mach-* directory, there are three types of files available:

For a given board, the proper configuration should be made first inside the arch/arm/mach-*/Kconfig file; for this, the machine ID should be identified for the board CPU. After the configuration is done, the compilation can begin, so for this, arch/arm/mach-*/Makefile should also be updated with the required files to ensure board support. Another step is represented by the machine structure that defines the board and the machine type number that needs to be defined in the board-<machine>.c file.

The machine structure uses two macros: MACHINE_START and MACHINE_END. Both are defined inside arch/arm/include/asm/march/arch.h and are used to define the machine_desc structure. The machine type number is available inside the arch/arm/tools/mach_types file. This file is used to generate the include/asm-arm/mach-types.h file for the board.

When the boot process starts in the first case, only the dtb is necessary to pass to the boot loader and loaded to initialize the Linux kernel, while in the second case, the machine type number needs to be loaded in the R1 register. In the early boot process, __lookup_machine_type looks for the machine_desc structure and loads it for the initialization of the board.

After this information has been presented to you, and if you are eager to contribute to the Linux kernel, then this section should be read next. If you want to really contribute to the Linux kernel project, then a few steps should be performed before starting this work. This is mostly related to documentation and investigation of the subject. No one wants to send a duplicate patch or replicate the work of someone else in vain, so a search on the Internet on the topic of your interest could save a lot of time. Other useful advice is that after you've familiarized yourself with the subject, avoid sending a workaround. Try to reach the problem and offer a solution. If not, report the problem and describe it thoroughly. If the solution is found, then make both the problem and solution available in the patch.

One of the most valuable things in the open source community is the help you can get from others. Share your question and issues, but do not forget to mention the solution also. Ask the questions in appropriate mailing lists and try to avoid the maintainers, if possible. They are usually very busy and have hundreds and thousands of e-mails to read and reply. Before asking for help, try to research the question you want to raise, it will help both when formulating it but also it could offer an answer. Use IRC, if available, for smaller questions and lastly, but most importantly, try to not overdo it.

When you are preparing for a patch, make sure that it is done on the corresponding branch, and also that you read the Documentation/BUG-HUNTING file first. Identify bug reports, if any, and make sure you link your patch to them. Do not hesitate to read the Documentation/SubmittingPatches guidelines before sending. Also, do not send your changes before testing them properly. Always sign your patches and make the first description line as suggestive as possible. When sending the patches, find appropriate mailing lists and maintainers and wait for the replies. Solve comments and resubmit them if this is needed, until the patch is considered acceptable.

by the architecture, PowerPC being very different from ARM or x86. Since the development board that this book focuses on is an Atmel with an ARM Cortex-A5 core, this section will try to focus on ARM architecture.

The main focus in our case is the arch/arm directory, which contains sub directories such as, boot, common, configs, crypto, firmware, kernel, kvm, lib, mm, net, nwfpe, oprofile, tools, vfp, and xen. It also contains an important number of directories that are specific for different CPU families, such as the mach-* directories or the plat-* directories. The first mach-* category contains support for the CPU and several boards that use that CPU, and the second plat-* category contains platform-specific code. One example is plat-omap, which contains common code for both mach-omap1 and mach-omap2.

The development for the ARM architecture has suffered a great change since 2011. If until then ARM did not use a device tree, it was because it needed to keep a large portion of the code inside the mach-* specific directory, and for each board that had support inside the Linux kernel, a unique machine ID was associated and a machine structure was associates with each board that contained specific information and a set of callbacks. The boot loader passed this machine ID to a specific ARM registry and in this way, the kernel knew the board.

The increase in popularity of the ARM architecture came with the refactoring of the work and the introduction of the device tree that dramatically reduced the amount of code available inside the mach-* directories. If the SoC is supported by the Linux kernel, then adding support for a board is as simple as defining a device tree in the /arch/arm/boot/dts directory with an appropriate name. For example, for <soc-name>-<board-name>.d, include the relevant dtsi files if necessary. Make sure that you build the device tree blob (DTB) by including the device tree into arch/arm/boot/dts/Makefile and add the missing device drivers for board.

In the eventuality that the board does not have support inside the Linux kernel, the appropriate additions would be required inside the mach-* directory. Inside each mach-* directory, there are three types of files available:

For a given board, the proper configuration should be made first inside the arch/arm/mach-*/Kconfig file; for this, the machine ID should be identified for the board CPU. After the configuration is done, the compilation can begin, so for this, arch/arm/mach-*/Makefile should also be updated with the required files to ensure board support. Another step is represented by the machine structure that defines the board and the machine type number that needs to be defined in the board-<machine>.c file.

The machine structure uses two macros: MACHINE_START and MACHINE_END. Both are defined inside arch/arm/include/asm/march/arch.h and are used to define the machine_desc structure. The machine type number is available inside the arch/arm/tools/mach_types file. This file is used to generate the include/asm-arm/mach-types.h file for the board.

When the boot process starts in the first case, only the dtb is necessary to pass to the boot loader and loaded to initialize the Linux kernel, while in the second case, the machine type number needs to be loaded in the R1 register. In the early boot process, __lookup_machine_type looks for the machine_desc structure and loads it for the initialization of the board.

After this information has been presented to you, and if you are eager to contribute to the Linux kernel, then this section should be read next. If you want to really contribute to the Linux kernel project, then a few steps should be performed before starting this work. This is mostly related to documentation and investigation of the subject. No one wants to send a duplicate patch or replicate the work of someone else in vain, so a search on the Internet on the topic of your interest could save a lot of time. Other useful advice is that after you've familiarized yourself with the subject, avoid sending a workaround. Try to reach the problem and offer a solution. If not, report the problem and describe it thoroughly. If the solution is found, then make both the problem and solution available in the patch.

One of the most valuable things in the open source community is the help you can get from others. Share your question and issues, but do not forget to mention the solution also. Ask the questions in appropriate mailing lists and try to avoid the maintainers, if possible. They are usually very busy and have hundreds and thousands of e-mails to read and reply. Before asking for help, try to research the question you want to raise, it will help both when formulating it but also it could offer an answer. Use IRC, if available, for smaller questions and lastly, but most importantly, try to not overdo it.

When you are preparing for a patch, make sure that it is done on the corresponding branch, and also that you read the Documentation/BUG-HUNTING file first. Identify bug reports, if any, and make sure you link your patch to them. Do not hesitate to read the Documentation/SubmittingPatches guidelines before sending. Also, do not send your changes before testing them properly. Always sign your patches and make the first description line as suggestive as possible. When sending the patches, find appropriate mailing lists and maintainers and wait for the replies. Solve comments and resubmit them if this is needed, until the patch is considered acceptable.

has been presented to you, and if you are eager to contribute to the Linux kernel, then this section should be read next. If you want to really contribute to the Linux kernel project, then a few steps should be performed before starting this work. This is mostly related to documentation and investigation of the subject. No one wants to send a duplicate patch or replicate the work of someone else in vain, so a search on the Internet on the topic of your interest could save a lot of time. Other useful advice is that after you've familiarized yourself with the subject, avoid sending a workaround. Try to reach the problem and offer a solution. If not, report the problem and describe it thoroughly. If the solution is found, then make both the problem and solution available in the patch.

One of the most valuable things in the open source community is the help you can get from others. Share your question and issues, but do not forget to mention the solution also. Ask the questions in appropriate mailing lists and try to avoid the maintainers, if possible. They are usually very busy and have hundreds and thousands of e-mails to read and reply. Before asking for help, try to research the question you want to raise, it will help both when formulating it but also it could offer an answer. Use IRC, if available, for smaller questions and lastly, but most importantly, try to not overdo it.

When you are preparing for a patch, make sure that it is done on the corresponding branch, and also that you read the Documentation/BUG-HUNTING file first. Identify bug reports, if any, and make sure you link your patch to them. Do not hesitate to read the Documentation/SubmittingPatches guidelines before sending. Also, do not send your changes before testing them properly. Always sign your patches and make the first description line as suggestive as possible. When sending the patches, find appropriate mailing lists and maintainers and wait for the replies. Solve comments and resubmit them if this is needed, until the patch is considered acceptable.

Before building a Linux kernel image, the proper configuration needs to be done. This is hard, taking into consideration that we have access to hundreds and thousands of components, such as drivers, filesystems, and other items. A selection process is done inside the configuration stage, and this is possible with the help of dependency definitions. The user has the chance to use and define a number of options that are enabled in order to define the components that will be used to build a Linux kernel image for a specific board.

All the configurations specific for a supported board are located inside a configuration file, simply named .config, and it is situated on the same level as the previously presented files and directory locations. Their form is usually represented as configuration_key=value. There are, of course, dependencies between these configurations, so they are defined inside the Kconfig files.