Instant Optimizing Embedded Systems Using BusyBox

BusyBox was designed for Linux in early 1996, originally as a Debian GNU/Linux installer and rescue system. With about 18 years of development, the latest 1.21.0 (at the time of this writing) release has more than 380 applets and about 100 projects claimed using it. The projects embrace Linux, Android, FreeBSD, and others.

It integrates a set of stripped-down Unix tools, including simple ones, such as echo and ls, as well as the larger ones, such as grep, awk, find, mount, and modprobe and other functional utilities; for example, ifconfig, route, dmesg, tar, wget, and even a tiny shell interpreter called ash.

It is modular, and can be customized to meet different requirements. To discuss and demonstrate its configuration system, first download its source code. It is maintained at http://git.busybox.net/busybox/ and released with .tar packages at http://busybox.net/downloads/. If you want a stable release, download the .tar packages or clone the Git repository with the latest (but perhaps not stable) features. At the time of this writing, we used the stable version busybox-1.21.0.tar.bz2; download it and decompress it.

$ wget http://busybox.net/downloads/busybox-1.20.0.tar.bz2
$ tar jxvf busybox-1.20.0.tar.bz2
$ cd busybox-1.20.0/

To simplify the command-line output, all command-line prompts in the desktop development system is prefixed with the $ symbol; ones that need root permission will follow a sudo command.

Before configuring, make sure the ncurses library required by the graphic configurator is installed on the development system.

$ sudo apt-get install libncurses5-dev

The friendliest way to configure BusyBox will be using make menuconfig. Issue the command to launch a graphic interface, configure the features, and exit to save them.

After configuring, Press Esc + Esc to exit, and the configuration will be saved to a file named .config.

With the previous configuration, we should be able to start the compiling process in the next recipe, but we still need to learn more details about the principle and the other methods of configuration.

BusyBox can be flexibly customized; the main configurable features include Busybox Settings and Applets. The former allows us to configure build/compiling and installation features. The latter allows us to configure all of the built-in applets and their features. Both of them will be introduced in the following recipes.

To configure them in fine detail, the Config.src files (for example, networking/Config.src) provide configure options for the features, and every option should have a default setting.

Hotkeys are provided to enable, disable, or set the option; the option's value will be converted to symbols prefixed with CONFIG_ and saved into the .config file, and meanwhile some C headers will be generated.

Besides graphic configuration with make menuconfig, some other configuration techniques you might want to run with the BusyBox make tool include:

Before compiling a software, we must get a compiler and the corresponding libraries, a build host, and a target platform. The build host is the one running the compiler. The target platform is the one running the target binary built by the compiler from the software source code. Here, our desktop development system is our build host, and an ARM Android system will be used as our target platform.

To compile BusyBox on our desktop development system, we need a local compiler. Generally, most Linux operating systems ship with a working C compiler, and it's usually GCC (the GNU Compiler Collection is by far the most widely used C compiler on Linux, and can be downloaded from http://gcc.gnu.org/) and the corresponding Glibc (the GNU C Library by far is the most widely used C library on Linux and can be downloaded from http://www.gnu.org/software/libc/). If it isn't present, get one from our system's package manager for Ubuntu.

$ sudo apt-get install gcc g++ make build-essential
$ sudo apt-get install ia32-libs ia32-libs-multiarch

To compile BusyBox on our desktop development system for a different target architecture (for example, an ARM Android system), we need a cross-compiler.

ARM, as the most popular architecture on embedded system, is being used by more and more portable devices, such as the embedded Raspberry Pi board and the Android mobile phone or tablet, so we will concern ourselves with this alone.

The gcc-arm-linux-gnueabi cross-compiler can be installed directly on Ubuntu with the following command:

$ sudo apt-get install gcc-arm-linux-gnueabi

On other Linux distributions, Google's official NDK is a good choice if you want to share Android's Bionic C library, but since Bionic C library lacks of many POSIX C header files, and because we want to get the most out of BusyBox applets building, the prebuilt version of Linaro GCC with Glibc is preferable. We can download it from http://www.linaro.org/downloads/; for example, http://releases.linaro.org/13.04/components/toolchain/binaries/gcc-linaro-aarch64-none-elf-4.7-2013.04-20130415_linux.tar.bz2.

Let's compile BusyBox with the default configuration for our desktop development system as a quick demonstration of the compiling procedure and cross-compile it for the ARM platform as a real embedded system building example.

Using BusyBox on an ARM platform needs an ARM device. This will be introduced in the Creating a virtual Android device (Simple) and Playing BusyBox on a virtual Android device (Intermediate) recipes.

The configuration not only generates a .config file with enabled features, but also produces a C header file, include/autoconf.h, that enables or disables the features with C macros. Based on these macros, the compiler determines which C context should be built in the last binary.

By default, the binary is dynamically linked. To avoid the installation of shared libraries, to reduce the whole system size, and to reduce the time cost of runtime linking, static linking is often used for embedded system compiling.

To enable static linking, configure BusyBox as follows:

   Busybox Settings --->
      Build Options --->
          [*] Build BusyBox as a static binary (no shared libs)

We get the following error when using a new Glibc to compile BusyBox with static linking:

inetd.c:(.text.prepare_socket_fd+0x7e): undefined reference to `bindresvport'

We need to disable CONFIG_FEATURE_INETD_RPC, as follows, to avoid it:

   Networking Utilities  --->
       [*] inetd
           [ ]   Support RPC services

Then, recompile it as follows:

$ make

Because it is statically linked, the required libraries will be linked into the last BusyBox binary. To run it on the target system, you only need to install the BusyBox binary (no shared libraries need to be installed).

The installation of BusyBox will be discussed in next recipe and the Creating a virtual Android device (Simple) recipe.

Just as a Swiss Army knife has multiple blades, a single BusyBox executable can be used to invoke many applets. BusyBox applets can be invoked in a number of different ways, explained as follows:

The installation of BusyBox means creating soft links for all of its built-in applets.

Busybox can be installed at runtime or at the earlier compiling stage.

With --install, hard links are created. To create soft links (symbolic links), the -s option should be appended. Soft links must be used if the target installation directory (for example, /bin/ provided by the Android ramdisk filesystem) is not in the same device as the directory (for example, /data/ provided by another filesystem of Android) storing the BusyBox binary; if they are not, we will get the following failure errors:

To use the -s option, it should be enabled with the following configuration:

Busybox Settings  --->
   General Configuration  --->
      [*] Support --install [-s] to install applet links at runtime

Apart from the --install option, the make install command will install the bin/busybox binary in the target directory specified by CONFIG_PREFIX, and it classifies the applets and puts them into different directories based on the applets' setting in include/applets.src.h; for example, the following line specifies that ls should be installed to /bin:

IF_LS(APPLET_NOEXEC(ls, ls, BB_DIR_BIN, BB_SUID_DROP, ls))

To demonstrate quickly how to experiment with the previous filesystem, we use the chroot command, which allows us to switch to a new root filesystem. In the Building BusyBox-based embedded system (Intermediate) recipe, we'll talk about how to fulfill such filesystems and make it bootable on virtual Android devices.

If BusyBox is dynamically linked, the shared libraries should be installed manually.

Use the shared libraries for our desktop development environment listed by ldd in the Configuring BusyBox recipe as an example.

$ ldd busybox
   linux-vdso.so.1 =>  (0x00007fff9edff000)
   libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1cf2cb4000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1cf28f5000)
   /lib64/ld-linux-x86-64.so.2 (0x00007f1cf2fcf000)

Except the virtual linux-vdso.so.1 (or linux-gate.so.1), we should create the same directory architecture:

$ cd ~/tools/busybox/ramdisk
$ mkdir lib lib64
$ mkdir lib/x86_64-linux-gnu/

The directory architecture looks as follows:

lib

└── x86_64-linux-gnu

├── ld-2.15.so

├── libc-2.15.so

├── libc.so.6 -> libc-2.15.so

├── libm-2.15.so

└── libm.so.6 -> libm-2.15.so

lib64

└── ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.15.so

Then, copy libraries under /lib and /lib64 of our desktop development environment to the directories under ~/tools/busybox/ramdisk and create the corresponding soft links as follows:

$ cp /lib/x86_64-linux-gnu/libc-2.15.so lib/x86_64-linux-gnu/
$ cp /lib/x86_64-linux-gnu/libm-2.15.so lib/x86_64-linux-gnu/
$ cp /lib/x86_64-linux-gnu/ld-2.15.so lib/x86_64-linux-gnu/
$ ln -s lib/x86_64-linux-gnu/ld-2.15.so lib64/ld-linux-x86-64.so.2
$ cd lib/x86_64-linux-gnu/
$ ln -s libc-2.15.so libc.so.6 
$ ln -s libm-2.15.so libm.so.6

After the installation of the shared libraries, we can start the BusyBox ash shell in a new root filesystem with chroot. chroot needs root permission with sudo.

$ sudo SHELL=/bin/ash chroot ~/tools/busybox/ramdisk/

The SHELL environment variable specifies the default shell interpreter for chroot; in our example, we use BusyBox ash.

With chroot, we are able to emulate the target embedded scene.

The installation of the shared libraries for ARM BusyBox is similar, so we will not give an example here. If BusyBox is statically linked, no need to install the extra libraries; we can simply install them and use chroot to perform the experiment.

To create a virtual Android device, the Android emulator and the related Android SDK Manager should be installed first. ADT (Android Development Tools) provides these tools; we can get it from http://developer.android.com/sdk/index.html.

Download and use adt-bundle-linux-x86_64-20130514.zip for our x86-64 desktop development system. For other architectures, please download the proper version.

$ wget http://dl.google.com/android/adt/adt-bundle-linux-x86_64-20130522.zip
$ sudo unzip adt-bundle-linux-x86_64-20130514.zip -d /usr/local/
$ sudo mv /usr/local/adt-bundle-linux-x86_64-20130514 /usr/local/adt-bundle-linux

The tools required are installed at /usr/local/adt-bundle-linux/sdk/. Set the PATH variable in the shell startup script as before to set them permanently, so that they are easy to use:

$ echo "export SDK_ROOT=/usr/local/adt-bundle-linux/sdk/" >> ~/.bashrc
$ echo "export PATH=\$SDK_ROOT/tools/:\$PATH" >> ~/.bashrc
$ echo "export PATH=\$SDK_ROOT/platform-tools/:\$PATH" >> ~/.bashrc
$ source ~/.bashrc

With these settings, the tools like android, emulator, and adb are installed.

We use a virtual Nexus 7 device as our emulated experiment platform; its CPU is ARM with armeabi-v7a ABI (Application Binary Interface), and runs Android 4.2 with API (Application Program Interface) level 17.

-scale auto lets the emulator scale the windows to the real size of the virtual device.

Google Android emulator, being based on QEMU, is an open source and free Bare Metal emulator. That is, it simulates the actual machine instruction set of the target system. This makes for a very accurate simulation, but it may be slow.

In reality, emulator is a wrapper and enhancement of QEMU. It adds the Goldfish platform emulation and adds some Android-specific features. It can be used to do most of the Android system development.

Just as the real Android smartphone or tablet, the virtual Android device just created supplies with some development features, such as Android Debug Bridge (ADB) and the virtual serial-port-based shell environment.

At this stage, we aren't constructing a full bootable filesystem from scratch, but only using the BusyBox applets. So, for simplicity and ease of testing, we will install our compiled BusyBox binary and scripts into an existing Android ramdisk image.

At first, get a copy of the existing ramdisk.img; its path can be found at ~/.android/avd/busybox-avd.avd/hardware-qemu.ini; for example, adt-bundle-linux-x86_64/sdk/system-images/android-17/armeabi-v7a/ramdisk.img.

Three examples will be shown in this section to demonstrate the usage of the BusyBox applets on a virtual Android device.

BusyBox applets cover the standard coreutils, console tools, e2fsprogs, procps, SELinux, editors, modutils, debianutils, login-utils, networking, sysklogd, util-linux, findutils, mailutils, printutils, and even a tiny shell interpreter. This recipe shows how to install some of them at runtime on an existing Android system and demonstrates the working of these three applets: ash, telnetd, and httpd.

To add more function for an embedded system, the other applets may also need to be configured via the configuration utility and used based on their full usage information with the --help command.

In order to start the previously mentioned service during boot, we can add httpd and telnetd in the previous /busybox-console.sh script.

# Start httpd service
httpd -h /data/www -p 8080
# Start telnetd service
telnetd -f /data/telnetd.issue -p 3333 -l /bin/ash

To access them, we must also forward the remote ports to the development system as follows:

$ adb forward tcp:3333 tcp:3333
$ adb forward tcp:8080 tcp:8080

This recipe will use BusyBox to build our own filesystem and compress it to a gzipped cpio package, just like ramdisk.img, which we used in the previous recipe.

To boot such a new system, a new Linux kernel image should be compiled from Google Android-specific Linux kernel source code. Clone the source code for preparation.

$ cd ~/tools/busybox/
$ git clone https://android.googlesource.com/kernel/goldfish.git

The source code is specifically designed for Android goldfish hardware; the Android emulator won't run on other hardware.

To cross-compile an Android Linux kernel, the cross-compiler arm-linux-gnueabi-gcc should have been installed when going through the Compiling BusyBox recipe.

Based on the Installing BusyBox recipe, let's build our own filesystem: ramdisk.img with BusyBox from scratch, cross-compile our own Linux kernel image, and then boot them on the Android emulator:

The ramdisk.img is an archive of format gzipped cpio, which is extracted as the default root filesystem when the kernel boots up. It generally contains a file named init, which will run after the kernel boots up. This init command parses the initialization configuration, /etc/inittab, and eventually starts the console based on the ash shell, and provides the applets' running environment.

In real embedded development with the minimal filesystem built with BusyBox and their built-in applets, we are able to verify the function and robustness of their device drivers through their exported /dev, /sys, and /proc interfaces.

With the help of the ash applet, shell scripts can be written to perform test automation to enhance the stability of an embedded system. With the help of the built-in PowerTop, bootchartd, top, iostat, devmem, and some other external utilities, the other aspects of an embedded system can be optimized to provide better user experience. The test automation and system optimization will be discussed in the coming recipes.

In order to extend a fork of BusyBox by adding functionality from an existing utility, just compile and install the new utility as a separate binary, which will be explained later. But to distribute one binary exactly to our embedded system and get a smaller-sized system, integrating new applets into BusyBox is a direct method.

To do so, use the small sstrip program as an example. sstrip is a small utility that removes the contents at the end of an ELF file that are not part of the program's memory image. ELF (http://en.wikipedia.org/wiki/Executable_and_Linkable_Format) is the common standard file format for Linux executables.

Download it with SVN or download it from this link: https://dev.openwrt.org/browser/trunk/tools/sstrip/src/sstrip.c. In our example, use svn to check it out.

$ cd ~/tools/
$ svn co svn://dev.openwrt.org/openwrt/trunk/tools/sstrip

To add the sstrip program into BusyBox, follow these steps:

BusyBox is modular and highly configurable. As seen previously, a new applet can be easily integrated, except renaming the main entry. The configure, build, and header files are added or modified for configuration, compiling, and installation respectively.

Since the main() entry is used by the BusyBox binary itself, the other applets' entries should be renamed with their own names.

The change in miscutils/Config.src adds a configure symbol, CONFIG_SSTRIP, with respective help information.

The modification in miscutils/Kbuild.src allows sstrip to be compiled while the CONFIG_SSTRIP symbol is enabled.

The addition in include/applets.src.h declares the prototypes of the new applet, sstrip, and the respective usage, main entry, and links.

Lightweight tools like sstrip can be simply integrated into the BusyBox package. But for getting more functional utilities for our embedded systems, such methods don't work. However, we do have two other methods.

In reality, with Bash (the popular shell-scripting environment) and with C (the popular native programming environment), we're able to re-use lots of existing shell scripts, C software, or write new ones for our BusyBox-based embedded system to enhance its functionality to a very flexible extent.

For most embedded systems (particularly consumer electronics like smartphone systems), due to the limitation of product cost, size of the hardware model, and rich-function requirements, system image size should be limited to fit the size of the disk and memory and reserve more free storage for end users.

The software of an embedded Linux system mainly includes bootloader, kernel, ramdisk, system, and applications. Use the embedded filesystem we just built with BusyBox in the Building BusyBox-based embedded system (Intermediate) recipe as an example. To reduce its size, we should:

The main storage consumers of an embedded Linux system are executables. A standard Linux executable is in an ELF format. Besides an ELF header, a program header table, and a section header table, it consists of the text segment, data segment (initialized), and BSS segment (uninitialized).

To optimize size means to remove unnecessary executables or to reduce the size of the previously mentioned segments of the executables by sharing common functions or data, or even via sharing a single ELF header.

BusyBox is highly modular and configurable, and therefore allows to choose only the necessary applets based on the real system requirements. And as a single binary, all of the built-in applets share one ELF header and share all of the common functions. Besides those, compiler size optimization options like -Os and --gc-sections are also well supported.

Also, the last binary can be stripped with strip and sstrip, and the generated filesystem image can be compressed.

We have discussed size optimization of ramdisk built with BusyBox. For the other parts of an embedded Linux system and the other methods, http://www.elinux.org/System_Size is a very good reference.

For the kernel parts, we can get more useful information from a project proposed by one of the authors of this book, at http://elinux.org/Work_on_Tiny_Linux_Kernel.

The whole power-cost optimization topic is very systematic; it is related to hardware system design and the power management policies of the software system.

The hardware elements that need to be considered in the power-cost balance are CPU, memory and external devices, and system components, such as clock, power domain, and regulator.

The corresponding software management topics include CPU idle, CPU frequency, CPU hotplug, memory hotplug, memory frequency, bus frequency, runtime power management, clock management, power domain management, and system suspend.

It's not possible to explain all of them, but we'll introduce an applet that can give us power optimization suggestions, and can measure top power-cost events.

PowerTOP (https://01.org/powertop/) is designed to measure, explain, and minimize a computer's electrical power consumption; it not only works on x86 systems, but also supports ARM, AMD, and other kinds of architectures.

The ideal working status that is power cost friendly is putting the system into an idle state as far as possible or eventually shutting down all of the devices.

To reduce the power cost of a single device, we should put it into a deeper idle state with less voltage and working frequency (if supported) or power it off eventually if nobody uses it. On the contrary, if some events wake up the device from the idle state to the active state frequently, the device will cost more power; such events are often used as the most important indicators to reduce power cost.

We learned about the switching of a system or device state from idle to active and introduced the PowerTOP tool, which can monitor this switching.

In Android, the command dumpsys alarm can track the alarms that wake up the system from a suspend state (a deeper idle state).

But even if we fix all potential wakeup causes, the power cost may still need to be optimized in other aspects. This means even in an idle state, there will be power leaking, which should be fixed up by the GPIO and register settings based on the datasheets of specific chips.

In an active state, power and performance should be balanced carefully with DVFS, clock gating, regulator gating, or power QOS policies.

The system or device should enter into an idle state from an active state timely if there is no activity.

In Android, if there is an activity request, wake_lock is held. The system will not switch from an active state to an idle state until the lock is released. To reduce power cost, the lock-holding time should be as short as possible. To track the holding of such wake locks, the /proc/wakelocks or /sys/kernel/debug/wakeup_sources interface helps, and the dumpsys power command may give more related information.

The following are the basic methods to speed system booting:

Bootchart (http://www.bootchart.org/) is for performance analysis and visualization of the Linux boot process; it can be used for boot speedup optimization.

Android does include the init method's built-in bootchartd support, but requires the init application to be rebuilt; for more details, visit http://elinux.org/Using_Bootchart_on_Android.

In our example, we introduce the BusyBox built-in bootchartd support to measure the Android system boot procedure.

At last, it's time to analyze the graph.

A bootchart picture will have a few of different bits of information that are of interest, especially the CPU and disk load graphs at the top.

There is a timeline running from left to right, with processes shown starting (and sometimes stopping) underneath it. The process CPU loads are indicated by the coloration of its process bar. The things to look for are the start and stop times of the different processes, and their CPU load. Long gaps with low load may mean a timeout or some other issue.

Once you have found out which processes are consuming time during the initialization, we can further check these by looking at the system logs with the logcat tool of Android; running them with strace, a profile (of Android), or perf; and reviewing the corresponding source code.

To speed system booting means to reduce the time cost at the system booting stage and also to increase the working frequency of all related devices and meanwhile reduce time cost resources. But, in real practice, the maximal frequency of the devices should be limited to reduce the power cost.

The whole booting procedure of an embedded system includes bootloader startup, kernel decompression and bootup, init process launching, and system booting. bootchartd is mainly for optimization of the booting stage after the init process launches.

To measure the boot time of bootloader or the Linux kernel, we can use the Grabserial tool from http://elinux.org/Grabserial. For more related resources, visit http://www.elinux.org/Boot_Time.

System stability is not only for personal computers and server workstations, but is also critically important for embedded systems. Every smartphone user has faced application non-responsiveness, system hangs, and abnormal restart issues; these fall in the scope of system stability. Some common optimization methods include the following:

Strengthening system stability means fixing the issues mentioned earlier as early as possible and much before the product release stage. System stability is a system feature throughout the development procedure and must be paid more attention.

We'll not discuss all of the methods mentioned previously in detail, but only discuss the part about function testing. To get a stable Linux kernel, we must do a lot of work to test the function of the kernel, device driver, and applications with enough pressure and coverage.

Use the kernel and device drivers as examples; we can test them by writing a set of programs in the shell, and even in the C language. The programs can verify the low-level kernel drivers through the exported sysfs and procfs or debugfs interfaces.

For example, to test the backlight interface of an LCD driver, we write the following piece of shell code:

#!/system/bin/sh
# This is a backlight test script
bl_path=/sys/class/backlight/bl_xx/brightness
for i in $(seq 255 -1 1)
do
   sleep 1
   echo $i > $bl_path
done

Save it to bl_test.sh, and push it to /system/bin/ in the Android emulator and run it.

$ adb remount
$ adb push bl_test.sh /system/bin
$ adb shell chmod 777 /system/bin/bl_test.sh
$ adb shell /system/bin/bl_test.sh

Then, we can change the brightness value from 255 to 1 automatically with a 1 second interval. Such scripts can be used to perform basic function verification and pressure testing.

System stability in its entirety is a vast topic; it is related to every part of a system, from hardware to software, from the Linux kernel to the Android system and applications.

To improve system stability, the entire project lifecycle must be strictly controlled with standard software QA systems.

Faults should be diagnosed and filtered out during the early requirement analysis, design, and development stages. Delaying them up to the release stage or maintenance stage may cost more resources and even result in financial loss. Errors should be tolerant at runtime.

Test automation is a good method to emulate the using scenes in the release stage. If the test cases are well designed with enough coverage and pressure, most potential issues can be discovered during the testing stage.

Here we would like to introduce a project; that is, the Device driver testing framework for TI OMAP. These test cases for OMAP-specific device drivers are written and maintained in an open Git tree. Refer to http://omappedia.org/wiki/OMAP_Kernel_driver_tests. To use this framework for Android, you only need BusyBox and Bash. Both of them have already been introduced before.

We can port this framework to other platforms and develop our own test cases based on this framework; it is good as practice, and left for readers who want to delve into the topic in depth.

For Android applications, CTS and monkeyrunner are available for compatibility and stability testing.

There are some tools available to fix reported failures as quickly as possible.

Android/Linux comes with logging utilities, such as RAM console and Logger/Logcat. They capture data that will allow the conditions associated with the failure to be determined.

The Ftrace-based systrace tool is also added for function-level tracing.

For debugging, gdbserver is available; besides this, the BusyBox top, iostat, and devmem options may give some help. In our example, we will only learn about some of them.

Let's talk about debug filesystems, such as debugfs, sysfs, and procfs and tools, such as Ftrace, top, iostat, and devmem:

Faults should be filtered out before the maintenance stage, but not all of them can be discovered. If failures occur, logging methods must be provided to record and report the failure scene. With the failure logs, the tracing tools can be used to track the system status and find the problematic areas; lastly, if necessary, use a debugger to debug interactively to locate the problems more precisely.

Introduce the Ftrace-based systrace.py tool provided by Android.

# systrace.py --disk --time=10 -o mynewtrace.html

The output mynewtrace.html shows the kernel activities graphically and allows us to track the system status easily. The output is in HTML5 format and can be parsed by most modern browsers. For more information about it, visit http://developer.android.com/tools/help/systrace.html.

Other similar tools are listed at http://www.elinux.org/Toolbox.

We have demonstrated some useful tools provided by BusyBox. With these tools, we can optimize the embedded system for size, stability, power consumption, boot speed, and serviceability. But this is just an initiation for us. We hope you will dig into it further and make more progress in your own embedded system development.