Chapter 1. Instant Optimizing Embedded System Using BusyBox

Welcome to Instant Optimizing Embedded System Using BusyBox.

BusyBox is a popular Unix toolbox; it integrates tiny versions of many common Unix utilities. It can be easily configured and compiled into a small, single-binary executable file. The built-in utilities are also known as BusyBox applets; they can be easily installed on an embedded system to add functionality or optimize performance. They can together also be used to build new embedded systems with extra device nodes, configuration files, and a Linux kernel.

This book will walk you through the configuration, compiling, and installation of the single-binary Unix toolbox, BusyBox. We'll show its basic usage on a desktop development system and the Android emulator, build a bootable embedded system from scratch and boot it on a virtual Android device, discuss how to meet diverse function requirements of BusyBox-based embedded systems, and explore how to utilize some powerful built-in applets to optimize different aspects of an embedded system.

Throughout this book, Ubuntu, as the most popular Linux distribution, will be used as our default desktop development system. Android, as the most popular embedded Linux system, will be used as our target embedded system. Android emulator, as the easy and cheap way to get a running Android device with root permission, will be used to build the Android experiment platform for demonstration.

Adding new applets to a BusyBox-based embedded system (Intermediate)

If BusyBox built-in applets are not enough for our embedded requirement, new tools should be added.

This recipe shows three methods to integrate new utilities to a BusyBox-based embedded system.

Getting ready

In order to extend a fork of BusyBox by adding functionality from an existing utility, just compile and install the new utility as a separate binary, which will be explained later. But to distribute one binary exactly to our embedded system and get a smaller-sized system, integrating new applets into BusyBox is a direct method.

To do so, use the small sstrip program as an example. sstrip is a small utility that removes the contents at the end of an ELF file that are not part of the program's memory image. ELF (http://en.wikipedia.org/wiki/Executable_and_Linkable_Format) is the common standard file format for Linux executables.

Download it with SVN or download it from this link: https://dev.openwrt.org/browser/trunk/tools/sstrip/src/sstrip.c. In our example, use svn to check it out.

$ cd ~/tools/
$ svn co svn://dev.openwrt.org/openwrt/trunk/tools/sstrip

How to do it...

To add the sstrip program into BusyBox, follow these steps:

Add the original tool to the BusyBox source code:
The new applet should be added to a suitable directory of BusyBox. In our example, sstrip is a misc utility so we add it to miscutils/.
```
$ cd ~/tools/busybox/
$ cp ../sstrip/src/sstrip.c miscutils/sstrip.c
```
Provide an applet-specific main entry, sstrip_main():
The entry of a new applet should be named applet_main(). For sstrip, it should be sstrip_main().
```
$ sed -i -e "s/int main/int sstrip_main/g" miscutils/sstrip.c
```

Add usage information:

Add the trivial and full usage macros into miscutils/sstrip.c.

//usage:# define sstrip_trivial_usage
//usage:# "sstrip FILE..."
//usage:# define sstrip_full_usage "\n\n"
//usage:# "sstrip discards all nonessential bytes from an executable.\n\n"
//usage:# "Version 2.0-X Copyright (C) 2000,2001 Brian Raiter.\n"
//usage:# "Cross-devel hacks Copyright (C) 2004 Manuel Novoa III.\n"
//usage:# "This program is free software, licensed under the GNU\n"
//usage:# "General Public License. There is absolutely no warranty.\n"

By default, the trivial usage is added to minimize the size of the compiled binary, but if you want verbose output with full usage information, the CONFIG_FEATURE_VERBOSE_USAGE option should be enabled.

Add configuration support:

We add the configuration description at the end of miscutils/Config.src.

config SSTRIP
   bool "sstrip"
   default n
   help
   sstrip is a small utility that removes the
   contents at the end of an ELF file that
   are not part of the program's memory image.

This allows users to enable or disable the sstrip applet via the configuration utility.

Add compiling support:
Add compiling support at the end of miscutils/Kbuild.src.
```
lib-$(CONFIG_SSTRIP)      += sstrip.o
```
This builds the sstrip binary in the last BusyBox binary when the CONFIG_SSTRIP configure option is enabled. 
Add macros declaration about main entry, usage information, links, and the installation directory:
Add the following line to include/applets.src.h to add the necessary macros.
```
IF_SSTRIP(APPLET(sstrip, BB_DIR_USR_SBIN, BB_SUID_DROP))
```
It lets BusyBox install sstrip to /usr/bin at the compiling stage with make install, and also exports the macros about the main entry, usage information, and links.
Use common header files:
Later, replace all of its header files except <elf.h> with libbb.h.
Configure and compile it:
To build sstrip in the BusyBox binary, enable it with make menuconfig:
```
   Miscellaneous Utilities  --->
      [*] sstrip
```
The docs/new-applet-HOWTO.txt file provides more information about how to add a new applet, but part of it is out of date. Real examples may be better references; for example, miscutils/chrt.c.

How it works...

BusyBox is modular and highly configurable. As seen previously, a new applet can be easily integrated, except renaming the main entry. The configure, build, and header files are added or modified for configuration, compiling, and installation respectively.

Since the main() entry is used by the BusyBox binary itself, the other applets' entries should be renamed with their own names.

The change in miscutils/Config.src adds a configure symbol, CONFIG_SSTRIP, with respective help information.

The modification in miscutils/Kbuild.src allows sstrip to be compiled while the CONFIG_SSTRIP symbol is enabled.

The addition in include/applets.src.h declares the prototypes of the new applet, sstrip, and the respective usage, main entry, and links.

There's more...

Lightweight tools like sstrip can be simply integrated into the BusyBox package. But for getting more functional utilities for our embedded systems, such methods don't work. However, we do have two other methods.

Build standalone tools:
One method is to compile the existing tools independently and then add them into a BusyBox-based embedded system.
As an example, we add a more powerful shell interpreter, Bash 4.2, into a BusyBox-based embedded system, and then download and compile it for ARM.
```
$ sudo apt-get install bison autoconf
$ cd ~/tools
$ git clone git://git.savannah.gnu.org/bash.git
$ cd bash
$ git checkout -b bash-4.2 f281b8f4f
$ ./configure –host=arm-linux-gnueabi --enable-static-link \ --without-bash-malloc
$ make
```
Now the compiled Bash can be put into ramdisk.img and used instead of BusyBox ash. Note that Bash is really powerful for writing shell scripts, but does cost more storage. It should not be used for a size-critical embedded system but rather for testing.

Automated building systems:

Another method is to use automated building system; such systems include Buildroot. Buildroot can automate the procedure we performed in the Building BusyBox-based embedded system (Intermediate) recipe. It facilitates the generation of a complete embedded Linux system. It not only includes a root filesystem and a kernel image, but also a cross-compilation toolchain and a bootloader image. The root filesystem integrates BusyBox and other powerful utilities, and even programming environments.

To build ramdisk.img automatically with Buildroot, create a configuration file, android_emulator_ramdisk_defconfig, first:

# architecture
BR2_arm=y
BR2_cortex_a8=y
BR2_EXTRA_GCC_CONFIG_OPTIONS="--with-fpu=vfpv3 --with-float=hard"
# BR2_SOFT_FLOAT is not set

# system
BR2_TARGET_GENERIC_HOSTNAME="android-emulator"

# development files, required by develop tools
BR2_DEPRECATED=y
BR2_HAVE_DEVFILES=y

# gcc
BR2_PACKAGE_GCC_TARGET=y

# package with cpio+gzip for init ramfs
BR2_TARGET_ROOTFS_CPIO=y
BR2_TARGET_ROOTFS_CPIO_GZIP=y
# BR2_TARGET_ROOTFS_TAR is not set

# bash shell
BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
BR2_PACKAGE_BASH=y

This configuration file allows us to build a BusyBox-based embedded system in the gzipped cpio archive with GCC and Bash support for the ARM target with Cortex A8 CPU.

Now download, configure, and compile it as follows:

$ cd ~/tools
$ wget http://www.buildroot.org/downloads/buildroot-2013.05.tar.bz2
$ tar jxf buildroot-2013.05.tar.bz2
$ cp android_emulator_ramdisk_defconfig .config
$ make oldconfig
$ make

The default major number of ttyS device is not the same as the Android ttyS device.

$ adb shell cat /proc/devices | grep ttyS
253 ttyS

The default tty console is not ttyS0, but ttyS2. So, we need to fix them and repackage ramdisk.img.

$ sed -i -e '/ttyS\t/s#4\t#253\t#g' system/device_table_dev.txt 
$ sed -i -e "/ttyS0/ittyS2::respawn:/sbin/getty -L ttyS2 115200 vt100" output/target/etc/inittab
$ make

After compiling, a target ramdisk image, output/images/rootfs.cpio.gz, should be ready for the Android emulator.

$ cp output/images/rootfs.cpio.gz ramdisk.img
$ emulator @busybox-avd -ramdisk ./ramdisk.img \
-show-kernel - shell -no-window
...
Welcome to Buildroot
android-emulator login:

After booting, log in as a root user. We can use the native GCC compiler directly without any extra configuration. The -no-window option disables window output for this simple root filesystem, and it doesn't include window support.

If you want to avoid some of the pitfalls of cross-compilation and want to build C applications directly on the Android device as an exercise, a native C programming environment can be built for Android system by creating another configuration file, android_emulator_defconfig.

# architecture
BR2_arm=y
BR2_cortex_a8=y
BR2_EXTRA_GCC_CONFIG_OPTIONS="--with-fpu=vfpv3 --with-float=hard"
# BR2_SOFT_FLOAT is not set

# system
BR2_TARGET_GENERIC_HOSTNAME="android-emulator"

# development files, required by develop tools
BR2_DEPRECATED=y
BR2_HAVE_DEVFILES=y

# gcc
BR2_PACKAGE_GCC_TARGET=y

# don't compile busybox in
BR2_INIT_NONE=y
# BR2_PACKAGE_BUSYBOX is not set

# don't tar of the rootfs
# BR2_TARGET_ROOTFS_TAR is not set

This configuration file tells us to compile the native compiler, GCC, for our ARM target device without BusyBox support and without packaging the target root filesystem.

Now, configure and compile it as follows:

$ cp android_emulator_defconfig .config
$ make oldconfig
$ make

Then prepare a configuration, native-gcc.rc, for the setting of environment variables as follows:

# Compile and Using uClibc library based native gcc compiler in Android system
export GCC_VER=4.7.3
export GCC_HOME=/data/gcc/
export GCC_NAME=arm-buildroot-linux-uclibcgnueabi
export GCC_LIB=$GCC_HOME/usr/lib/gcc/$GCC_NAME/$GCC_VER/
export GCC_LIBEXEC=$GCC_HOME/usr/libexec/gcc/$GCC_NAME/$GCC_VER/
export PATH=$GCC_HOME/bin:$GCC_HOME/sbin:$GCC_HOME/usr/bin:$GCC_HOME/usr/sbin:$PATH
export PATH=$GCC_HOME/usr/lib/ldscripts/:$GCC_LIB:$GCC_LIBEXEC:$PATH
export LD_LIBRARY_PATH=$GCC_HOME/usr/lib:$GCC_HOME/lib:$GCC_LIB:$GCC_LIBEXEC:$LD_LIBRARY_PATH
export LIBRARY_PATH=$LD_LIBRARY_PATH
export C_INCLUDE_PATH=$GCC_HOME/usr/include:$GCC_LIBEXEC/include/:$GCC_LIBEXEC/include-fixed

[ ! -d /lib ] && mkdir /lib
ln -sf $GCC_HOME/lib/ld-uClibc.so.0 /lib/ld-uClibc.so.0

This file mainly configures the PATH, LD_LIBRARY_PATH, LIBRARY_PATH, and C_LINCLUDE_PATH environment variables to let GCC works as is.

At last, let's push the target directory and configuration to the Android system, and use the native GCC to compile a simple program as follows:

$ adb push output/target /data/gcc
$ adb push native-gcc.rc /data/
$ adb shell
root@android:/ # source /data/native-gcc.rc
root@android:/ # echo -e '#include <stdio.h>\nmain(void){ printf("Hello, Android\\n"); }' | gcc -xc -o t - && ./t
Hello, Android

In reality, with Bash (the popular shell-scripting environment) and with C (the popular native programming environment), we're able to re-use lots of existing shell scripts, C software, or write new ones for our BusyBox-based embedded system to enhance its functionality to a very flexible extent.

Tailoring the system size of an embedded (Android) system (Advanced)

We have now discussed the basic configuration, compiling, installation, and usage of BusyBox on virtual Android devices and demonstrated how to build a bootable embedded system with BusyBox from scratch and enhanced the system through integrating additional utilities, including lightweight applets, standalone tools, and even programming language environments. In short, the functionality implementation of a scalable embedded system with BusyBox has been discussed in depth in previous recipes.

This recipe onwards, we will learn how to enhance user experience—it's time to talk about system optimization. Size optimization will be discussed in this recipe.

Getting ready

For most embedded systems (particularly consumer electronics like smartphone systems), due to the limitation of product cost, size of the hardware model, and rich-function requirements, system image size should be limited to fit the size of the disk and memory and reserve more free storage for end users.

How to do it...

The software of an embedded Linux system mainly includes bootloader, kernel, ramdisk, system, and applications. Use the embedded filesystem we just built with BusyBox in the Building BusyBox-based embedded system (Intermediate) recipe as an example. To reduce its size, we should:

Demand determine features:
Remove the functions that are not required by the target system by disabling those features, via the configuration utility.
For example, use defconfig as a configuration base and enable the necessary features and disable the rest.
Share common features in libraries:
When adding new features to BusyBox, try to use the library functionality already present in BusyBox and remove the code from the utility that duplicates that functionality.
For example, for the sstrip program we just integrated, if some of its functions already exist in the common libraries of BusyBox, they should be removed.
Static linking versus dynamic linking:
If possible, link BusyBox with a lightweight standard library such as uClibc (http://www.uclibc.org/; it is optimized for an embedded system and is smaller than Glibc), not Glibc.
If extra tools are required and support dynamic linking to share the same library, don't link BusyBox statically. But if you only need BusyBox, link statically to avoid the need of a dynamic linker and loader.
Utilize compiler and linker support:
Enable the -Os option of GCC by disabling the CONFIG_DEBUG_PESSIMIZE configuration option of BusyBox and enable --gc-sections using not Glibc (see the comments in scripts/trylink), and with a new enough ld tool.
Strip everything that is not required at runtime:
Use strip to discard symbols from object files, and use sstrip to remove the section header table at the end of the executable.
It should be noted that an executable file that has no section header table cannot be used with gdb or objdump for debugging. You can learn more from the comments of sstrip.c.
Compression is a technique with external perspective:
Instead of optimizing the size of the filesystem, compressing the whole filesystem image with a suitable algorithm is a technique with external perspective. The algorithms include a general gzip or lzma format with higher compression rate, and LZO or LZ4 with lower compression rate but faster decompression speed.
For example, the embedded filesystem we created in the Building BusyBox-based embedded system (Intermediate) recipe is packaged in cpio format and finally compressed with gzip.
To use such algorithms, corresponding decompression support is required in bootloader, or else the Linux kernel for the filesystem will be unpacked into memory at boot time.
Measure before starting optimization:
Find the objects consuming large storage by measuring the size of each object or executable file and then optimize them one by one. The tools include ls -l, du, size, and nm –size.
For example, the size utility can be used to output the size of different segments of an executable as follows:
```
$ size busybox
   text   data    bss    dec    hex   filename
 824447   4154   9520  838121  cc9e9   busybox
```

How it works...

The main storage consumers of an embedded Linux system are executables. A standard Linux executable is in an ELF format. Besides an ELF header, a program header table, and a section header table, it consists of the text segment, data segment (initialized), and BSS segment (uninitialized).

To optimize size means to remove unnecessary executables or to reduce the size of the previously mentioned segments of the executables by sharing common functions or data, or even via sharing a single ELF header.

BusyBox is highly modular and configurable, and therefore allows to choose only the necessary applets based on the real system requirements. And as a single binary, all of the built-in applets share one ELF header and share all of the common functions. Besides those, compiler size optimization options like -Os and --gc-sections are also well supported.

Also, the last binary can be stripped with strip and sstrip, and the generated filesystem image can be compressed.

There's more...

We have discussed size optimization of ramdisk built with BusyBox. For the other parts of an embedded Linux system and the other methods, http://www.elinux.org/System_Size is a very good reference.

For the kernel parts, we can get more useful information from a project proposed by one of the authors of this book, at http://elinux.org/Work_on_Tiny_Linux_Kernel.

Reducing the power consumption of an embedded (Android) system (Advanced)

For most embedded systems, particularly for consumer electronics like smartphone systems, in order to extend battery life and reduce charging time per day, power consumption should be controlled.

The power issue becomes more and more critical in smartphone systems because of the conflicts of higher function requirements and performance requirements, but limited battery capacity and a smaller hardware model.

Getting ready

The whole power-cost optimization topic is very systematic; it is related to hardware system design and the power management policies of the software system.

The hardware elements that need to be considered in the power-cost balance are CPU, memory and external devices, and system components, such as clock, power domain, and regulator.

The corresponding software management topics include CPU idle, CPU frequency, CPU hotplug, memory hotplug, memory frequency, bus frequency, runtime power management, clock management, power domain management, and system suspend.

It's not possible to explain all of them, but we'll introduce an applet that can give us power optimization suggestions, and can measure top power-cost events.

How to do it...

PowerTOP (https://01.org/powertop/) is designed to measure, explain, and minimize a computer's electrical power consumption; it not only works on x86 systems, but also supports ARM, AMD, and other kinds of architectures.

Enable basic Linux kernel support for PowerTOP:
To make PowerTOP work, the kernel must be configured with CONFIG_TIMER_STATS=y.
```
   Kernel hacking  --->
      [*] Collect kernel timers statistics
```
Then compile and flush it to the target platform.
Configure PowerTOP in BusyBox:
BusyBox already has a tiny version of PowerTOP integrated; to use it, make sure it is enabled in the BusyBox configuration.
```
   Process Utilities  --->
       [*] powertop
```

Using PowerTOP:

Let's run it as follows on the Android emulator:

$ adb shell
shell@android:/ # powertop
C-state information is not available
Wakeups-from-idle in 10 seconds: 308
Top causes for wakeups:
  38.1% (  115)       <interrupt> : Goldfish Timer Tick
  17.9% (   54)     <kernel core> : hrtimer_start (tick_sched_timer)
  14.9% (   45)        yaffs-bg-1 : add_timer (yaffs_background_waker)
  12.3% (   37)     <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
   4.6% (   14)       <interrupt> : goldfish_pipe
   3.6% (   11)   m.android.phone : hrtimer_start_range_ns (hrtimer_wakeup)
   3.3% (   10)     <kernel core> : dev_watchdog (dev_watchdog)
   3.0% (    9)       VSyncThread : hrtimer_start_range_ns 
(hrtimer_wakeup)
   0.7% (    2)     <kernel core> : bdi_arm_supers_timer (sync_supers_timer_fn)
   0.3% (    1)       <interrupt> : goldfish_fb
   0.3% (    1)   FinalizerWatchd : hrtimer_start_range_ns (hrtimer_wakeup)
   0.3% (    1)   er.ServerThread : hrtimer_start (devalarm_hrthandler)

We can see that the Goldfish Timer Tick interrupt wakes up the system that is idle for 115 times in 10 seconds. In a real Android device, it may differ from the mentioned outputs.

If the events wake up the system that is idle too frequently, the power cost will be increased remarkably. So, the major causes for wakeups should be attended to and their working frequency should be decreased by tuning the respective kernel drivers.

For example, to reduce the interrupt of Goldfish Timer Tick, the HZ configuration of the Linux kernel should be decreased to reduce the ticks per second; for example, replacing CONFIG_HZ=1000 with CONFIG_HZ=100 may reduce 90 percent system timer interrupts in a second.

How it works...

The ideal working status that is power cost friendly is putting the system into an idle state as far as possible or eventually shutting down all of the devices.

To reduce the power cost of a single device, we should put it into a deeper idle state with less voltage and working frequency (if supported) or power it off eventually if nobody uses it. On the contrary, if some events wake up the device from the idle state to the active state frequently, the device will cost more power; such events are often used as the most important indicators to reduce power cost.

There's more...

We learned about the switching of a system or device state from idle to active and introduced the PowerTOP tool, which can monitor this switching.

In Android, the command dumpsys alarm can track the alarms that wake up the system from a suspend state (a deeper idle state).

But even if we fix all potential wakeup causes, the power cost may still need to be optimized in other aspects. This means even in an idle state, there will be power leaking, which should be fixed up by the GPIO and register settings based on the datasheets of specific chips.

In an active state, power and performance should be balanced carefully with DVFS, clock gating, regulator gating, or power QOS policies.

The system or device should enter into an idle state from an active state timely if there is no activity.

In Android, if there is an activity request, wake_lock is held. The system will not switch from an active state to an idle state until the lock is released. To reduce power cost, the lock-holding time should be as short as possible. To track the holding of such wake locks, the /proc/wakelocks or /sys/kernel/debug/wakeup_sources interface helps, and the dumpsys power command may give more related information.

Speeding up the system boot of an embedded (Android) system (Advanced)

For most embedded systems, particularly for consumer electronics like smartphone systems, to reduce the wait time of system power on or start up, the system must be fast enough.

Most embedded devices have strict time limits from power on to showing graphic UI, so speeding the boot time should be considered an important feature.

Getting ready

The following are the basic methods to speed system booting:

Disable unused features and functions in all booting components (bootloader, Linux kernel, system, and applications) of an embedded Linux system. This means let the system perform less but necessary jobs.
Some initialization procedures can be deferred to the background after the device has booted to a basically workable state.
If a multicore CPU is being used, parallize the boot process with multithreads.
Based on the considerations of power-cost optimization, set CPU frequency, bus frequency, memory frequency, and even external device working frequency (such as EMMC) as high as possible.
Measure the boot procedure and find out the time-cost bottlenecks.

How to do it...

Bootchart (http://www.bootchart.org/) is for performance analysis and visualization of the Linux boot process; it can be used for boot speedup optimization.

Android does include the init method's built-in bootchartd support, but requires the init application to be rebuilt; for more details, visit http://elinux.org/Using_Bootchart_on_Android.

In our example, we introduce the BusyBox built-in bootchartd support to measure the Android system boot procedure.

Configure Bootchart in BusyBox:

First, enable the Bootchart applet in BusyBox.

   Init Utilities  --->
      [*] bootchartd
       [*]   Compatible, bloated header
       [*]   Support bootchartd.conf

Install BusyBox with bootchartd in Android ramdisk.img:
Then, copy the busybox binary to Android ramdisk.img, create a link /bootchartd to busybox, and create a var/log directory.
```
$ cd ramdisk/
$ ln -sf busybox /bootchartd
$ mkdir var/log
```
Recompress the ramdisk/ to ramdisk.img with the following:
```
$ find . | cpio -H newc -o | gzip -9 > ../ramdisk.img
```
The link is for directly using bootchartd as the init process; the var/log directory is for saving the boot log generated by bootchartd.
Measure the Android system booting with BusyBox bootchard:
Now start the Android emulator.
```
$ emulator @busybox-avd -shell -show-kernel -kernel ./zImage \
   -ramdisk ./ramdisk.img \
   -qemu -append "rdinit=/bootchartd bootchard_init=/init"
```
The -qemu -append options allow us to pass extra arguments to the Linux kernel; the preceding rdinit=/bootchartd support replaces the default ramdisk init with bootchartd, and then bootchartd_init=/init tells bootchartd to start the real /init process.
Get the bootchartd measuring log:
After a while, the Android UI will be started and a new file will be created to save the boot logs.
```
shell@android # ls -l /var/log
total 72
-rw-r--r--    1 0        0            73334 Jul  5 17:24 bootlog.tgz
```
Then, pull the logfile into our development system.
```
$ adb pull /var/log/bootlog.tgz
```

Draw the bootchartd log with pybootchartgui, and analyze it.

Analyze the boot log with pybootchartgui. This tool can generate a readable graphic output based on the previous boot logfile. In Ubuntu, we can install it simply with the following:

$ sudo apt-get install bootchart pybootchartgui

Then draw a graph with it.

$ bootchart ./bootlog.tgz

It may generate an error, such as the header file format is not compatible with the original larger version of the bootchart:

  File "/usr/lib/pymodules/python2.7/pybootchartgui/draw.py", line 341, in draw_header
      txt = headertitle + ': ' + mangle(headers.get(headerkey))
      TypeError: cannot concatenate 'str' and 'NoneType' objects

To work around it, just let the draw.py file of pybootchartgui ignore the errors with except handling.

$ vim /usr/lib/pymodules/python2.7/pybootchartgui/draw.py
338     for (headerkey, headertitle, mangle) in toshow:
339         header_y += ctx.font_extents()[2]
340         try:
341                 txt = headertitle + ': ' + mangle(headers.get(headerkey))
342         except:
343                 continue
344         draw_text(ctx, txt, TEXT_COLOR, off_x, header_y)

If there are no other issues, we should get results successfully, as shown in the following screenshot:

At last, it's time to analyze the graph.

A bootchart picture will have a few of different bits of information that are of interest, especially the CPU and disk load graphs at the top.

There is a timeline running from left to right, with processes shown starting (and sometimes stopping) underneath it. The process CPU loads are indicated by the coloration of its process bar. The things to look for are the start and stop times of the different processes, and their CPU load. Long gaps with low load may mean a timeout or some other issue.

Once you have found out which processes are consuming time during the initialization, we can further check these by looking at the system logs with the logcat tool of Android; running them with strace, a profile (of Android), or perf; and reviewing the corresponding source code.

How it works...

To speed system booting means to reduce the time cost at the system booting stage and also to increase the working frequency of all related devices and meanwhile reduce time cost resources. But, in real practice, the maximal frequency of the devices should be limited to reduce the power cost.

There's more...

The whole booting procedure of an embedded system includes bootloader startup, kernel decompression and bootup, init process launching, and system booting. bootchartd is mainly for optimization of the booting stage after the init process launches.

To measure the boot time of bootloader or the Linux kernel, we can use the Grabserial tool from http://elinux.org/Grabserial. For more related resources, visit http://www.elinux.org/Boot_Time.

Enhancing the system stability of an embedded (Android) system (Advanced)

For most embedded systems, particularly for consumer electronics like smartphones, due to their complexity, we must apply some policies to strengthen the stability of the system.

Getting ready

System stability is not only for personal computers and server workstations, but is also critically important for embedded systems. Every smartphone user has faced application non-responsiveness, system hangs, and abnormal restart issues; these fall in the scope of system stability. Some common optimization methods include the following:

Static checkers and analyzers, such as GCC, Sparse, CocciCheck (http://coccinelle.lip6.fr/coccicheck.php), and Smatch (http://smatch.sourceforge.net/)
Peer to Peer Review; an example of a tool for P2P review is Patchwork (http://jk.ozlabs.org/projects/patchwork/)
Suitable internal error handling and error tolerance
A hardware or software watchdog for external error checks with timeout detection
Function testing with enough coverage and batch testing with pressure

How to do it...

Strengthening system stability means fixing the issues mentioned earlier as early as possible and much before the product release stage. System stability is a system feature throughout the development procedure and must be paid more attention.

We'll not discuss all of the methods mentioned previously in detail, but only discuss the part about function testing. To get a stable Linux kernel, we must do a lot of work to test the function of the kernel, device driver, and applications with enough pressure and coverage.

Use the kernel and device drivers as examples; we can test them by writing a set of programs in the shell, and even in the C language. The programs can verify the low-level kernel drivers through the exported sysfs and procfs or debugfs interfaces.

For example, to test the backlight interface of an LCD driver, we write the following piece of shell code:

#!/system/bin/sh
# This is a backlight test script
bl_path=/sys/class/backlight/bl_xx/brightness
for i in $(seq 255 -1 1)
do
   sleep 1
   echo $i > $bl_path
done

Save it to bl_test.sh, and push it to /system/bin/ in the Android emulator and run it.

$ adb remount
$ adb push bl_test.sh /system/bin
$ adb shell chmod 777 /system/bin/bl_test.sh
$ adb shell /system/bin/bl_test.sh

Then, we can change the brightness value from 255 to 1 automatically with a 1 second interval. Such scripts can be used to perform basic function verification and pressure testing.

How it works...

System stability in its entirety is a vast topic; it is related to every part of a system, from hardware to software, from the Linux kernel to the Android system and applications.

To improve system stability, the entire project lifecycle must be strictly controlled with standard software QA systems.

Faults should be diagnosed and filtered out during the early requirement analysis, design, and development stages. Delaying them up to the release stage or maintenance stage may cost more resources and even result in financial loss. Errors should be tolerant at runtime.

Test automation is a good method to emulate the using scenes in the release stage. If the test cases are well designed with enough coverage and pressure, most potential issues can be discovered during the testing stage.

There's more...

Here we would like to introduce a project; that is, the Device driver testing framework for TI OMAP. These test cases for OMAP-specific device drivers are written and maintained in an open Git tree. Refer to http://omappedia.org/wiki/OMAP_Kernel_driver_tests. To use this framework for Android, you only need BusyBox and Bash. Both of them have already been introduced before.

We can port this framework to other platforms and develop our own test cases based on this framework; it is good as practice, and left for readers who want to delve into the topic in depth.

For Android applications, CTS and monkeyrunner are available for compatibility and stability testing.

Increasing the serviceability of an embedded (Android) system (Advanced)

At the maintenance stage of a software's lifecycle, techniques to restore the failure generation scene are critical. These include system logging, system debugging, and tracing.

Getting ready

There are some tools available to fix reported failures as quickly as possible.

Android/Linux comes with logging utilities, such as RAM console and Logger/Logcat. They capture data that will allow the conditions associated with the failure to be determined.

The Ftrace-based systrace tool is also added for function-level tracing.

For debugging, gdbserver is available; besides this, the BusyBox top, iostat, and devmem options may give some help. In our example, we will only learn about some of them.

How to do it...

Let's talk about debug filesystems, such as debugfs, sysfs, and procfs and tools, such as Ftrace, top, iostat, and devmem:

Kernel debugging interfaces, such as debugfs, sysfs, and procfs:
To debug the Linux kernel, we should enable its debugging features with CONFIG_DEBUG_KERNEL=y, and further, CONFIG_DEBUG_FS=y creates a virtual debug filesystem (debugfs). We can mount it at runtime.
```
shell@android:/ #  mount -t debugfs none /sys/kernel/debug/
shell@android:/ #  ls /sys/kernel/debug/
bdi             hid             mmc0            suspend_stats   wakeup_sources binder          memblock        sched_features  tracing
```
In an Android system, there is a /d directory linked to /sys/kernel/debug/ under the root directory. As we've mentioned before, the Android ramdisk has already mounted sysfs and procfs to the system. They are useful kernel interfaces, and we can use them to communicate with the kernel, monitor the runtime kernel status, and diagnose issues.

Ftrace – tracing kernel functions and events:

Ftrace is an internal tracing framework in the Linux kernel; it can be used to trace kernel functions. The interface can be found at /sys/kernel/debug/tracing. To use it, the following configuration options should be enabled:

CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y

Provided we've configured this and started an emulator with this kernel, we can mount debugfs on our Android system, and then enter the tracing directory.

shell@android:/ #  cd /sys/kernel/debug/tracing

For example, if want to trace kernel functions, we can enable the function tracer.

shell@android:/ #  echo function > current_tracer
shell@android:/ #  echo 1 > tracing_on
shell@android:/ #  ls
shell@android:/ #  echo 0 > tracing_on
shell@android:/ #  cat trace | head -20
tracer: function
entries-in-buffer/entries-written: 60151/335518   #P:1
                             _-----=> irqs-off
                            / _----=> need-resched
                           | / _---=> hardirq
/softirq
                           || / _--=> preempt-depth
                           ||| /     delay
          TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
             | |       |   ||||       |         |
ActivityManager-328   [000] ....   173.080000: find_pid_ns <-find_ge_pid
ActivityManager-328   [000] ....   173.080000: pid_nr_ns <-next_
tgid
ActivityManager-328   [000] ....   173.080000: pid_task <-next_tgid
ActivityManager-328   [000] ....   173.080000: find_ge_pid <-next_tgid
ActivityManager-328   [000] ....   173.080000: find_pid_ns <-find_ge_pid
ActivityManager-328   [000] ....   173.080000: pid_nr_ns <-next_tgid
ActivityManager-328   [000] ....   173.080000: pid_task <-next_tgid
ActivityManager-328   [000] ....   173.080000: find_ge_pid <-next_tgid
ActivityManager-328   [000] ....   173.080000: find_pid_ns <-find_ge_pid

In reality, Ftrace also provides the other specific tracers, such as the irqsoff and preemptoff tracers. Both of them can be used to find the bottlenecks that may increase system latency. Tracers that are precompiled are listed in the available_tracers interface.

shell@android:/ # cat /sys/kernel/debug/tracing/available_tracers
blk function_graph mmiotrace wakeup_rt wakeup function nop

Besides, it also provides tracepoint- and kprobe-based event tracers that trace some specific group of kernel functions. These groups include a subsystem or a particular kernel module, such as ext4, IRQ, GPIO, power, syscalls, or workqueue. Look at the available events by going to /sys/kernel/debug/tracing/available_events. To learn more about Ftrace, read the documents under the Documentation/trace directory of theLinux kernel source code.

Top – monitors process statuses:

The top applet of BusyBox can be used to monitor processes' status, including CPU and memory utilization. It can display Linux processes, including CPU occupancy, memory usage, and the status of each process. For example:

shell@android:/ #  busybox top
Mem: 253776K used, 511960K free, 0K shrd, 7920K buff, 131144K 
cached
CPU: 75.7% usr 23.0% sys  1.1% nic  0.0% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 3.92 0.92 0.30 14/245 592
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  318    57 1000     S     224m 29.9   0 25.1 system_server
   56     1 1000     S    60308  7.8   0 24.1 /system/bin/surfaceflinger
  377    57 10019    R     162m 21.6   0 14.0 {ndroid.systemui} com.android.syst
  460    57 10022    S     169m 22.7   0 11.0 {ndroid.launcher} com.android.laun
  452    57 1001     S     162m 21.6   0  9.1 {m.android.phone} com.android.phon
  527    57 10009    S     159m 21.2   0  4.1 {d.process.acore} android.process.
   91     1 1003     S <  23036  3.0   0  3.3 /system/bin/bootanimation
   59     1 1013     S    21000  2.7   0  2.1 /system/bin/mediaserver
  382    51 0        R    21072  2.7   0  1.3 /system/bin/fsck_msdos -p -f /dev/
  559    65 2000     R     1596  0.2   0  0.4 busybox top
   44     2 0        SW       0  0.0   0  0.4 [mmcqd/0]
   57     1 0        S     140m 18.7   0  0.2 zygote /bin/app_process -Xzygote /
   50     1 1000     S      888  0.1   0  0.2 /system/bin/servicemanager
    1     0 0        S      348  0.0   0  0.2 /init
  567    57 10009    D     153m 20.4   0  0.0 {ndroid.contacts} com.android.cont
  505    57 10001    S     151m 20.2   0  0.0 {d.process.media} android.process.
  439    57 10003    S     151m 20.1   0  0.0 {putmethod.latin} com.android.inpu
  482    57 10031    S     150m 20.0   0  0.0 {m.android.music} com.android.musi
   53     1 0        S     9716  1.2   0  0.0 /system/bin/netd
   58     1 1019     S     6612  0.8   0  0.0 /system/bin/drmserver

We can see the verbose outputs. PID shows the process ID, PPID shows the parent process ID of the current process, USER specifies the user ID of each process, STAT represents the status of process, where R is for run and S is for sleep, VSZ shows the virtual memory size used by the process, and %CPU shows the percentage of CPU occupancy of the process.

According to the outputs, we can find which process consumes more system power and system memory and which process is dead, so we can take proper decisions to improve code efficiency and system performance.

iostat – monitors the I/O status

The iostat applet of BusyBox is a tool to report CPU and I/O statistics. It will show the average occupation for CPU in idle state; I/O wait response; and the user, system, and steal modes.

The device status will also be displayed in read/write speed. We can show only CPU stats with the -c command and device stats with the -d option:

shell@android:/d/tracing # busybox iostat -cd
Linux 3.4.0-g18554c7 (localhost) 	07/02/13 	_armv7l_	(1 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.30    0.03    1.20    0.00    0.00   97.47
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
mmcblk0           0.22         1.80         0.00      25114    1

devmem – monitors the hardware status:
The devmem applet of BusyBox is a useful tool to read from or write values to a physical address. This is a very efficient trick to read the hardware register or write an expected value to that register. For example:
Read the 0x1000000 physical address.
```
shell@android:/ # devmem 0x10000000
0x00000000
```
Write 0xF000000F to the 0x10000000 physical address:
```
shell@android:/ # devmem 0x10000000 32 0xF000000F
shell@android:/ # devmem 0x10000000
0xF000000F
```
In a real Android device, the SOC (System On Chip) has its own CoreSight TM (the ARM debug and trace technology is the most complete on-chip debug and real-time trace solution); it can access from 0 to 4 GB address space, but there are some limits to using these spaces. Some part of this space is used as an SFR (Special Function Register) area, and some are used for the physical memory area. So, use devmem to see the value of an SFR. We can read its status and change its value if it is writable; this is often used while the system stays in an unstable state.

How it works...

Faults should be filtered out before the maintenance stage, but not all of them can be discovered. If failures occur, logging methods must be provided to record and report the failure scene. With the failure logs, the tracing tools can be used to track the system status and find the problematic areas; lastly, if necessary, use a debugger to debug interactively to locate the problems more precisely.

There's more...

Introduce the Ftrace-based systrace.py tool provided by Android.

# systrace.py --disk --time=10 -o mynewtrace.html

The output mynewtrace.html shows the kernel activities graphically and allows us to track the system status easily. The output is in HTML5 format and can be parsed by most modern browsers. For more information about it, visit http://developer.android.com/tools/help/systrace.html.

Other similar tools are listed at http://www.elinux.org/Toolbox.

We have demonstrated some useful tools provided by BusyBox. With these tools, we can optimize the embedded system for size, stability, power consumption, boot speed, and serviceability. But this is just an initiation for us. We hope you will dig into it further and make more progress in your own embedded system development.