At the maintenance stage of a software's lifecycle, techniques to restore the failure generation scene are critical. These include system logging, system debugging, and tracing.
There are some tools available to fix reported failures as quickly as possible.
Android/Linux comes with logging utilities, such as RAM console and Logger/Logcat. They capture data that will allow the conditions associated with the failure to be determined.
The Ftrace-based systrace tool is also added for function-level tracing.
For debugging, gdbserver is available; besides this, the BusyBox top, iostat, and devmem options may give some help. In our example, we will only learn about some of them.
Let's talk about debug filesystems, such as debugfs, sysfs, and procfs and tools, such as Ftrace, top, iostat, and devmem:
Kernel debugging interfaces, such as
debugfs,sysfs, andprocfs:To debug the Linux kernel, we should enable its debugging features with
CONFIG_DEBUG_KERNEL=y, and further,CONFIG_DEBUG_FS=ycreates a virtual debug filesystem (debugfs). We can mount it at runtime.shell@android:/ # mount -t debugfs none /sys/kernel/debug/ shell@android:/ # ls /sys/kernel/debug/ bdi hid mmc0 suspend_stats wakeup_sources binder memblock sched_features tracing
In an Android system, there is a
/ddirectory linked to/sys/kernel/debug/under the root directory. As we've mentioned before, the Android ramdisk has already mountedsysfsandprocfsto the system. They are useful kernel interfaces, and we can use them to communicate with the kernel, monitor the runtime kernel status, and diagnose issues.Ftrace – tracing kernel functions and events:
Ftrace is an internal tracing framework in the Linux kernel; it can be used to trace kernel functions. The interface can be found at
/sys/kernel/debug/tracing. To use it, the following configuration options should be enabled:CONFIG_FTRACE=y CONFIG_FUNCTION_TRACER=y CONFIG_FUNCTION_GRAPH_TRACER=y
Provided we've configured this and started an emulator with this kernel, we can mount
debugfson our Android system, and then enter the tracing directory.shell@android:/ # cd /sys/kernel/debug/tracingFor example, if want to trace kernel functions, we can enable the function tracer.
shell@android:/ # echo function > current_tracer shell@android:/ # echo 1 > tracing_on shell@android:/ # ls shell@android:/ # echo 0 > tracing_on shell@android:/ # cat trace | head -20 tracer: function entries-in-buffer/entries-written: 60151/335518 #P:1 _-----=> irqs-off / _----=> need-resched | / _---=> hardirq /softirq || / _--=> preempt-depth ||| / delay TASK-PID CPU# |||| TIMESTAMP FUNCTION | | | |||| | | ActivityManager-328 [000] .... 173.080000: find_pid_ns <-find_ge_pid ActivityManager-328 [000] .... 173.080000: pid_nr_ns <-next_ tgid ActivityManager-328 [000] .... 173.080000: pid_task <-next_tgid ActivityManager-328 [000] .... 173.080000: find_ge_pid <-next_tgid ActivityManager-328 [000] .... 173.080000: find_pid_ns <-find_ge_pid ActivityManager-328 [000] .... 173.080000: pid_nr_ns <-next_tgid ActivityManager-328 [000] .... 173.080000: pid_task <-next_tgid ActivityManager-328 [000] .... 173.080000: find_ge_pid <-next_tgid ActivityManager-328 [000] .... 173.080000: find_pid_ns <-find_ge_pid
In reality, Ftrace also provides the other specific tracers, such as the irqsoff and preemptoff tracers. Both of them can be used to find the bottlenecks that may increase system latency. Tracers that are precompiled are listed in the
available_tracersinterface.shell@android:/ # cat /sys/kernel/debug/tracing/available_tracers blk function_graph mmiotrace wakeup_rt wakeup function nop
Besides, it also provides tracepoint- and kprobe-based event tracers that trace some specific group of kernel functions. These groups include a subsystem or a particular kernel module, such as ext4, IRQ, GPIO, power, syscalls, or workqueue. Look at the available events by going to
/sys/kernel/debug/tracing/available_events. To learn more about Ftrace, read the documents under theDocumentation/tracedirectory of theLinux kernel source code.Top – monitors process statuses:
The
topapplet of BusyBox can be used to monitor processes' status, including CPU and memory utilization. It can display Linux processes, including CPU occupancy, memory usage, and the status of each process. For example:shell@android:/ # busybox top Mem: 253776K used, 511960K free, 0K shrd, 7920K buff, 131144K cached CPU: 75.7% usr 23.0% sys 1.1% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq Load average: 3.92 0.92 0.30 14/245 592 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 318 57 1000 S 224m 29.9 0 25.1 system_server 56 1 1000 S 60308 7.8 0 24.1 /system/bin/surfaceflinger 377 57 10019 R 162m 21.6 0 14.0 {ndroid.systemui} com.android.syst 460 57 10022 S 169m 22.7 0 11.0 {ndroid.launcher} com.android.laun 452 57 1001 S 162m 21.6 0 9.1 {m.android.phone} com.android.phon 527 57 10009 S 159m 21.2 0 4.1 {d.process.acore} android.process. 91 1 1003 S < 23036 3.0 0 3.3 /system/bin/bootanimation 59 1 1013 S 21000 2.7 0 2.1 /system/bin/mediaserver 382 51 0 R 21072 2.7 0 1.3 /system/bin/fsck_msdos -p -f /dev/ 559 65 2000 R 1596 0.2 0 0.4 busybox top 44 2 0 SW 0 0.0 0 0.4 [mmcqd/0] 57 1 0 S 140m 18.7 0 0.2 zygote /bin/app_process -Xzygote / 50 1 1000 S 888 0.1 0 0.2 /system/bin/servicemanager 1 0 0 S 348 0.0 0 0.2 /init 567 57 10009 D 153m 20.4 0 0.0 {ndroid.contacts} com.android.cont 505 57 10001 S 151m 20.2 0 0.0 {d.process.media} android.process. 439 57 10003 S 151m 20.1 0 0.0 {putmethod.latin} com.android.inpu 482 57 10031 S 150m 20.0 0 0.0 {m.android.music} com.android.musi 53 1 0 S 9716 1.2 0 0.0 /system/bin/netd 58 1 1019 S 6612 0.8 0 0.0 /system/bin/drmserver
We can see the verbose outputs.
PIDshows the process ID,PPIDshows the parent process ID of the current process,USERspecifies the user ID of each process,STATrepresents the status of process, whereRis for run andSis for sleep,VSZshows the virtual memory size used by the process, and%CPUshows the percentage of CPU occupancy of the process.According to the outputs, we can find which process consumes more system power and system memory and which process is dead, so we can take proper decisions to improve code efficiency and system performance.
iostat – monitors the I/O status
The
iostatapplet of BusyBox is a tool to report CPU and I/O statistics. It will show the average occupation for CPU in idle state; I/O wait response; and the user, system, and steal modes.The device status will also be displayed in read/write speed. We can show only CPU stats with the
-ccommand and device stats with the-doption:shell@android:/d/tracing # busybox iostat -cd Linux 3.4.0-g18554c7 (localhost) 07/02/13 _armv7l_ (1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.30 0.03 1.20 0.00 0.00 97.47 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn mmcblk0 0.22 1.80 0.00 25114 1
devmem – monitors the hardware status:
The
devmemapplet of BusyBox is a useful tool to read from or write values to a physical address. This is a very efficient trick to read the hardware register or write an expected value to that register. For example:Read the
0x1000000physical address.shell@android:/ # devmem 0x10000000 0x00000000
Write
0xF000000Fto the0x10000000physical address:shell@android:/ # devmem 0x10000000 32 0xF000000F shell@android:/ # devmem 0x10000000 0xF000000F
In a real Android device, the SOC (System On Chip) has its own CoreSight TM (the ARM debug and trace technology is the most complete on-chip debug and real-time trace solution); it can access from 0 to 4 GB address space, but there are some limits to using these spaces. Some part of this space is used as an SFR (Special Function Register) area, and some are used for the physical memory area. So, use
devmemto see the value of an SFR. We can read its status and change its value if it is writable; this is often used while the system stays in an unstable state.
Faults should be filtered out before the maintenance stage, but not all of them can be discovered. If failures occur, logging methods must be provided to record and report the failure scene. With the failure logs, the tracing tools can be used to track the system status and find the problematic areas; lastly, if necessary, use a debugger to debug interactively to locate the problems more precisely.
Introduce the Ftrace-based systrace.py tool provided by Android.
# systrace.py --disk --time=10 -o mynewtrace.html
The output mynewtrace.html shows the kernel activities graphically and allows us to track the system status easily. The output is in HTML5 format and can be parsed by most modern browsers. For more information about it, visit http://developer.android.com/tools/help/systrace.html.
Other similar tools are listed at http://www.elinux.org/Toolbox.
We have demonstrated some useful tools provided by BusyBox. With these tools, we can optimize the embedded system for size, stability, power consumption, boot speed, and serviceability. But this is just an initiation for us. We hope you will dig into it further and make more progress in your own embedded system development.
