In this chapter, we will cover the following recipes:
Debugging an embedded Linux product is a common task not only during development, but also in deployed production systems.
Application debugging in embedded Linux is different from debugging in a traditional embedded device in that we don't have a flat memory model with an operating system and applications sharing the same address space. Instead, we have a virtual memory model with the Linux operating system sharing the address space, and assigning virtual memory areas to running processes.
With this model, the mechanisms used for kernel and user space debugging differ. For example, the traditional model of using a JTAG-based hardware debugger is useful for kernel debugging, but unless it knows about the user space process' memory mapping, it will not be able to debug user space applications.
Application debugging is approached with the use of a user space debugger service. We have seen an example of this methodology in action with the TCF agent used in the Eclipse GDB. The other...
Even after extensive quality assurance testing, embedded systems in-field also fail and need to be debugged. Moreover, often the failure is not something that can be easily reproduced in a laboratory environment, so we are left with production, often hardened, systems to debug.
Assuming we have designed our system with the aforementioned scenario in mind, our first debugging choice is usually to extract as much information about the failing system, for example, by obtaining and analyzing a core dump of the misbehaving processes.
In the process of debugging embedded Linux systems, we can use the same toolbox as standard Linux systems. One of these tools enables applications to generate a memory core dump upon crashing. This assumes that we have enough disk space to store the application's entire memory map, and that writing to disk is quick enough that it will not drag the system to a halt.
Once the memory core dump is generated, we use the host's GDB to analyze...
On devices as powerful as the Wandboard, native debugging is also an option for debugging sporadic failures. This recipe will explore the native debugging method.
For native development, Yocto offers the -dev and -sdk target images. We can create -dev and -sdk versions of any image recipe by adding the following image features:
tools-sdk: Adds developing tools to the target image. This is already included in the available -sdk images but not on -dev images.tools-debug: Adds debugging tools to the target image. This is already included in the available -sdk images but not on -dev images.dev-pkgs: Adds development packages with headers and libraries to the target image. This is included in both -sdk and -dev image types available.dbg-pkgs: Adds debug information to all packages in our image.To just add native GDB debugging capabilities to an image such as core-image-minimal, we only need the dbg-pkgs and tools-debug features, which we can add with the following...
When we run a cross compiled GDB in the host that connects to a native gdbserver running on the target, it is referred to as cross debugging. This is the same scenario we saw in the Using the Eclipse IDE recipe in Chapter 4, Application Development, except that Eclipse was using the Target Communications Framework (TCF). Cross debugging has the advantage of not needing debug information on target images, as they are already available in the host.
This recipe will show how to use a cross GDB and gdbserver.
To include gdbserver in your target image, you can use an -sdk image if there is one, or you can add the tools-debug feature to your image by adding the following to your conf/local.conf configuration file:
EXTRA_IMAGE_FEATURES += "tools-debug"
We will now build our images and program them in the target.
The images running on the target and the toolchain's sysroot need to match, and the sysroot needs to contain debug information on shared libraries and executables...
Debugging does not always involve working with source code. Sometimes it is a change in an external factor that is causing the problem.
Strace is a tool that is useful for scenarios where we are looking for problems outside of the binary itself; for example, configuration files, input data, and kernel interfaces. This recipe will explain how to use it.
To include strace in your system, add the following to your conf/local.conf file:
IMAGE_INSTALL_append = " strace"
Strace is also part of the tools-debug image feature, so you can also add it with:
EXTRA_IMAGE_FEATURES += "tools-debug"
Strace is also included in the -sdk images.
Before starting, we will also include pgrep, a process utility that will make our debugging easier by looking up process IDs by name. To do so, add the following to your conf/local.conf configuration file:
IMAGE_INSTALL_append = " procps"
Hardware performance counters are perfect for code optimization, especially in embedded systems with a single workload. They are actively used by a wide range of tracing and profiling tools. This recipe will introduce the Linux performance counters subsystem and show how to use it.
The Linux Kernel Performance Counters Subsystem (LPC), commonly known as linux_perf, is an abstraction interface for different CPU-specific performance measurements. The perf_events subsystem not only exposes hardware performance counters from the CPU, but also kernel software events using the same API. It also allows the mapping of events to processes, although this has a performance overhead. Furthermore, it provides generalized events that are common across architectures.
Events can be categorized into three main groups:
The Linux kernel is continuously being instrumented with static probe points called tracepoints, which when disabled have a negligible overhead. They allow us to record more information than the function tracer we saw in Chapter 2, The BSP Layer. Tracepoints are used by multiple tracing and profiling tools in Yocto.
This recipe will explain how to use and define static tracepoints independently of user space tools.
Static tracepoints can be instrumented using custom kernel modules, and also through the event tracing infrastructure. Enabling any of the tracing features in the kernel will create a /sys/kernel/debug/tracing/ directory; for example, the function tracing feature as explained in the Using the kernel function tracing system in Chapter 2, The BSP Layer.
So before continuing with this recipe, you need to configure the function tracing feature in the Linux kernel as explained before.
Kprobes is a kernel debugging facility that allows us to dynamically break into almost any kernel function (except kprobe itself) to collect debugging and profiling information non-disruptively. Architectures can keep an array of blacklisted functions, which cannot be probed using Kprobes.
Because kprobes can be used to change a function's data and registers, it should only be used in development environments.
There are three types of probes:
kprobes: This is the kernel probe, which can be inserted into any location with more than one kprobe added at a single location, if needed.jprobe: This is the jumper probe inserted at the entry point of a kernel function to provide access to its arguments. Only one jprobe may be added at a given location.kretprobe: This is the return probe and triggers on a function return. Also, only one kretprobe may be added to the same location.They are packaged into a kernel module, with the init function registering the probes and the...
Although dynamic tracing is a very useful feature, custom kernel modules do not have a user-friendly interface. Fortunately, the Linux kernel has been extended with the support of kprobe events, which allow us to set kprobes probes using a debugfs interface.
To make use of this feature, we need to configure our kernel with the CONFIG_KPROBE_EVENT configuration variable.
The debugfs interface adds probes via the /sys/kernel/debug/tracing/kprobe_events file.
For example, to add a kprobe called example_probe to the do_sys_open function, you can execute the following command:
# echo 'p:example_probe do_sys_open dfd=%r0 filename=%r1 flags=%r2
mode=%r3' > /sys/kernel/debug/tracing/kprobe_eventsThe probe will print the function's argument list, according to the function's declaration arguments as seen in the function's definition as follows:
long do_sys_open(int dfd, const char __user *filename, int flags,
umode_t mode);You can then manage...
Tracing and profiling tools are used to increase the performance, efficiency, and quality of both, applications and systems. User space tracing and profiling tools make use of performance counters and static and dynamic tracing functionality that the Linux kernel offers, as we have seen in the previous recipes.
Tracing enables us to log an application's activity so that its behavior can be analyzed, optimized, and corrected.
Yocto offers several tracing tools including:
ftrace kernel subsystem, and kernelshark, a graphical interface to trace-cmd.The perf Linux tool can instrument the Linux kernel with both hardware and software performance counter events as well as static and dynamic kernel trace points. For this, it uses the kernel functionality we have seen in previous recipes, providing a common interface to all of them.
This tool can be used to debug, troubleshoot, optimize, and measure applications, workloads, or the full system, and covers the processor, kernel, and applications. Perf is probably the most complete of the tracing and profiling tools available for a Linux system.
The perf source is part of the Linux kernel. To include perf in your system, add the following to your conf/local.conf file:
IMAGE_INSTALL_append = " perf"
Perf is also part of the tools-profile image feature, so you can also add it with the following:
EXTRA_IMAGE_FEATURES += "tools-profile"
Perf is also included in the -sdk images.
To take the maximum advantage of this tool, we need to have symbols both in user...
SystemTap is a GPLv2 licensed system-wide tool that allows you to gather tracing and profiling data from a running Linux system. The user writes a systemtap script, which is then compiled into a Linux kernel module linked against the same kernel source it is going to run under.
The script sets events and handlers, which are called by the kernel module on the specified events triggering. For this, it uses the kprobes and uprobes (if available) interfaces in the kernel, as we saw in the Using dynamic kernel events recipe before.
To use SystemTap, we need to add it to our target image by adding it specifically, as follows:
IMAGE_INSTALL_append = " systemtap"
Or we can also add it by using the tools-profile image feature, or an -sdk image.
We will also need an SSH server running on the target. This is already available on the -sdk image; otherwise we can add one to our image with the following:
EXTRA_IMAGE_FEATURES += "ssh-server-openssh"
We will also need to compile...
LTTng is a set of dual licensed GPLv2 and LGPL tracing and profiling tools for both applications and kernels. It produces binary trace files in the production optimized Compact Trace Format (CTF), which can then be analyzed by tools such as babeltrace.
To include the different LTTng tools in your system, add the following to your conf/local.conf file:
IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
They are also part of the tools-profile image feature, so you can also add them with the following:
EXTRA_IMAGE_FEATURES += "tools-profile"
These are also included in the -sdk images.
The default Wandboard Linux kernel is already configured to use LTTng, but other platforms might need to enable CONFIG_TRACEPOINTS.
The LTTng command-line tool is the main user interface to LTTng. It can be used to trace both the Linux kernel-using the kernel tracing interfaces we have seen in previous recipes—as well as instrumented user space applications.
There are a few tools available to perform block devices I/O monitoring and profiling.
iotop, which we mentioned in the Exploring Yocto's tracing and profiling tools recipe, gives a general idea of the throughput on a system and a particular process. iostat provides many more statistics regarding CPU usage and device utilization, but does not provide per process details. Finally, blktrace, a GPLv2 licensed tool that monitors specific block devices' I/O at a low level, and can also compute I/O operations per second (IOPS).
This recipe will explain how to use blktrace to trace block devices and blkparse, to convert the traces into human readable format.
Flash filesystems, even if they are presented to the user space as block devices, cannot be profiled with blktrace.
© 2018 Packt Publishing Limited All Rights Reserved
