Chapter 19: Debugging with GDB

Bugs happen. Identifying and fixing them is part of the development process. There are many different techniques for finding and characterizing program defects, including static and dynamic analysis, code review, tracing, profiling, and interactive debugging. I will look at tracers and profilers in the next chapter, but here I want to concentrate on the traditional approach of watching code execution through a debugger, which in our case is the GNU Project Debugger (GDB). GDB is a powerful and flexible tool. You can use it to debug applications, examine the postmortem files (core files) that are created after a program crash, and even step through kernel code.

In this chapter, we will cover the following topics:

• The GNU debugger
• Preparing to debug
• Debugging applications
• Just-in-time debugging
• Debugging forks and threads
• Core files
• GDB user interfaces
• Debugging kernel code

Technical requirements

To follow along with the examples, make sure you have the following:

• Linux-based host system with a minimum of 60 GB of available disk space
• Buildroot 2020.02.9 LTS release
• Yocto 3.1 (Dunfell) LTS release
• Etcher for Linux
• MicroSD card reader and card
• USB to TTL 3.3V serial cable
• Raspberry Pi 4
• 5V 3A USB-C power supply
• Ethernet cable and port for network connectivity
• BeagleBone Black
• 5V 1A DC power supply

You should have already installed the 2020.02.9 LTS release of Buildroot for Chapter 6, Selecting a Build System. If you have not, then refer to the System requirements section of the The Buildroot user manual (https://buildroot.org/downloads/manual/manual.html) before installing Buildroot on your Linux host according to the instructions from Chapter 6.

You should have already installed the 3.1 (Dunfell) LTS release of Yocto for Chapter 6, Selecting a Build System. If you have not, then refer to the Compatible...

The GNU debugger

GDB is a source-level debugger for compiled languages, primarily C and C++, although there is also support for a variety of other languages, such as Go and Objective-C. You should read the notes for the version of GDB you are using to find out the current status of support for the various languages.

The project website is https://www.gnu.org/software/gdb/ and it contains a lot of useful information, including the GDB user manual, Debugging with GDB.

Out of the box, GDB has a command-line user interface that some people find off-putting, although in reality, it is easy to use with a little practice. If command-line interfaces are not to your liking, there are plenty of frontend user interfaces to GDB, and I will describe three of them later in this chapter.

Preparing to debug

You need to compile the code you want to debug with debug symbols. GCC offers two options for this: -g and -ggdb. The latter adds debug information that is specific to GDB, whereas the former generates information in an appropriate format for whichever target operating system you are using, making it the more portable option. In our particular case, the target operating system is always Linux, and it makes little difference whether you use -g or -ggdb. Of more interest is the fact that both options allow you to specify the level of debug information, from 0 to 3:

• 0: This produces no debug information at all and is equivalent to omitting the -g or -ggdb switch.
• 1: This produces minimal information, but includes function names and external variables, which is enough to generate a backtrace.
• 2: This is the default and includes information about local variables and line numbers so that you can perform source-level debugging and single-step through the...

Debugging applications

You can use GDB to debug applications in one of two ways: if you are developing code to run on desktops and servers, or indeed any environment where you compile and run the code on the same machine, it is natural to run GDB natively. However, most embedded development is done using a cross toolchain, and hence you want to debug code running on the device but control it from the cross-development environment, where you have the source code and the tools. I will focus on the latter case, since it is the most likely scenario for embedded developers, but I will also show you how to set up a system for native debugging. I am not going to describe the basics of using GDB here since there are many good references on that topic already, including the GDB user manual and the suggested Further reading section at the end of the chapter.

Remote debugging using gdbserver

The key component for remote debugging is the debug agent, gdbserver, which runs on the target and...

Just-in-time debugging

Sometimes, a program will start to misbehave after it has been running for a while, and you would like to know what it is doing. The GDB attach feature does exactly this. I call it just-in-time debugging. It is available with both native and remote debug sessions.

In the case of remote debugging, you need to find the PID of the process to be debugged and pass it to gdbserver with the --attach option. For example, if the PID is 109, you would type this:

# gdbserver --attach :10000 109
Attached; pid = 109
Listening on port 10000

This forces the process to stop as if it were at a breakpoint, allowing you to start your cross GDB in the normal way and connect to gdbserver. When you are done, you can detach, allowing the program to continue running without the debugger:

(gdb) detach
Detaching from program: /home/chris/MELP/helloworld/helloworld, process 109
Ending remote debugging.

Attaching to a running process by PID is certainly handy, but what about...

Debugging forks and threads

What happens when the program you are debugging forks? Does the debug session follow the parent process or the child? This behavior is controlled by follow-fork-mode, which may be parent or child, with parent being the default. Unfortunately, current versions (10.1) of gdbserver do not support this option, so it only works for native debugging. If you really need to debug the child process while using gdbserver, a workaround is to modify the code so that the child loops on a variable immediately after the fork, giving you the opportunity to attach a new gdbserver session to it and then to set the variable so that it drops out of the loop.

When a thread in a multithreaded process hits a breakpoint, the default behavior is for all threads to halt. In most cases, this is the best thing to do as it allows you to look at static variables without them being changed by the other threads. When you recommence execution of the thread, all the stopped threads start...

Core files

Core files capture the state of a failing program at the point that it terminates. You don't even have to be in the room with a debugger when the bug manifests itself. So, when you see Segmentation fault (core dumped), don't shrug; investigate the core file and extract the goldmine of information in there.

The first observation is that core files are not created by default, but only when the core file resource limit for the process is non-zero. You can change it for the current shell using ulimit -c. To remove all limits on the size of core files, type the following command:

$ ulimit -c unlimited

By default, the core file is named core and is placed in the current working directory of the process, which is the one pointed to by /proc/<PID>/cwd. There are a number of problems with this scheme. Firstly, when looking at a device with several files named core, it is not obvious which program generated each one. Secondly, the current working directory...

GDB user interfaces

GDB is controlled at a low level through the GDB machine interface, GDB/MI, which can be used to wrap GDB in a user interface or as part of a larger program, and it considerably extends the range of options available to you.

In this section, I will describe three that are well suited to debugging embedded targets:
the Terminal User Interface (TUI), the Data Display Debugger (DDD), and Visual
Studio Code.

Terminal User Interface

Terminal User Interface (TUI) is an optional part of the standard GDB package. The main feature is a code window that shows the line of code about to be executed, together with any breakpoints. It is a definite improvement on the list command in command-line mode GDB.

The attraction of TUI is that it just works without any extra setup, and since it is in text mode, it is possible to use over an SSH terminal session, for example, when running gdb natively on a target. Most cross toolchains configure GDB with TUI. Simply add -tui...

Debugging kernel code

You can use kgdb for source-level debugging, in a manner similar to remote debugging with gdbserver. There is also a self-hosted kernel debugger, kdb, that is handy for lighter-weight tasks such as seeing whether an instruction is executed and getting the backtrace to find out how it got there. Finally, there are kernel Oops messages and panics, which tell you a lot about the cause of a kernel exception.

Debugging kernel code with kgdb

When looking at kernel code using a source debugger, you must remember that the kernel is a complex system, with real-time behaviors. Don't expect debugging to be as easy as it is for applications. Stepping through code that changes the memory mapping or switches context is likely to produce odd results.

kgdb is the name given to the kernel GDB stubs that have been part of mainline Linux for many years now. There is a user manual in the kernel DocBook, and you can find an online version at https://www.kernel.org/doc...

Summary

Knowing how to use GDB for interactive debugging is a useful tool in the embedded system developer's tool chest. It is a stable, well-documented, and well-known entity. It has the ability to debug remotely by placing an agent on the target, be it gdbserver for applications or kgdb for kernel code, and although the default command-line user interface takes a while to get used to, there are many alternative frontends. The three I mentioned were TUI, DDD, and Visual Studio Code. Eclipse is another popular frontend that supports debugging with GDB by way of the CDT plugin. I will refer you to the references in the Further reading section for information on how to configure CDT to work with a cross toolchain and connect to a remote device.

A second and equally important way to approach debugging is to collect crash reports and analyze them offline. In this category, we looked at application core dumps and kernel Oops messages.

However, this is only one way of identifying...

Further reading

The following resources have further information on the topics introduced in this chapter:

• The Art of Debugging with GDB, DDD, and Eclipse, by Norman Matloff and Peter
  Jay Salzman
• GDB Pocket Reference, by Arnold Robbins
• Python Interpreter in GNU Debugger, by crazyguitar: https://www.pythonsheets.com/appendix/python-gdb.html
• Extending GDB with Python, by Lisa Roach: https://www.youtube.com/watch?v=xt9v5t4_zvE
• Cross-compiling with CMake and VS Code, by Enes ÖZTÜRK: https://enes-ozturk.medium.com/cross-compiling-with-cmake-and-vscode-9ca4976fdd1
• Remote Debugging with GDB, by Enes ÖZTÜRK: https://enes-ozturk.medium.com/remote-debugging-with-gdb-b4b0ca45b8c1
• Getting to grips with Eclipse: cross compiling: https://2net.co.uk/tutorial/eclipse-cross-compile
• Getting to grips with Eclipse: remote access and debugging: https://2net.co.uk/tutorial/eclipse-rse