Troubleshooting Concurrency Issues

All non-trivial programs have bugs. When you realize that there are some anomalies in a program, starting a debugging session is usually not the first thing you should do. This chapter is about some of the techniques you can use for troubleshooting without using a debugger. You may find, especially when dealing with concurrent programs, that debuggers sometimes do not offer much help and the solution relies on careful code reading, log reading, and understanding stack traces.

In this chapter, we will explore the following:

• How to read stack traces
• How to detect failures using additional code to monitor program behavior and sometimes heal the program
• Debugging anomalies using timeouts and stack traces

Technical Requirements

The source code for this particular chapter is available on GitHub at https://github.com/PacktPublishing/Effective-Concurrency-in-Go/tree/main/chapter10.

Reading stack traces

If you are lucky, your program panics when something goes wrong, and prints out lots of diagnostic information. You would be lucky because if you have the output of a panicked program, you can usually figure out what went wrong by just looking at it together with the source code. So, let’s take a look at some stack traces. The first example is a deadlock-prone implementation of the dining philosophers problem, with just two philosophers:

func philosopher(firstFork, secondFork *sync.Mutex) {
for {
    firstFork.Lock()
    secondFork.Lock() // line: 10
    secondFork.Unlock()
    firstFork.Unlock()
  }
}
func main() {
    forks := [2]sync.Mutex{}
    go philosopher(&forks[1], &forks[0]) // line: 18
    go philosopher(&forks[0], &forks[1]) // line: 19
    select {} // line...

Detecting failures and healing

Most software systems will fail despite the efforts spent on testing them. This suggests there are limits to what can be achieved by testing. These limitations stem from several facts about non-trivial systems. Any non-trivial system interacts with its environment, and it is simply not practical (and in many cases, not possible) to enumerate all possible environments in which the system will run. Also, it is usually possible to test a system to make sure it behaves as expected, but it is much harder to develop tests to make sure the system does not behave unexpectedly. Concurrency adds additional complexities: a program that was successfully tested for a particular scenario may fail for the same scenario when put into production.

In other words, no matter how much you test your programs, all sufficiently complex programs will eventually fail. So, it makes sense to architect systems for graceful failure and quick recovery. Part of this architecture...

Debugging anomalies

Concurrent algorithms have a way of working when observed and failing when not. Many times, a program that runs just fine in a debugger fails mysteriously in production environments. Sometimes, such failures come with a stack trace, and you can track it back to why it happened. But sometimes, failures are much more subtle with no clear indication of what went wrong.

Consider the monitor in the previous section. You might want to find out why SlowFunc hangs. You cannot really run it in a debugger and step through the code because you simply have no control over which invocation of the function hangs. But what you can do is print a stack trace when it happens. This is the nature of most anomalies in concurrent programs: you don’t know when it is going to happen, but you can usually tell that it did. So, you can print all sorts of diagnostic information to backtrack how the program got there. For instance, you can print the stack trace when the monitor raises...

Summary

This chapter showed some techniques that can be useful to troubleshoot concurrent programs. The key idea when dealing with such programs is that you can usually tell when something bad happened only after it happens. So, adding additional code that generates alerts with diagnostic information can be a lifesaver. Many times, this is simply additional logging or Printf statements and panics (a.k.a Poor Man’s Debugger). Add such code to your programs and keep that code alive in production. Immediate failure is almost always better than incorrect computation.

Further reading

The Go development environment comes with numerous tools for diagnostics:

• The Go unit testing framework: https://pkg.go.dev/testing
• The runtime/pprof package makes your program’s internals available to monitoring tools: https://pkg.go.dev/runtime/pprof
• The Go Race Detector ensures your code is race-free: https://go.dev/blog/race-detector
• The Go Profiler is an indispensable tool for analyzing leaks and performance bottlenecks: https://go.dev/blog/pprof