When we refer to a “process” in Linux, we’re referring to the operating system’s internal model of what exactly a running program is. Linux needs a general abstraction that works for all programs, which can encapsulate the things the operating system cares about. A process is that abstraction, and it enables the OS to track some of the important context around programs that are executing; namely:

• Memory usage
• Processor time used
• Other system resource usage (disk access, network usage)
• Communication between processes
• Related processes that a program starts, for example, firing off a shell command

You can get a listing of all system processes (at least the ones your user is allowed to see) by running the ps program with the aux flags:

Figure 2.1: List of system processes

We’ll cover the attributes most relevant to your work as a developer in this chapter.

What is a Linux process made of?

From the perspective of the operating system, a “process” is simply a data structure that makes it easy to access information like:

• Process ID (PID in the ps output above). PID 1 is the init system – the original parent of all other processes, which bootstraps the system. The kernel starts this as one of the first things it does after starting to execute. When a process is created, it gets the next available process ID, in sequential order. Because it is so important to the normal functioning of the operating system, init cannot be killed, even by the root user. Different Unix operating systems use different init systems – for example, most Linux distributions use systemd, while macOS uses launchd, and many other Unixes use SysV. Regardless of the specific implementation, we’ll refer to this process by the name of the role it fills: “init.”

  Note

  In containers, processes are namespaced – in the “real” environment, all container processes might be PID 3210, while that single PID maps to lots of processes (1..n, where n is the number of running processes in the container). You can see this from outside but not inside the container.

• Parent Process PID (PPID). Each process is spawned by a parent. If the parent process dies while the child is alive, the child becomes an “orphan.” Orphaned processes are re-parented to init (PID 1).
• Status (STAT in the ps output above). man ps will show you an overview:
  • D – uninterruptible sleep (usually IO)
  • I – idle kernel thread
  • R – running or runnable (on run queue)
  • S – interruptible sleep (waiting for an event to complete)
  • T – stopped by job control signal
  • t – stopped by debugger during tracing
  • X – dead (should never be seen)
  • Z – defunct (“zombie”) process, terminated but not reaped by its parent
• Priority status (“niceness” – does this process allow other processes to take priority over it?).
• A process Owner (USER in the ps output above); the effective user ID.
• Effective Group ID (EGID), which is used.
• An address map of the process’s memory space.
• Resource usage – open files, network ports, and other resources the process is using (VSZ and RSS for memory usage in the ps output above).

(Citation: from the Unix and Linux System Administration Handbook, 5th edition, p.91.)

Let’s take a closer look at a few of the process attributes that are most important for developers and occasional troubleshooters to understand.

Process ID (PID)

Each process is uniquely identifiable by its process ID, which is just a unique integer that is assigned to a process when it starts. Much like a relational database with IDs that uniquely identify each row of data, the Linux operating system keeps track of each process by its PID.

A PID is by far the most useful label for you to use when interacting with processes.

Effective User ID (EUID) and Effective Group ID (EGID)

These determine which system user and group your process is running as. Together, user and group permissions determine what a process is allowed to do on the system.

As you’ll see in Chapter 5, Introducing Files, files have user and group ownership set on them, which determines who their permissions apply to. If a file’s ownership and permissions are essentially a lock, then a process with the right user/group permissions is like a key that opens the lock and allows access to the file. We’ll dive deeper into this later, when we talk about permissions.

Environment variables

You’ve probably used environment variables in your applications – they’re a way for the operating system environment that launches your process to pass in data that the process needs. This commonly includes things like configuration directives (LOG_DEBUG=1) and secret keys (AWS_SECRET_KEY), and every programming language has some way to read them out from the context of the program.

For example, this Python script gets the user’s home directory from the HOME environment variable, and then prints it:

import os
home_dir = os.environ['HOME']
print("The home directory for this user is", home_dir)

In my case, running this program in the python3 REPL on a Linux machine results in the following output:

The home directory for this user is /home/dcohen 

Working directory

A process has a “current working directory,” just like your shell (which is just a process, anyway). Typing pwd in your shell prints its current working directory, and every process has a working directory. The working directory for a process can change, so don’t rely on it too much.

This concludes our overview of the process attributes that you should know about. In the next section, we’ll step away from theory and look at some commands you can use to start working with processes right away.

Here are some of the commands you’ll use most often:

• ps – Shows processes on the system; you saw an example of this command earlier in the chapter. Flags modify which process attributes are displayed as columns. This command is usually used with filters to control how much output you get, for example, (ps aux | head –n 10) to cut your output down to just the top 10 lines. A few more useful tricks:
  • ps –eLf shows thread information for processes
  • ps -ejH is useful for seeing the relationships between parent and child processes visually (children are indented under their parents)

  

  Figure 2.2: Examples of outputs of the ps command with flags

• pgrep – Find process IDs by name. Can use regular expressions.

Figure 2.3: Examples of outputs of the pgrep command with flags

• top – An interactive program that polls all processes (once a second, by default) and outputs a sorted list of resource usage (you can configure what it sorts by). Also displays total system resource usage. Press Q or use Ctrl + C to quit. You’ll see an example of this command’s output later in this chapter.
• iotop – Like top, but for disk IO. Extremely useful for finding IO-hungry processes. Not installed on all systems by default, but available via most package managers.

Figure 2.4: Example of output of the iotop command

• nethogs – Like top, but for network IO. Groups network usage by process, which is incredibly convenient. Available via most package managers.

• kill – Allows users to send signals to processes, usually to stop them or make them re-read their configuration files. We’ll explain signals and kill command usage later in this chapter.

This marks the beginning of the “advanced” section of this chapter. While you don’t need to master all the concepts in this section to work effectively with Linux processes, they can be extremely helpful. If you have a few extra minutes, we recommend at least familiarizing yourself with each one.

Signals

How does systemctl tell your web server to re-read its configuration files? How can you politely ask a process to shut down cleanly? And how can you kill a malfunctioning process immediately, because it’s bringing your production application to its knees?

In Unix and Linux, all of this is done with signals. Signals are numerical messages that can be sent between programs. They’re a way for processes to communicate with each other and with the operating system, allowing processes to send and receive specific messages.

These messages can be used to communicate a variety of things to a process, for example, indicating that a particular event has happened or that a specific action or response is required.

Practical uses of signals

Let’s look at a few examples of the practical value that the signal mechanism enables. Signals can be used to implement inter-process communication; for example, one process can send a signal to another process indicating that it’s finished with a particular task and that the other process can now start working. This allows processes to coordinate their actions and work together in a smooth and efficient manner, much like execution threads in programming languages (but without the associated memory sharing).

Another common application of process signals is to handle program errors. For example, a process can be designed to catch the SIGSEGV signal, which indicates a segmentation fault. When a process receives this signal, it can trap that signal and then take action to log the error, dump core for debugging purposes, or clean up any resources that were being used before shutting down gracefully.

Process signals can also be used to implement graceful shutdowns. For example, when a system is shutting down, a signal can be sent to all processes to give them a chance to save their state and clean up any resources they were using, via “trapping” signals.

Trapping

Many of the signals can be “trapped” by the processes that receive them: this is essentially the same idea as catching and handling an error in a programming language.

If the receiving process has a handler function for the signal that’s being sent, then that handler function is run. That’s how programs re-read their configuration without restarting, and finish their database writes and close their file handles after receiving the shutdown signal.

The kill command

However, it’s not just processes that communicate via signals: the frighteningly named (and, technically speaking, incorrectly named) kill is a program that allows users to send signals to processes, too.

One of the most common uses of user-sent processes via the kill command is to interrupt a process that is no longer responding. For example, if a process is stuck in an infinite loop, a “kill” signal can be sent to force it to stop.

The kill command allows you to send a signal to a process by specifying its PID. If the process you’d like to terminate has PID 2600, you’d run:

kill 2600

This command would send signal 15 (SIGTERM, or “terminate”) to the process, which would then have a chance to trap the signal and shut down cleanly.

If you don’t want to send the default signal 15, you can specify the signal you’d like to send with a dash; to send a SIGHUP to the same process, you’d run:

kill –1 2600

Running man signal will give you a list of signals that you can send:

Figure 2.6: Example of output of the man signal command

It pays – sometimes quite literally, in engineering interviews – to be familiar with a few of these:

• SIGHUP (1) – “hangup”: interpreted by many applications – for example, nginx – as “re-read your configuration because I’ve made changes to it.”
• SIGINT (2) – “interrupt”: often interpreted the same as SIGTERM - “please shut down cleanly.”
• SIGTERM (15) – “terminate”: nicely asks a process to shut down.
• SIGUSR1 (30) and SIGUSR2 (31) are sometimes used for application-defined messaging For example, SIGUSR1 asks nginx to re-open the log files it’s writing to, which is useful if you’ve just rotated them.
• SIGKILL (9) – SIGKILL cannot be trapped and handled by processes. If this signal is sent to a program, the operating system will kill that program immediately. Any cleanup code, like flushing writes or safe shutdown, is not performed, so this is generally a last resort, since it could lead to data corruption.

If you want to explore Linux a bit deeper, feel free to poke around the /proc directory. That’s definitely beyond the basics, but it’s a directory that contains a filesystem subtree for every process, where live information about the processes is looked up as you read those files.

/proc

In practice, this knowledge can come in handy during troubleshooting when you’ve identified a misbehaving (or mysterious) process and want to know exactly what it’s doing in real time.

You can learn a lot about a process by poking around in its /proc subdirectory and casually googling.

Many of the tools we show you in this chapter actually use /proc to gather process information, and only show you a subset of what’s there. If you want to see everything and do the filtering yourself, /proc is the place to look.

lsof – show file handles that a process has open

The lsof command shows all files that a process has opened for reading and writing. This is useful because it only takes one small bug for a program to leak file handles (internal references to files that it has requested access to). This can lead to resource usage issues, file corruption, and a long list of strange behavior.

Thankfully, getting a list of files that a process has open is easy. Just run lsof and pass the –p flag with a PID (you’ll usually have to run this as root). This will return the list of files that the process (in this case, with PID 1589) has open:

  ~ lsof -p 1589

Figure 2.7: Example of list of files opened by the 1589 process using the lsof -p 1589 command

The above is the output for an nginx web server process. The first line shows you the current working directory for the process: in this case, the root directory (/). You can also see that it has file handles open on its own binary (/usr/sbin/nginx) and various libraries in /usr/lib/.

Further down, you might notice a few more interesting filepaths:

Figure 2.8: Further opened files of the 1589 process

This listing includes the log files nginx is writing to, and socket files (Unix, IPv4, and IPv6) that it’s reading and writing to. In Unix and Linux, network sockets are just a special kind of file, which makes it easy to use the same core toolset across a wide variety of use cases – tools that work with files are extremely powerful in an environment where almost everything is represented as a file.

Inheritance

Except for the very first process, init (PID 1), all processes are created by a parent process, which essentially makes a copy of itself and then “forks” (splits) that copy off. When a process is forked, it typically inherits its parent’s permissions, environment variables, and other attributes.

Although this default behavior can be prevented and changed, it’s a bit of a security risk: software that you run manually receives the permissions of your current user (or even root privileges, if you use sudo). All child processes that might be created by that process – for example, during installation, compilation, and so on – inherit those permissions.

Imagine a web server process that was started with root privileges (so it could bind to a network port) and environment variables containing cloud authentication keys (so it could grab data from the cloud). When this main process forks off a child process that needs neither root privileges nor sensitive environment variables, it’s an unnecessary security risk to pass those along to the child. As a result, dropping privileges and clearing environment variables is a common pattern in services spawning child processes.

From a security perspective, it is important to keep this in mind to prevent situations where information such as passwords or access to sensitive files could be leaked. While it is outside the scope of this book to go into details of how to avoid this, it’s important to be aware of this if you’re writing software that’s going to run on Linux systems.

Let’s look at an example troubleshooting session. All we know is that one specific Linux server is running extremely slowly.

To begin with, we want to see what’s happening on the system. You just learned that you can see a live view of processes running on a system by running the interactive top command. Let’s try that now.

Figure 2.9: Example of output of the top command

By default, the top command sorts processes by CPU usage, so we can simply look at the first listed process to find the offending one. Indeed, the top process is using 94% of one CPU’s available processing time.

As a result of running top, we’ve gotten a few useful pieces of information:

• The problem is CPU usage, as opposed to some other kind of resource contention.
• The offending process is PID 1763, and the command being run (listed in the COMMAND column) is bzip2, which is a compression program.

We determine that this bzip2 process doesn’t need to be running here, and we decide to stop it. Using the kill command, we ask the process to terminate:

kill 1763

After waiting a few seconds, we check to see if this (or any other) bzip2 process is running:

pgrep bzip2

Unfortunately, we see that the same PID is still running. It’s time to get serious:

kill –9 1763

This orders the operating system to kill the process without allowing the process to trap (and potentially ignore) the signal. A SIGKILL (signal #9) simply kills the process where it stands.

Now that you’ve killed the offending process, the server is running smoothly again and you can start tracking down the developer who thought it was a good idea to compress large source directories on this machine.

In this example, we followed the most common systems troubleshooting pattern in existence:

1. We looked at resource usage (via top in this example). This can be any of the other tools we discussed, depending on which resource is the one being exhausted.
2. We found a PID to investigate.
3. We acted on that process. In this example, no further investigation was necessary and we sent a signal, asking it to shut down (15, SIGTERM).

In this chapter, we took a close look at the process abstraction that Linux wraps around executing programs. You’ve seen the common components that all processes have and learned the basic commands you need to find and inspect running processes. With these tools, you’ll be able to identify when a process is misbehaving, and more importantly, which process is misbehaving.

Learn more on Discord

To join the Discord community for this book – where you can share feedback, ask questions to the author, and learn about new releases – follow the QR code below:

https://packt.link/SecNet