Linux Kernel Programming

By Kaiwan N Billimoria
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Kernel Workspace Setup

About this book

Linux Kernel Programming is a comprehensive introduction for those new to Linux kernel and module development. This easy-to-follow guide will have you up and running with writing kernel code in next-to-no time. This book uses the latest 5.4 Long-Term Support (LTS) Linux kernel, which will be maintained from November 2019 through to December 2025. By working with the 5.4 LTS kernel throughout the book, you can be confident that your knowledge will continue to be valid for years to come.

This Linux book begins by showing you how to build the kernel from the source. Next, you’ll learn how to write your first kernel module using the powerful Loadable Kernel Module (LKM) framework. The book then covers key kernel internals topics including Linux kernel architecture, memory management, and CPU scheduling. Next, you’ll delve into the fairly complex topic of concurrency within the kernel, understand the issues it can cause, and learn how they can be addressed with various locking technologies (mutexes, spinlocks, atomic, and refcount operators). You’ll also benefit from more advanced material on cache effects, a primer on lock-free techniques within the kernel, deadlock avoidance (with lockdep), and kernel lock debugging techniques.

By the end of this kernel book, you’ll have a detailed understanding of the fundamentals of writing Linux kernel module code for real-world projects and products.

Publication date:
March 2021

Building the 5.x Linux Kernel from Source - Part 1

Building the Linux kernel from source code is an interesting way to begin your kernel development journey! Rest assured, it's a long and arduous journey, but that's the fun of it, right? The kernel build topic itself is large enough to merit being divided into two chapters, this one and the next.

The primary purpose of this chapter and the next is to describe in detail how exactly you can build a Linux kernel from scratch, from source code. In this chapter, you will initially learn how to download a stable vanilla Linux kernel source tree onto a guest Linux Virtual Machine (VM) (by vanilla kernel, we mean the plain and regular default kernel source code released by the Linux kernel community on its repository, Next, we will learn a little bit about the layout of the kernel source code...


Technical requirements 

I assume that you have gone through Chapter 1Kernel Workspace Setup, and have appropriately prepared a guest VM running Ubuntu 18.04 LTS (or CentOS 8, or later stable releases of these distributions) and installed all the required packages. If not, I highly recommend you do this first.

To get the most out of this book, I strongly recommend you first set up the workspace environment, including cloning this book's GitHub repository ( for the code, and work on it in a hands-on fashion.


Preliminaries for the kernel build

It's important to understand a few things right from the outset that will help you as we proceed on our journey of building and working with a Linux kernel. Firstly, the Linux kernel and its sister projects are completely decentralized - it's a virtual, online open-source community! The closest we come to an office is this: stewardship of the Linux kernel (as well as several dozen related projects) is in the capable hands of the Linux Foundation (; further, it manages the Linux Kernel Organization, a private foundation that distributes the Linux kernel to the public for no charge (

Some of the key points we discuss in this section includes the following:

  • The kernel release, or version number nomenclature
  • The typical kernel development workflow
  • The existence of different types of kernel source trees within the repository

With this information...


Kernel release nomenclature

To see the kernel version number, simply run uname -r on your shell. How do you precisely interpret the output of uname -r? On our Ubuntu distribution version 18.04 LTS guest VM, we run uname(1), passing the -r option switch to display just the current kernel release or version:

$ uname -r
Of course, by the time you read this, the Ubuntu 18.04 LTS kernel has certainly been upgraded to a later release; that's perfectly normal. The 5.0.0-36-generic kernel was the one I encountered with the Ubuntu 18.04.3 LTS at the time of writing this chapter.

The modern Linux kernel release number nomenclature is as follows:


This is also often written or described as w.x[.y][-z].

The square brackets around the patchlevel and EXTRAVERSION components indicate that they are optional. The following table summarizes the meaning of the components of the release number:

Release # component Meaning Example numbers...

Kernel development workflow – the basics

Here, we provide a brief overview of the typical kernel development workflow. Anyone like you who is interested in kernel development should at least minimally understand the process.

A detailed description can be found within the kernel documentation here:

A common misconception, especially in its baby years, was that the Linux kernel is developed in a very ad hoc fashion. This is not true at all! The kernel development process has evolved to become a (mostly) well-oiled system with a thoroughly documented process and expectation of what a kernel contributor should know in order to use it well. I refer you to the preceding link for the complete details.

In order for us to take a peek into a typical development cycle, let's assume we have the latest mainline Linux Git kernel tree cloned on to our system.

The details regarding the use...

Types of kernel source trees

There are several types of Linux kernel source trees. The key one is the Long Term Support (LTS) kernel. Okay, so what exactly is an LTS release kernel? It’s simply a "special" release in the sense that the kernel maintainers will continue to backport important bug and security fixes upon it (well, security issues are typically nothing but bugs), until a given EOL date.

The "life" of an LTS kernel will usually be a minimum of 2 years, and it can go for several more (it's extended at times). The 5.4 LTS kernel that we will use throughout this book is the 20th LTS kernel and has a lifespan of just over 6 years – from November 2019 to December 2025.

There are several types of release kernels in the repository. However, here, we mention an incomplete list, ordered from least to most stable (thus, their life, from shortest to longest time span):

  • -next trees: This is indeed the bleeding edge, subsystem...

Steps to build the kernel from source

As a convenient and quick reference, the following are the key steps required to build a Linux kernel from source. As the explanation for each of them is pretty detailed, you can refer back to this summary to see the bigger picture. The steps are as follows:

  1. Obtain a Linux kernel source tree through either of the following options:
    • Downloading a specific kernel source as a compressed file
    • Cloning a (kernel) Git tree
  2. Extract the kernel source tree into some location in your home directory (skip this step if you obtained a kernel by cloning a Git tree).
  3. Configuration: Select the kernel support options as required for the new kernel,
    make [x|g|menu]config, with make menuconfig being the preferred way.
  4. Build the kernel's loadable modules and any Device Tree Blobs (DTBs) with make [-j'n'] all. This builds the compressed kernel image (arch/<arch>/boot/[b|z|u]image), the uncompressed kernel image (vmlinux), System...

Step 1 – obtaining a Linux kernel source tree

In this section, we will see two broad ways in which you can obtain a Linux kernel source tree:

  • By downloading and extracting a specific kernel source tree from the Linux kernel public repository (
  • By cloning Linus Torvalds' source tree (or others') – for example, the linux-next Git tree

But how do you decide which approach to use? For the majority of developers like you working on a project or product, the decision has already been made – the project uses a very specific Linux kernel version. You will thus download that particular kernel source tree, quite possibly apply project-specific patches to it if required, and use it.

For folks whose intention is to contribute or "upstream" code to the mainline kernel, the second approach – cloning the Git tree – is the way to go for you. (Of course, there's more to it; we described some details in the&...


Downloading a specific kernel tree

Firstly, where is the kernel source code? The short answer is that it's on the public kernel repository server visible at The home page of this site displays the latest stable Linux kernel version, as well as the latest longterm and linux-next releases (the following screenshot shows the site as of 29 November 2019. It shows dates in the well-known yyyy-mm-dd format):

Figure 2.2 – The site (as of 29 November 2019)
A quick reminder: we also provide a PDF file that has the full-color images of the screenshots/diagrams used in this book. You can download it here:

There are many ways to download a (compressed) kernel source file. Let's look at two of them:

  • An interactive, and perhaps simplest way, is to visit the preceding website and simply click on the appropriate tarball link. The browser will download the image...

Cloning a Git tree

For a developer like you working on and looking to contribute code upstream, you must work on the very latest version of the Linux kernel code base. Well, there are fine gradations of the latest version within the kernel community. As mentioned earlier, the linux-next tree, and some specific branch or tag within it, is the one to work on for this purpose.

In this book, though, we do not intend to delve into the gory details of setting up a linux-next tree. This process is already very well documented and we would prefer not to merely repeat instructions (see the Further reading section for detailed links). The detailed page on how exactly you should clone a linux-next tree is here: Working with linux-next,, and, as mentioned there, the linux-next tree, is the holding area for patches aimed at the next...


Step 2 – extracting the kernel source tree

As mentioned earlier, this section is meant for those of you who have downloaded a particular Linux kernel from the repository, https://www.kernel.organd aim to build it. In this book, we use the 5.4 LTS kernel release. On the other hand, if you have performed git clone on the mainline Linux Git tree, as shown in the immediately preceding section, you can safely skip this section and move on to the next one on kernel configuration.

Now that the download is done, let's proceed further. The next step is to extract the kernel source tree – remember, it's a tar-ed and compressed (typically .tar.xz) file.

We assume that, as shown in detail earlier in this chapter, you have by now downloaded the Linux kernel version 5.4 code base as a compressed file (into the ~/Downloads directory):

$ cd ~/Downloads ; ls -lh linux-5.4.tar.xz
-rw-rw-r-- 1 llkd llkd 105M Nov 26 08:04 linux-5.4.tar.xz

The simple way to extract...


A brief tour of the kernel source tree

The kernel source code is now available on your system! Cool, let's take a quick look at it:

Figure 2.3 – The root of the 5.4 Linux kernel source tree

Great! How big is it? A quick du -m . in the root of the kernel source tree reveals that this particular kernel source tree (recall, it's version 5.4) is a little over 1,000 MB in size – almost a gigabyte!

FYI, the Linux kernel has grown to be big and is getting bigger in terms of Source Lines Of Code (SLOCs). Current estimates are well over 20 million SLOCs. Of course, do realize that not all of this code will get compiled when building a kernel.

How do we know which version exactly of the Linux kernel this code is by just looking at the source? That's easy, one quick way is to just check out the first few lines of the project's Makefile. Incidentally, the kernel uses Makefile's all over the place; most directories have one. We...


Step 3 – configuring the Linux kernel

Configuring the new kernel is perhaps the most critical step in the kernel build process. One of the many reasons Linux is a critically acclaimed OS is its versatility. It's a common misconception to think that there is a separate Linux kernel code base for an (enterprise-class) server, a data center, a workstation, and a tiny embedded Linux device – no, they all use the very same unified Linux kernel source! Thus, carefully configuring the kernel for a particular use case (server, desktop, embedded, or hybrid/custom) is a powerful feature and a requirement. This is precisely what we are delving into here.

Do carry out this kernel configuration step regardless. Even if you feel you do not require any changes to the existing (or default) config, it's very important to run this step at least once as part of the build process. Otherwise, certain headers that are auto-generated here will be missing and cause...

Understanding the kbuild build system

The infrastructure that the Linux kernel uses to configure and build the kernel is known as the kbuild system. Without delving into the gory details, the kbuild system ties together the complex kernel configuration and build process via four key components:

  • The CONFIG_FOO symbols
  • The menu specification file(s), called Kconfig
  • The Makefile(s)
  • The overall kernel config file itself

The purpose of these components is summarized as follows:

Kbuild component Purpose in brief

Config symbol: CONFIG_FOO

Every kernel configurable FOO is represented by a CONFIG_FOO macro. Depending on the user's choice, the macro will resolve to one of y, m, or n:

- y=yes: Implying to build the feature into the kernel image itself


- m=module: Implying to build it as a separate object, a kernel module


- n=no: Implying not to build the feature

Note that CONFIG_FOO is an alphanumeric string (as we will soon see, you can look up the precise...

Arriving at a default configuration

So, how do you decide the initial kernel configuration to begin with? Several techniques exist; a few common ones are as follows:

  • Don't specify anything; the kbuild system will pull in a default kernel configuration.
  • Use the existing distribution's kernel configuration.
  • Build a custom configuration based on the kernel modules currently loaded in memory.

The first approach has the benefit of simplicity. The kernel will handle the details, giving you a default configuration. The downside is that the default config is really pretty large (here, we mean with reference to building Linux for an x86-based desktop or server-type system) – a huge number of options are turned on, just in case, which can make the build time very long and kernel image size very large. Of course, you are then expected to manually configure the kernel to the desired settings.

This brings up the question, where is the default kernel config stored? The kbuild...


Obtaining a good starting point for kernel configuration

This brings us to a really important point: playing around with the kernel configuration is okay to do as a learning exercise (as we do here), but for a production system, it's really critical that you use a proven – known, tested, and working – kernel configuration.

Here, to help you understand the nuances of selecting a valid starting point for kernel configuration, we will see three approaches to obtaining a starting point for kernel configuration that (we hope) are typical:

  • First, the approach to follow for a typical small embedded Linux system
  • Next, an approach where you emulate the distribution's configuration
  • Finally, an approach where you base the kernel configuration on the existing (or another) system's kernel modules (the localmodconfig approach)

Let's examine each of these approaches in a bit more detail.


Kernel config for typical embedded Linux systems

The typical target system for using this approach is a small embedded Linux system. The goal here is to begin with a proven – a known, tested, and working – kernel configuration for our embedded Linux project. Well, how exactly can we achieve this?

Interestingly, the kernel code base itself provides known, tested, and working kernel configuration files for various hardware platforms. We merely have to select the one that matches (or is the nearest match to) our embedded target board. These kernel config files are present within the kernel source tree in the arch/<arch>/configs/ directory. The config files are in the format <platform-name>_defconfig. A quick peek is in order; see the following screenshot showing the command ls arch/arm/configs being performed on the v5.4 Linux kernel code base:

Figure 2.4 – The contents of arch/arm/configs on the 5.4 Linux kernel

Thus, for example...


Kernel config using distribution config as a starting point

The typical target system for using this approach is a desktop or server Linux system.

Moving along, this second approach is also quick:

cp /boot/config-5.0.0-36-generic ${LLKD_KSRC}/.config

Here, we simply copy the existing Linux distribution's (here, it's our Ubuntu 18.04.3 LTS guest VM) config file into the .config file in the root of the kernel source tree, of course, thereby making the distribution config the starting point, which can then be further edited (a more generic command: cp /boot/config-$(uname -r) ${LLKD_KSRC}/.config).


Tuned kernel config via the localmodconfig approach

The typical target system for using this approach is a desktop or server Linux system. 

This third approach we consider is a good one to use when the goal is to begin with a kernel config that is based on your existing system and is thus (usually) relatively compact compared to the typical default config on a desktop or server Linux system. Here, we provide the kbuild system with a snapshot of the kernel modules currently running on the system by simply redirecting the output of lsmod(8) into a temporary file, and then providing that file to the build. This can be achieved as follows:

lsmod > /tmp/
make LSMOD=/tmp/ localmodconfig

The lsmod(8) utility simply lists all the kernel modules currently residing in system (kernel) memory. We will see (a lot) more on this in Chapter 4, Writing Your First Kernel Module – LKMs Part 1. We save its output in a temporary file, which we pass within the...


Getting started with the localmodconfig approach

Now, let's quickly get started on creating a base kernel configuration for our new kernel by using the third approach we discussed previously – the localmodconfig technique. As mentioned, this existing kernel modules-only approach is a good one when the goal is to obtain a starting point for kernel config on an x86-based system by keeping it relatively small and thus make the build quicker as well.

Don't forget: the kernel configuration being performed right now is appropriate for your typical x86-based desktop/server systems. For embedded targets, the approach is different (as seen in the Kernel config for typical embedded Linux systems section). We further cover this practically in Chapter 3, Building the 5.x Linux Kernel from Source - Part 2, under the Kernel build for the Raspberry Pi section.

As described previously, first obtain a snapshot of the currently loaded kernel modules,...


Tuning our kernel configuration via the make menuconfig UI

Okay, great, we now have an initial kernel config file (.config) generated for us via the localmodconfig Makefile target, as shown in detail in the previous section, which is a good starting point. Now, we want to further examine and fine-tune our kernel's configuration. One way to do this – in fact, the recommended way – is via the menuconfig Makefile target. This target has the kbuild system generate a pretty sophisticated (C-based) program executable (scripts/kconfig/mconf), which presents to the end user a neat menu-based UI. In the following code block, when we invoke the command for the first time, the kbuild system builds the mconf executable and invokes it:

$ make menuconfig
UPD scripts/kconfig/.mconf-cfg
HOSTCC scripts/kconfig/mconf.o
HOSTCC scripts/kconfig/lxdialog/checklist.o
HOSTCC scripts/kconfig/lxdialog/inputbox.o
HOSTCC scripts/kconfig/lxdialog/menubox.o
HOSTCC scripts/kconfig/lxdialog...

Sample usage of the make menuconfig UI

To get a feel for using the kbuild menu system via the convenient menuconfig target, let's step through the process to navigate to the tristate menu item called Kernel .config support. It will be off by default, so let's turn it on; that is, let's make it y, built into the kernel image. We can find it under the General Setup main menu item on the home screen.

What exactly does turning this feature on achieve? When turned on to y (or, of course, if made to M, then a kernel module will become available, and once it's loaded up), then the currently running kernel's configuration settings can be looked up at any time in two ways:

  • By running the scripts/extract-ikconfig script
  • By directly reading the content of the /proc/config.gz pseudo-file (of course, it's gzip(1)-compressed; first uncompress it, and then read it)

As a learning exercise, we will now learn how to configure our 5.4 Linux kernel...


More on kbuild

The creation of, or edits to, the .config file within the root of the kernel source tree via make menuconfig or other methods is not the final step in how the kbuild system works with the configuration. No, it now proceeds to internally invoke a target called syncconfig, which was earlier (mis)named silentoldconfig. This target has kbuild generate a few header files that are further used in the setup to build the kernel. These files include some meta headers under include/config, as well as the include/generated/autoconf.h header file, which stores the kernel config as C macros, thus enabling both the kernel Makefile(s) and kernel code to make decisions based on whether or not a kernel feature is available.

Moving along, what if you are looking for a particular kernel configuration option but are having difficulty spotting it? No problem, the menuconfig UI system has a Search Configuration Parameter feature. Just as with the famous vi(1) editor, press the / ...


Looking up the differences in configuration

The moment the .config kernel configuration file is to be written to, the kbuild system checks whether it already exists, and if so, it backs it up with the name .config.old. Knowing this, we can always differentiate the two to see the changes we have wrought. However, using your typical diff(1) utility to do so makes the differences quite hard to interpret. The kernel helpfully provides a better way, a console-based script that specializes in doing precisely this. The scripts/diffconfig script (within the kernel source tree) is really useful for this. To see why, let's just run its help screen first:

$ scripts/diffconfig --help
Usage: diffconfig [-h] [-m] [<config1> <config2>]

Diffconfig is a simple utility for comparing two .config files.
Using standard diff to compare .config files often includes extraneous and
distracting information. This utility produces sorted output with...

Customizing the kernel menu – adding our own menu item

So, let's say you have developed a device driver, an experimental new scheduling class, a custom debugfs (debug filesystem) callback, or some other cool kernel feature. How will you let others on the team – or for that matter, your customer – know that this fantastic new kernel feature exists and allow them to select it (as either a built-in or as a kernel module) and thus build and make use of it? The answer is to insert a new menu item at an appropriate place in the kernel configuration menu.

To do so, it's useful to first understand a little more about the various Kconfig* files and where they reside. Let's find out.


The Kconfig* files

The Kconfig file at the root of the kernel source tree is used to fill in the initial screen of the menuconfig UI. Take a look at it if you wish. It works by sourcing various other Kconfig files in different folders of the kernel source tree. The following table summarizes the more important Kconfig* files and which menu they serve in the kbuild UI:

Menu Kconfig file location for it

The main menu, the initial screen


General setup

+ Enable loadable module support


Processor types and features
+ Bus options + Binary Emulations
(arch-specific; above the menu title is for x86; in general, the Kconfig file is here: arch/<arch>/Kconfig)


Power management


Firmware drivers




General architecture-dependent options


Enable the block layer
+ IO Schedulers




Creating a new menu item in the Kconfig file

As a trivial example, let's add our own Boolean dummy config option within the General Setup menu. We want the config name to be CONFIG_LLKD_OPTION1. As can be seen from the preceding table, the relevant Kconfig file to edit is the init/Kconfig one as this is the menu meta file that defines the General Setup menu.

Let's get to it:

  1. To be safe, always make a backup copy:
cp init/Kconfig init/Kconfig.orig
  1. Now, edit the init/Kconfig file:
vi init/Kconfig

Scroll down to an appropriate location within the file; here, we choose to insert our menu entry just after the CONFIG_LOCALVERSION_AUTO one. The following screenshot shows our new entry:

Figure 2.12 – Editing init/Kconfig and inserting our own menu entry
We have provided the preceding text as a patch to the original init/Kconfig file in our book's GitHub source tree. Find it under ch2/Kconfig.patch.

The new item starts with...


A few details on the Kconfig language

Our usage of the Kconfig language so far is just the tip of the proverbial iceberg. The fact is, the kbuild system uses the Kconfig language (or syntax) to express and create menus using simple ASCII text directives. The language includes menu entries, attributes, (reverse) dependencies, visibility constraints, help text, and so on. 

The kernel documents the Kconfig language constructs and syntax here: Do refer to this document for complete details.

A brief (and incomplete) mention of the more common Kconfig constructs is given in the following table:



config <FOO>

Specifies the menu entry name (of the form CONFIG_FOO) here; just put the FOO part.

Menu attributes

  bool ["<description>"]

Specifies the config option as a Boolean; its value in .config will be either Y (built into the kernel...



In this chapter, you first learned how to obtain for yourself a Linux kernel source tree. You then understood its release (or version) nomenclature, the various types of Linux kernels (-next trees, -rc/mainline trees, stable, LTS, SLTS and distributions), and the basic kernel development workflow. Along the way, you even got a quick tour of the kernel source tree so that its layout is clearer. Next, you saw how to extract the compressed kernel source tree to disk and, critically, how to configure the kernel – a key step in the process. Furthermore, you learned how to customize the kernel menu, adding your own entries to it, and a bit about the kbuild system and the associated Kconfig files it uses, among others.

Knowing how to fetch and configure the Linux kernel is a useful skill to possess. We have just begun this long and exciting journey. You will realize that with more experience and knowledge of kernel internals, drivers, and the target system hardware, your...




Further reading

About the Author

  • Kaiwan N Billimoria

    Kaiwan N Billimoria taught himself BASIC programming on his dad's IBM PC back in 1983. He was programming in C and Assembly on DOS until he discovered the joys of Unix, and by around 1997, Linux!

    Kaiwan has worked on many aspects of the Linux system programming stack, including Bash scripting, system programming in C, kernel internals, device drivers, and embedded Linux work. He has actively worked on several commercial/FOSS projects. His contributions include drivers to the mainline Linux OS and many smaller projects hosted on GitHub. His Linux passion feeds well into his passion for teaching these topics to engineers, which he has done for well over two decades now. He's also the author of Hands-On System Programming with Linux. It doesn't hurt that he is a recreational ultrarunner too.

    Browse publications by this author
Book Title
Access this book, plus 7,500 other titles for FREE
Access now