The CPU Scheduler – Part 2

In this, our second chapter on the Linux kernel CPU (or task) scheduler, we continue our coverage from the previous chapter. In the preceding chapter, we covered several key areas regarding the workings (and visualization) of the CPU scheduler on the Linux OS. This included topics on what exactly the Kernel Schedulable Entity (KSE) on Linux is (it’s the thread!), the POSIX scheduling policies that Linux implements, using perf (and other tools) to see the thread/scheduler flow, and how the design of the modern scheduler is based upon modular scheduling classes. We also covered how to query any thread’s scheduling policy and priority (using a couple of command-line utilities), and finished by delving a lot deeper into the internal workings of the OS task scheduler.

With this background in place, we’re now ready to explore more on the CPU scheduler on Linux; in this chapter, we shall cover the following areas:

• Understanding...

Technical requirements

I assume you have gone through the complete chapter on Kernel Workspace Setup, that has been uploaded online, and have appropriately prepared a guest Virtual Machine (VM) running Ubuntu 22.04 LTS (or a later stable release, or a recent Fedora distro) and installed all the required packages. If not, I highly recommend you do this first.

To get the most out of this book, I strongly recommend you first set up the workspace environment, including cloning this book’s GitHub repository for the code, and work on it in a hands-on fashion. The repository can be found here: https://github.com/PacktPublishing/Linux-Kernel-Programming_2E.

Understanding, querying, and setting the CPU affinity mask

The task structure – the root data structure for the thread (or task), containing several dozen thread attributes – has a few attributes directly pertaining to scheduling: the priority (the nice as well as the Real-Time (RT) priority values), the scheduling class structure pointer, the runqueue the thread is on (if any), and so on. (FYI, we covered generic details on the task structure back in Chapter 6, Kernel Internals Essentials – Processes and Threads).

Among these is an important member, the CPU affinity bitmask (the actual structure member is cpumask_t *cpus_ptr. FYI, until the 5.3 kernel, it was a member named cpus_allowed; this was changed in this commit: https://github.com/torvalds/linux/commit/3bd3706251ee8ab67e69d9340ac2abdca217e733). This bit mask is just that: a bit mask of the CPU cores that the thread (represented by that task structure) is allowed to run on. A simple visualization helps...

Querying and setting a thread’s scheduling policy and priority

In Chapter 10, The CPU Scheduler – Part 1, in the Thread Priorities section, you learned how to query the scheduling policy and priority of any given thread via the chrt utility (we also demonstrated a simple Bash script to do so). There, we mentioned the fact that chrt internally invokes the sched_getattr() system call in order to query these attributes.

Very similarly, setting the scheduling policy and priority can be performed either by using the chrt utility (making it simple to do so within a script, for example), or programmatically within a (user space) C application with the sched_setattr() system call. In addition, the kernel exposes other APIs: sched_{g,s}etscheduler() and its pthread library wrapper APIs, pthread_{g,s}etschedparam() (as these are all user space APIs, we leave it to you to browse through their man pages to get the details and try them out for yourself).

Setting the policy...

An introduction to cgroups

In the hazy past, the kernel community struggled mightily with a rather vexing issue: though scheduling algorithms and their implementations – the early 2.6.0 O(1) scheduler, and a little later (with 2.6.23), the Completely Fair Scheduler (CFS) – promised, well, completely fair scheduling, it really wasn’t “completely fair” in any meaningful sense of the term!

Think about this for a moment: let’s say you are logged in to a Linux server along with nine other people. Everything else being equal, it is likely that processor time is (more or less) fairly shared between all ten of you; of course, you will understand that it’s not really people that run on the processor(s) and eat memory, it’s processes and threads that do so on their behalf.

For now, at least, let’s assume it’s (mostly) fairly shared. But, what if you, one of the ten users logged in, write a user space program that...

Running Linux as an RTOS – an introduction

Mainline or vanilla Linux (the kernel you download from https://kernel.org, or even a typical Linux Git kernel tree) is decidedly not an RTOS; it’s a General Purpose Operating System (GPOS; as is Windows, macOS, and Unix). In an RTOS, where hard real-time (RT) characteristics come into play, not only must the software obtain the correct result but there are deadlines associated with doing so; it must guarantee it meets these deadlines, every single time.

One can very broadly categorize an OS based on its RT characteristics in this manner (see Figure 11.16); at the left extreme are the non-RT OSs, and at the right extreme is the RTOS:

Figure 11.16: Categorizing an OS on the RT scale

The mainline or “vanilla” Linux OS, though not an RTOS, does a tremendous job performance-wise without even breaking a sweat. It easily qualifies as being a soft real-time OS: one where deadlines are met most of the...

Miscellaneous scheduling related topics

We close this chapter with a couple of miscellaneous yet useful topic mentions.

A few small (kernel space) routines to check out

Here are a few kernel routines you may find useful (we leave it to you to look up the details and sample usage within the kernel):

• rt_prio(): Given the priority as a parameter, returns a Boolean to indicate whether it’s a real-time task or not.
• rt_task(): Based on the priority value of the task, given the task structure pointer as a parameter, returns a Boolean to indicate whether it’s a real-time task or not (a wrapper over rt_prio()).
• task_is_realtime(): Similar, but based on the scheduling policy of the task. Given the task structure pointer as a parameter, returns a Boolean to indicate whether it’s a real-time task or not.

As the above routines all fall within a header (include/linux/sched/rt.h), simply include it and directly use them within your module...

Summary

In this, our second chapter on CPU (or task) scheduling on the Linux OS, you have learned several key things. Among them, you learned how to programmatically query and set any (user or kernel) thread’s CPU affinity mask; this naturally led to how you can programmatically query and set any thread’s scheduling policy and priority.

The whole notion of being “completely fair” (via the CFS implementation) was brought into question, and some light (quite a bit!) was shed on the elegant solution called cgroups (v2), which is now deeply embedded into the Linux kernel. We covered how systemd helps auto-integrate cgroups into modern distros, servers and even embedded systems, automatically and dynamically creating and maintaining various cgroups (via its slice and scope artifacts). You even learned how to leverage the cgroups v2 CPU controller to allocate CPU bandwidth (or utilization) as desired to processes in a sub-group, both via systemd unit files...

Questions

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter’s material: https://github.com/PacktPublishing/Linux-Kernel-Programming_2E/blob/main/questions/ch11_qs_assignments.txt. You will find some of the questions answered in the book’s GitHub repo: https://github.com/PacktPublishing/Linux-Kernel-Programming_2E/tree/main/solutions_to_assgn.

Further reading

To help you delve deeper into the subject with useful materials, we provide a rather detailed list of online references and links (and at times, even books) in a Further reading document in this book’s GitHub repository. The Further reading document is available here: https://github.com/PacktPublishing/Linux-Kernel-Programming_2E/blob/main/Further_Reading.md.

Leave a review!

Enjoying this book? Help readers like you by leaving an Amazon review. Scan the QR code below for a 20% discount code.

*Limited Offer