Understanding Kernel-Mode Rootkits

In this chapter, we are going to dig deeper into the Windows kernel and its internal structures and mechanisms. We will cover different techniques used by malware authors to hide the presence of their malware from users and antivirus products.

We will look at different advanced kernel-mode hooking techniques, process injection in kernel mode, and how to perform static and dynamic analysis there.

Before we get into rootkits and learn how they are implemented, we need to understand how the operating system (OS) works and how rootkits can target different parts of the OS and use it to their advantage.

In this chapter, we will cover the following topics:

• Kernel mode versus user mode
• Windows internals
• Rootkits and device drivers
• Hooking mechanisms
• DKOM
• Process injection in kernel mode
• KPP in x64 systems (PatchGuard)
• Static and dynamic analysis in kernel mode

Kernel mode versus user mode

You have already seen several user-mode processes on your computer (all the applications you see are running in user mode) and learned how to modify files, connect to the internet, and perform lots of activities. However, you might be surprised to know that user-mode applications don't have privileges to do all of this.

For any process to create a file or connect to a domain, it needs to send a request to the kernel mode to perform that action. This request is done through what is known as a system call, and this system call switches to kernel mode to perform this action (if permission is granted). Kernel mode and user mode are not only supported by the OS – they are also supported by the processors through protection rings (or hardware restrictions).

Protection rings

x86 processors provide four rings of privileges (x64 is slightly different). Each ring has lower privileges than the previous one, as shown in the following diagram:

...

Windows internals

Before we dive into the malicious activities of rootkits, let's take a look at how the Windows OS works and how the interaction between the user mode and kernel mode is organized. This knowledge will allow us to understand the specifics of kernel-mode malware and what parts of the system it may target.

The anatomy of Windows

As we mentioned previously, the OS is divided into two parts: user mode and kernel mode. This is shown in the following diagram:

Figure 7.2 – The Windows OS design

Now, let's learn about the scope of these applications:

• User mode: This contains all the processes running in the system (which you can see in Task Manager). These processes run under subsystems such as POSIX, the Win32 subsystem, and (more recently) the Windows Subsystem for Linux. All of these subsystems call different APIs, which are tailored for that system through specific libraries, such as kernel32.dll in the Win32 and Win64...

Rootkits and device drivers

Now that you understand Windows internals and how user mode and kernel mode interactions work, let's dig into rootkits. In this section, we will understand what rootkits are and how they are designed. After we grasp the basic concepts of rootkits, we will discuss device drivers.

What is a rootkit?

Rootkits are essentially low-level tools that provide stealth capabilities to malicious modules. This way, their main purpose is generally to complicate the malware detection and remediation procedures on the target machine by hiding the presence of related artifacts. There are multiple ways this can be done, so let's discuss them in detail.

Types of rootkits

There are various types of rootkits in user mode, kernel mode, and even boot mode:

• User-mode or application rootkits: We covered user-mode rootkits in Chapter 5, Inspecting Process Injection and API Hooking; they inject malicious code into other processes and hook their APIs to...

Hooking mechanisms

In this section, we will explore different types of hooking mechanisms. In the following diagram, we can see various types of hooking techniques that rootkits use at different stages of the request processing flow:

Figure 7.6 – The hooking mechanisms of rootkits

Rootkits can install hooks at different stages of this process flow:

• User-mode hooking/API hooking: These are the user-mode API hooking mechanisms that are used for hiding malware processes, files, registry keys, and more. We covered this in Chapter 5, Inspecting Process Injection and API Hooking.
• SYSENTER hooking: This is the first option that's available for the kernel-mode rootkits to hook. In this case, they change the address that sysenter will transfer the execution to and intercept all requests from user mode to kernel mode.
• SSDT hooking: This technique works more closely with the functions that the rootkit wants to hook. This type of hooking modifies...

DKOM

DKOM is one of the most common techniques used by rootkits to hide malicious user-mode processes. This technique relies on how the OS represents processes and threads. To understand this technique, you need to learn more about the objects that are being manipulated by the rootkit: EPROCESS and ETHREAD.

The kernel objects – EPROCESS and ETHREAD

Windows creates an object called EPROCESS for each process that's created in the system. This object includes all the important information about this process, such as its Virtual Address Descriptors (VADs), which store the map of this process's virtual memory and its representation in physical memory. It also includes the process ID, the parent process ID, and a doubly linked list called ActiveProcessLinks, which connects all EPROCESS objects of all processes. Each EPROCESS includes an address to the next EPROCESS object (which represents the next process) called FLink and the address to the previous EPROCESS object...

Process injection in kernel mode

Process injection in kernel mode is a popular technique used by multiple malware families, including Stuxnet (with its MRxCls rootkit), to create another way of maintaining persistence and disguising malware activities under a legitimate process name. For a device driver to be able to read and write memory inside a process, it needs to attach itself to this process's memory space.

Once the driver is attached to this process's memory space, it can see this process's virtual memory, and it becomes possible to read and write directly to it. For example, if the process executable's ImageBase is 0x00400000, then the driver can access it normally, as follows:

CMP WORD PTR [00400000h], 'ZM'
JNZ <not_mz>

For a driver to be able to attach to the process memory, it needs to get its EPROCESS using the PsLookupProcessByProcessId API and then use the KeStackAttachProcess API to attach to this process's memory space...

KPP in x64 systems (PatchGuard)

In x64 systems, Microsoft has introduced new protection against kernel-mode hooking and patching called KPP, or PatchGuard. This protection disables any patching of the SSDT and the core kernel code. It doesn't allow the usage of kernel stacks beyond what was allocated by the kernel itself.

Additionally, Microsoft allows only signed drivers to be loaded in the x64 systems, except for situations when the system is running in test mode or driver signature enforcement is disabled.

KPP received lots of criticism from antivirus and firewall vendors when it was first introduced because SSDT hooking and other hooking types were heavily used in multiple security products. Microsoft has created a new API to help antivirus products replace their hooking methods.

Although several ways of bypassing PatchGuard have been documented, for the last several years, Microsoft has released only a few major updates to deal with these techniques. In addition...

Static and dynamic analysis in kernel mode

Once we know how rootkits work, it becomes possible to analyze them. The first thing worth mentioning is that not all kernel-mode malware families just hide the presence of actual payloads – some of them can perform malicious actions on their own as well. In this section, we will familiarize ourselves with tools that can facilitate rootkit analysis to understand malware functionalities and learn some particular usage-related nuances.

Static analysis

It always makes sense to start from static analysis, especially if the debugging setup is not available straight away. In some cases, it is possible to perform both static and dynamic analysis using the same tools.

Rootkit file structure

Rootkit samples are usually drivers that implement the traditional MZ-PE structure with the IMAGE_SUBSYSTEM_NATIVE value specified in the subsystem field of the IMAGE_OPTIONAL_HEADER32 structure. They use the usual x86 or x64 instructions that...

Summary

In this chapter, we familiarized ourselves with Windows kernel mode and learned how requests are passed from user mode to kernel mode and back again. Then, we discussed rootkits, what parts of this process may be targeted by them, and for what reason. We also covered various techniques that are implemented in modern rootkits, including how existing security mechanisms can be bypassed by malware.

Finally, we explored the tools that are available to perform static and dynamic analysis of kernel-mode threats, learned how to set up a testing environment, and summarized generic guidelines that can be followed when performing the analysis. By completing this chapter, you should have a strong understanding of how advanced kernel-mode threats work and how they can be analyzed using various tools and approaches.

In Chapter 8, Handling Exploits and Shellcode, we will explore the various types of exploits and learn how legitimate software can be abused to let attackers perform malicious...