Chapter 18: Managing Memory

This chapter covers issues related to memory management, which is an important topic for any Linux system but especially for embedded Linux, where system memory is usually in limited supply. After a brief refresher on virtual memory, I will show you how to measure memory usage, how to detect problems with memory allocation, including memory leaks, and what happens when you run out of memory. You will have to understand the tools that are available, from simple tools such as free and top, to complex ones such as mtrace and Valgrind.

We will learn the difference between kernel and user space memory, and how the kernel maps physical pages of memory to the address space of a process. Then we will locate and read the memory maps for individual processes under the proc filesystem. We will see how the mmap system call can be used to map a program's memory to a file, so that it can allocate memory in bulk or share it with another process. In the second half...

Technical requirements

To follow along with the examples, make sure you have the following:

• A Linux-based host system with gcc, make, top, procps, valgrind, and smem installed

All of these tools are available on most popular Linux distributions (such as Ubuntu, Arch, and so on).

All of the code for this chapter can be found in the Chapter18 folder in the book's GitHub repository: https://github.com/PacktPublishing/Mastering-Embedded-Linux-Programming-Third-Edition.

Virtual memory basics

To recap, Linux configures the memory management unit (MMU) of the CPU to present a virtual address space to a running program that begins at zero and ends at the highest address, 0xffffffff, on a 32-bit processor. This address space is divided into pages of 4 KiB by default. If 4 KiB pages are too small for your application, then you can configure the kernel to use HugePages, reducing the amount of system resources needed to access page table entries and increasing the Translation Lookaside Buffer (TLB) hit ratio.

Linux divides this virtual address space into an area for applications, called user space, and an area for the kernel, called kernel space. The split between the two is set by a kernel configuration parameter named PAGE_OFFSET. In a typical 32-bit embedded system, PAGE_OFFSET is 0xc0000000, giving the lower 3 gigabytes to user space and the top gigabyte to kernel space. The user address space is allocated per process so that each process runs in...

Kernel space memory layout

Kernel memory is managed in a fairly straightforward way. It is not demand-paged, which means that for every allocation using kmalloc() or similar function, there is real physical memory. Kernel memory is never discarded or paged out.

Some architectures show a summary of the memory mapping at boot time in the kernel log messages. This trace is taken from a 32-bit Arm device (a BeagleBone Black):

Memory: 511MB = 511MB total
Memory: 505980k/505980k available, 18308k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000  (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000  ( 896 kB)
    vmalloc : 0xe0800000 - 0xff000000  ( 488 MB)
    lowmem  : 0xc0000000 - 0xe0000000  ( 512 MB)
    pkmap   : 0xbfe00000 - 0xc0000000  ...

User space memory layout

Linux employs a lazy allocation strategy for user space, only mapping physical pages of memory when the program accesses it. For example, allocating a buffer of 1 MiB using malloc(3) returns a pointer to a block of memory addresses but no actual physical memory. A flag is set in the page table entries such that any read or write access is trapped by the kernel. This is known as a page fault. Only at this point does the kernel attempt to find a page of physical memory and add it to the page table mapping for the process. It is worthwhile demonstrating this with a simple program, MELP/Chapter18/pagefault-demo:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#define BUFFER_SIZE (1024 * 1024)
void print_pgfaults(void)
{
     int ret;
     struct rusage usage;
     ret = getrusage(RUSAGE_SELF, &usage);
   ...

The process memory map

Each running process in user space has a process map that we can inspect. These memory maps tell us how a program's memory is allocated and what shared libraries it is linked to.

You can see the memory map for a process through the proc filesystem. As an example, here is the map for the init process, PID 1:

# cat /proc/1/maps
00008000-0000e000 r-xp 00000000 00:0b 23281745 /sbin/init
00016000-00017000 rwxp 00006000 00:0b 23281745 /sbin/init
00017000-00038000 rwxp 00000000 00:00 0        [heap]
b6ded000-b6f1d000 r-xp 00000000 00:0b 23281695 /lib/libc-2.19.so
b6f1d000-b6f24000 ---p 00130000 00:0b 23281695 /lib/libc-2.19.so
b6f24000-b6f26000 r-xp 0012f000 00:0b 23281695 /lib/libc-2.19.so
b6f26000-b6f27000 rwxp 00131000 00:0b 23281695 /lib/libc-2.19.so
b6f27000-b6f2a000 rwxp 00000000 00:00 0
b6f2a000-b6f49000 r-xp 00000000 00:0b 23281359 /lib/ld-2.19.so
b6f4c000-b6f4e000 rwxp 00000000 00:00 0
b6f4f000-b6f50000 r...

Swapping

The idea of swapping is to reserve some storage where the kernel can place pages of memory that are not mapped to a file, freeing up the memory for other uses. It increases the effective size of physical memory by the size of the swap file. It is not a panacea: there is a cost to copying pages to and from a swap file, which becomes apparent on a system that has too little real memory for the workload it is carrying and so swapping becomes the main activity. This is sometimes known as disk thrashing.

Swap is seldom used on embedded devices because it does not work well with flash storage, where constant writing would wear it out quickly. However, you may want to consider swapping to compressed RAM (zram).

Swapping to compressed memory (zram)

The zram driver creates RAM-based block devices named /dev/zram0, /dev/zram1, and so on. Pages written to these devices are compressed before being stored. With compression ratios in the range of 30% to 50%, you can expect an overall...

Mapping memory with mmap

A process begins life with a certain amount of memory mapped to the text (the code) and data segments of the program file, together with the shared libraries that it is linked with. It can allocate memory on its heap at runtime using malloc(3) and on the stack through locally scoped variables and memory allocated through alloca(3). It may also load libraries dynamically at runtime using dlopen(3). All of these mappings are taken care of by the kernel. However, a process can also manipulate its memory map in an explicit way using mmap(2):

void *mmap(void *addr, size_t length, int prot, int flags,
int fd, off_t offset);

This function maps length bytes of memory from the file with the fd descriptor, starting at offset in the file, and returns a pointer to the mapping, assuming it is successful. Since the underlying hardware works in pages, length is rounded up to the nearest whole number of pages. The protection parameter, prot, is a combination of read...

How much memory does my application use?

As with kernel space, the different ways of allocating, mapping, and sharing user space memory make it quite difficult to answer this seemingly simple question.

To begin, you can ask the kernel how much memory it thinks is available, which you can do using the free command. Here is a typical example of the output:

   total used free shared buffers cached
Mem: 509016 504312 4704 0 26456 363860
-/+ buffers/cache: 113996 395020
Swap: 0 0 0

At first sight, this looks like a system that is almost out of memory with only 4,704 KiB free out of 509,016 KiB: less than 1%. However, note that 26,456 KiB is in buffers and
a whopping 363,860 KiB is in caches. Linux believes that free memory is wasted memory; the kernel uses free memory for buffers and caches with the knowledge that they can be shrunk when the need arises. Removing buffers and cache from the measurement provides true free memory, which is 395,020 KiB: 77% of the total...

Per-process memory usage

There are several metrics to measure the amount of memory a process is using. I will begin with the two that are easiest to obtain: the virtual set size (VSS) and the resident memory size (RSS), both of which are available in most implementations of the ps and top commands:

• VSS: Called VSZ in the ps command and VIRT in top, this is the total amount
  of memory mapped by a process. It is the sum of all the regions shown in
  /proc/<PID>/map. This number is of limited interest since only part of the virtual memory is committed to physical memory at any time.
• RSS: Called RSS in ps and RES in top, this is the sum of memory that is mapped to physical pages of memory. This gets closer to the actual memory budget of the process, but there is a problem: if you add the RSS of all the processes, you will get an overestimate of the memory in use because some pages will be shared.

Let's learn more about the top and ps commands.

Using top and...

Identifying memory leaks

A memory leak occurs when memory is allocated but not freed when it is no longer needed. Memory leakage is by no means unique to embedded systems, but it becomes an issue partly because targets don't have much memory in the first place and partly because they often run for long periods of time without rebooting, allowing the leaks to become
a large puddle.

You will realize that there is a leak when you run free or top and see that free memory is continually going down even if you drop caches, as shown in the preceding section. You will be able to identify the culprit (or culprits) by looking at the USS and RSS per process.

There are several tools to identify memory leaks in a program. I will look at two: mtrace and valgrind.

mtrace

mtrace is a component of glibc that traces calls to malloc, free, and related functions, and identifies areas of memory not freed when the program exits. You need to call the mtrace() function from within the program...

Running out of memory

The standard memory allocation policy is to over-commit, which means that the kernel will allow more memory to be allocated by applications than there is physical memory. Most of the time, this works fine because it is common for applications to request more memory than they really need. This also helps in the implementation of fork(2): it is safe to make a copy of a large program because the pages of memory are shared with the copy on write flag set. In the majority of cases, fork is followed by an exec function call, which unshares the memory and then loads a new program.

However, there is always the possibility that a particular workload will cause a group of processes to try to cash in on the allocations they have been promised simultaneously and so demand more than there really is. This is an out of memory situation, or OOM. At this point, there is no other alternative but to kill off processes until the problem goes away. This is the job of the out of...

Summary

Accounting for every byte of memory used in a virtual memory system is just not possible. However, you can find a fairly accurate figure for the total amount of free memory, excluding that taken by buffers and the cache, using the free command. By monitoring it over a period of time and with different workloads, you should become confident that it will remain within a given limit.

When you want to tune memory usage or identify sources of unexpected allocations, there are resources that give more detailed information. For kernel space, the most useful information is in /proc: meminfo, slabinfo, and vmallocinfo.

When it comes to getting accurate measurements for user space, the best metric is PSS,
as shown by smem and other tools. For memory debugging, you can get help from
simple tracers such as mtrace, or you have the heavyweight option of the Valgrind memcheck tool.

If you have concerns about the consequence of an OOM situation, you can fine-tune the allocation mechanism...

Further reading

The following resources have further information on the topics introduced in this chapter:

• Linux Kernel Development, 3rd Edition, by Robert Love
• Linux System Programming, 2nd Edition, by Robert Love
• Understanding the Linux VM Manager by Mel Gorman: https://www.kernel.org/doc/gorman/pdf/understand.pdf
• Valgrind 3.3 - Advanced Debugging and Profiling for Gnu/Linux Applications by J Seward, N. Nethercote, and J. Weidendorfer