Chapter 9: Creating a Storage Strategy

The mass storage options for embedded devices have a great impact on the rest of the system in terms of the robustness, speed, and methods used for in-field updates. Most devices employ flash memory in some form or another. Flash memory has become much less expensive over the past few years as storage capacities have increased from tens of megabytes to tens of gigabytes.

In this chapter, we will begin with a detailed look at the technology behind flash memory, as well as how different memory organization strategies affect the low-level driver software that has to manage it, including the Linux memory technology device (MTD) layer.

For each flash technology, there are different choices when it comes to the filesystem. I will describe those most commonly found on embedded devices and complete the survey by providing a summary of choices for each type of flash memory. Finally, we will consider some techniques that make the best use of flash...

Technical requirements

To follow along with the examples in this chapter, make sure you have the following:

• A Linux-based host system with e2fsprogs, genext2fs, mtd-utils, squashfs-tools, and util-linux or their equivalents installed
• The U-Boot source tree from Chapter 3, All About Bootloaders
• A microSD card reader and card
• A USB to TTL 3.3V serial cable
• The Linux kernel source tree from Chapter 4, Configuring and Building the Kernel
• BeagleBone Black
• A 5V 1A DC power supply

You should have already downloaded and built U-Boot for the BeagleBone Black back in Chapter 3, All About Bootloaders. You should have obtained the Linux kernel source tree from Chapter 4, Configuring and Building the Kernel.

Ubuntu provides packages for most of the tools needed to create and format various filesystems. To install the tools on an Ubuntu 20.04 LTS system, use the following command:

$ sudo apt install e2fsprogs genext2fs mtd-utils squashfs-tools util-linux...

Storage options

Embedded devices need storage that takes little power and is physically compact, robust, and reliable over a lifetime of perhaps tens of years. In almost all cases, this means solid-state storage. Solid-state storage was introduced many years ago with read-only memory (ROM), but for the past 20 years, it has been a flash memory of some kind. There have been several generations of flash memory in that time, progressing from NOR to NAND to managed flash such as eMMC.

NOR flash is expensive but reliable and can be mapped into the CPU address space, which allows you to execute code directly from flash. NOR flash chips are low capacity, ranging from a few megabytes to a gigabyte or so.

NAND flash memory is much cheaper than NOR and is available in higher capacities, in the range of tens of megabytes to tens of gigabytes. However, it needs a lot of hardware and software support to turn it into a useful storage medium.

Managed flash memory consists of one or more...

Accessing flash memory from the bootloader

In Chapter 3, All About Bootloaders, I mentioned the need for the bootloader to load kernel binaries and other images from various flash devices, and to perform system maintenance tasks such as erasing and reprogramming flash memory. It follows that the bootloader must have the drivers and infrastructure needed to support read, erase, and write operations on the type of memory you have, whether it be NOR, NAND, or managed. I will use U-Boot in the following examples; other bootloaders follow a similar pattern.

U-Boot and NOR flash

U-Boot has drivers for NOR CFI chips in drivers/mtd and utilizes various erase commands to erase memory and cp.b to copy data byte by byte, programming the flash cells. Suppose that you have NOR flash memory mapped from 0x40000000 to 0x48000000, of which 4 MiB, starting at 0x40040000, is a kernel image. Here, you would load a new kernel into flash using these U-Boot commands:

=> tftpboot 100000 uImage...

Accessing flash memory from Linux

Raw NOR and NAND flash memory is handled by the Memory Technology Device subsystem, or MTD, which provides you with basic interfaces to read, erase, and write blocks of flash memory. In the case of NAND flash, there are also functions that handle the OOB area and are used to identify bad blocks.

For managed flash, you need drivers to handle a particular hardware interface. MMC/SD cards and eMMC use the mmcblk driver, while CompactFlash and hard drives use the SCSI disk driver, sd. USB flash drives use the usb_storage driver, together with the
sd driver.

Memory technology devices

The MTD subsystem was started by David Woodhouse in 1999 and has been extensively developed over the intervening years. In this section, I will concentrate on the way it handles the two main technologies, NOR and NAND flash.

MTD consists of three layers: a core set of functions, a set of drivers for various types of chips, and user-level drivers that present the...

Filesystems for flash memory

There are several challenges when it comes to making efficient use of flash memory for mass storage: the mismatch between the size of an erase block and a disk sector, the limited number of erase cycles per erase block, and the need for bad block handling on NAND chips. These differences are resolved by a flash translation layer, or FTL.

Flash translation layers

A flash translation layer has the following features:

• Sub allocation: Filesystems work best with a small allocation unit, traditionally a 512-byte sector. This is much smaller than a flash erase block of 128 KiB or more. Therefore, erase blocks have to be subdivided into smaller units to avoid wasting large amounts of space.
• Garbage collection: A consequence of suballocation is that an erase block will contain a mixture of good data and stale data once the filesystem has been in use for a while. Since we can only free up whole erase blocks, the only way to reclaim this free space...

Filesystems for NOR and NAND flash memory

To use raw flash chips for mass storage, you have to use a filesystem that understands the peculiarities of the underlying technology. There are three such filesystems:

• JFFS2 (Journaling Flash File System 2): This was the first flash filesystem for Linux and is still in use today. It works for NOR and NAND memory, but is notoriously slow during mount.
• YAFFS2 (Yet Another Flash File System 2): This is similar to JFFS2, but specifically for NAND flash memory. It was adopted by Google as the preferred raw flash filesystem on Android devices.
• UBIFS (Unsorted Block Image File System): This works in conjunction with the UBI block driver to create a reliable flash filesystem. It works well with both NOR and NAND memory, and since it generally offers better performance than JFFS2 or YAFFS2, it should be the preferred solution for new designs.

All of these use MTD as the common interface to flash memory.

JFFS2

The Journaling...

Filesystems for managed flash

As the trend toward managed flash technologies continues, particularly eMMC, we need to consider how to use it effectively. While they appear to have the same characteristics as hard disk drives, the underlying NAND flash chips have the limitations of large erase blocks with limited erase cycles and bad block handling. And, of course, we need robustness in the event of losing power.

It is possible to use any of the normal disk filesystems, but we should try to choose one that reduces disk writes and has a fast restart after an unscheduled shutdown.

Flashbench

To make optimum use of the underlying flash memory, you need to know the erase block size and page size. Manufacturers do not publish these numbers as a rule, but it is possible to deduce them by observing the behavior of the chip or card.

Flashbench is one such tool. It was initially written by Arnd Bergman, as described in the LWN article available at https://lwn.net/Articles/428584...

Read-only compressed filesystems

Compressing data is useful if you don't have quite enough storage to fit everything in. Both JFFS2 and UBIFS do on-the-fly data compression by default. However, if the files are never going to be written, as is usually the case with the root filesystem, you can achieve better compression ratios by using a read-only compressed filesystem. Linux supports several of these: romfs, cramfs, and squashfs. The first two are obsolete now, so I will only describe SquashFS.

SquashFS

The SquashFS filesystem was written by Phillip Lougher in 2002 as a replacement for cramfs. It existed as a kernel patch for a long time, eventually being merged into mainline Linux in version 2.6.29 in 2009. It is very easy to use: you create a filesystem image using mksquashfs and install it to the flash memory:

$ mksquashfs rootfs rootfs.squashfs

The resulting filesystem is read-only, so there is no mechanism for modifying any of the files at runtime. The only way...

Temporary filesystems

There are always some files that have a short lifetime or have no significance after a reboot. Many such files are put into /tmp, and so it makes sense to keep these files from reaching permanent storage.

The temporary filesystem, tmpfs, is ideal for this purpose. You can create a temporary RAM-based filesystem by simply mounting tmpfs:

# mount -t tmpfs tmp_files /tmp

As with procfs and sysfs, there is no device node associated with tmpfs, so you have to supply a placekeeper string, which is tmp_files in the preceding example.

The amount of memory used will grow and shrink as files are created and deleted. The default maximum size is half the physical RAM. In most cases, it would be a disaster if tmpfs grew to be that large, so it is a very good idea to cap it with the -o size parameter. The parameter can be given in bytes, KiB (k), MiB (m), or GiB (g), like this, for example:

# mount -t tmpfs -o size=1m tmp_files /tmp

In addition to /tmp, some...

Making the root filesystem read-only

You need to make your target device able to survive unexpected events, including file corruption, and still be able to boot and achieve at least a minimum level of functionality. Making the root filesystem read-only is a key part of achieving this ambition because it eliminates accidental overwrites. Making it read-only is easy: replace rw with ro on the kernel command line or use an inherently read-only filesystem such as SquashFS. However, you will find that there are a few files and directories that are traditionally writable:

• /etc/resolv.conf: This file is written by network configuration scripts to record the addresses of DNS name servers. The information is volatile, so you simply have to make it a symlink to a temporary directory; for example, /etc/resolv.conf -> /var/run/resolv.conf.
• /etc/passwd: This file, along with /etc/group, /etc/shadow, and
  /etc/gshadow, stores user and group names and passwords. They need to
  be symbolically...

Filesystem choices

So far, we have looked at the technology behind solid-state memory and at the many types of filesystems. Now, it is time to summarize the options that are available. In most cases, you will be able to divide your storage requirements into these three categories:

• Permanent, read-write data: Runtime configuration, network parameters, passwords, data logs, and user data
• Permanent, read-only data: Programs, libraries, and configurations files that are constant; for example, the root filesystem
• Volatile data: Temporary storage; for example, /tmp

The choices for read-write storage are as follows:

• NOR: UBIFS or JFFS2
• NAND: UBIFS, JFFS2, or YAFFS2
• eMMC: ext4 or F2FS

For read-only storage, you can use any of these, mounted with the ro attribute. Additionally, if you want to save space, you could use SquashFS. Finally, for volatile storage, there is only one choice: tmpfs.

Summary

Flash memory has been the storage technology of choice for embedded Linux from the beginning, and over the years, Linux has gained very good support, from low-level drivers up to flash-aware filesystems, with the latest being UBIFS.

As the rate at which new flash technologies are introduced increases, it is becoming harder to keep pace with the changes at the high end. System designers are increasingly turning to managed flash in the form of eMMC, to provide a stable hardware and software interface that is independent of the memory chips inside. Embedded Linux developers
are beginning to get to grips with these new chips. Support for TRIM in ext4 and F2FS is well-established, and it is slowly finding its way into the chips themselves. Also, the appearance of new filesystems that have been optimized to manage flash, such as F2FS, is a welcome step forward.

However, the fact remains that flash memory is not the same as a hard disk drive. You have to be careful when you...

Further reading

The following resources contain further information about the topics that were introduced in this chapter:

• XIP: The past, the present... the future?, by Vitaly Wool: https://archive.fosdem.org/2007/slides/devrooms/embedded/Vitaly_Wool_XIP.pdf
• General MTD documentation: http://www.linux-mtd.infradead.org/doc/general.html
• Optimizing Linux with cheap flash drives, by Arnd Bergmann: https://lwn.net/Articles/428584/
• eMMC/SSD File System Tuning Methodology, Cogent Embedded, Inc.:
  https://elinux.org/images/b/b6/EMMC-SSD_File_System_Tuning_Methodology_v1.0.pdf
• Flash-Friendly File System (F2FS), by Joo-Young Hwang: https://elinux.org/images/1/12/Elc2013_Hwang.pdf
• An F2FS teardown, by Neil Brown: https://lwn.net/Articles/518988/