Chapter 5: Libvirt Storage

This chapter provides you with an insight into the way that KVM uses storage. Specifically, we will cover both storage that's internal to the host where we're running virtual machines and shared storage. Don't let the terminology confuse you here – in virtualization and cloud technologies, the term shared storage means storage space that multiple hypervisors can have access to. As we will explain a bit later, the three most common ways of achieving this are by using block-level, share-level, or object-level storage. We will use NFS as an example of share-level storage, and Internet Small Computer System Interface (iSCSI) and Fiber Channel (FC) as examples of block-level storage. In terms of object-based storage, we will use Ceph. GlusterFS is also commonly used nowadays, so we'll make sure that we cover that, too. To wrap everything up in an easy-to-use and easy-to-manage box, we will discuss some open source projects that might...

Introduction to storage

Unlike networking, which is something that most IT people have at least a basic understanding of, storage tends to be quite different. In short, yes, it tends to be a bit more complex. There are loads of parameters involved, different technologies, and…let's be honest, loads of different types of configuration options and people enforcing them. And a lot of questions. Here are some of them:

• Should we configure one NFS share per storage device or two?
• Should we create one iSCSI target per storage device or two?
• Should we create one FC target or two?
• How many Logical Unit Numbers (LUNs) per target?
• What kind of cluster size should we use?
• How should we carry out multipathing?
• Should we use block-level or share-level storage?
• Should we use block-level or object-level storage?
• Which technology or solution should we choose?
• How should we configure caching?
• How should we configure zoning or masking?
• How...

Storage pools

When you first start using storage devices—even if they're cheaper boxes—you're faced with some choices. They will ask you to do a bit of configuration—select the RAID level, configure hot-spares, SSD caching...it's a process. The same process applies to a situation in which you're building a data center from scratch or extending an existing one. You have to configure the storage to be able to use it.

Hypervisors are a bit picky when it comes to storage, as there are storage types that they support and storage types that they don't support. For example, Microsoft's Hyper-V supports SMB shares for virtual machine storage, but it doesn't really support NFS storage for virtual machine storage. VMware's vSphere Hypervisor supports NFS, but it doesn't support SMB. The reason is simple—a company developing a hypervisor chooses and qualifies technologies that its hypervisor is going to support. Then...

NFS storage pool

As a protocol, NFS has been around since the mid-80s. It was originally developed by Sun Microsystems as a protocol for sharing files, which is what it's been used for up to this day. Actually, it's still being developed, which is quite surprising for a technology that's so old. For example, NFS version 4.2 came out in 2016. In this version, NFS received a very big update, such as the following:

• Server-side copy: A feature that significantly enhances the speed of cloning operations between NFS servers by carrying out cloning directly between NFS servers
• Sparse files and space reservation: Features that enhance the way NFS works with files that have unallocated blocks, while keeping an eye on capacity so that we can guarantee space availability when we need to write data
• Application data block support: A feature that helps applications that work with files as block devices (disks)
• Better pNFS implementation

There are other bits...

iSCSI and SAN storage

Using iSCSI for virtual machine storage has long been the regular thing to do. Even if you take into account the fact that iSCSI isn't the most efficient way to approach storage, it's still so widely accepted that you'll find it everywhere. Efficiency is compromised for two reasons:

• iSCSI encapsulates SCSI commands into regular IP packages, which means segmentation and overhead as IP packages have a pretty large header, which means less efficiency.
• Even worse, it's TCP-based, which means that there are sequence numbers and retransmissions, which can lead to queueing and latency, and the bigger the environment is, the more you usually feel these effects affect your virtual machine performance.

That being said, the fact that it's based on an Ethernet stack makes it easier to deploy iSCSI-based solutions, while at the same time offering some unique challenges. For example, sometimes it's difficult to explain to a customer...

Storage redundancy and multipathing

Redundancy is one of the keywords of IT, where any single component failure could mean big problems for a company or its customers. The general design principle of avoiding SPOF is something that we should always stick to. At the end of the day, no network adapter, cable, switch, router, or storage controller is going to work forever. So, calculating redundancy into our designs helps our IT environment during its normal life cycle.

At the same time, redundancy can be combined with multipathing to also ensure higher throughput. For example, when we connect our physical host to FC storage with two controllers with four FC ports each, we can use four paths (if the storage is active-passive) or eight paths (if it's active-active) to the same LUN(s) exported from this storage device to a host. This gives us multiple additional options for LUN access, on top of the fact that it gives us more availability, even in the case of failure.

Getting...

Gluster and Ceph as a storage backend for KVM

There are other advanced types of filesystems that can be used as the libvirt storage backend. So, let's now discuss two of them—Gluster and Ceph. Later, we'll also check how libvirt works with GFS2.

Gluster

Gluster is a distributed filesystem that's often used for high-availability scenarios. Its main advantages over other filesystems are the fact that it's scalable, it can use replication and snapshots, it can work on any server, and it's usable as a basis for shared storage—for example, via NFS and SMB. It was developed by a company called Gluster Inc., which was acquired by RedHat in 2011. However, unlike Ceph, it's a file storage service, while Ceph offers block and object-based storage. Object-based storage for block-based devices means direct, binary storage, directly to a LUN. There are no filesystems involved, which theoretically means less overhead as there's no filesystem...

Virtual disk images and formats and basic KVM storage operations

Disk images are standard files stored on the host's filesystem. They are large and act as virtualized hard drives for guests. You can create such files using the dd command, as shown:

# dd if=/dev/zero of=/vms/dbvm_disk2.img bs=1G count=10

Here is the translation of this command for you:

Duplicate data (dd) from the input file (if) of /dev/zero (virtually limitless supply of zeros) into the output file (of) of /vms/dbvm_disk2.img (disk image) using blocks of 1 G size (bs = block size) and repeat this (count) just once (10).

Important note:

dd is known to be a resource-hungry command. It may cause I/O problems on the host system, so it's good to first check the available free memory and I/O state of the host system, and only then run it. If the system is already loaded, lower the block size to MB and increase the count to match the size of the file you wanted (use bs=1M, count=10000 instead of...

The latest developments in storage – NVMe and NVMeOF

In the past 20 or so years, by far the biggest disruption in the storage world in terms of technology has been the introduction of Solid State Drives (SSDs). Now, we know that a lot of people have gotten quite used to having them in their computers—laptops, workstations, whichever type of device we use. But again, we're discussing storage for virtualization, and enterprise storage concepts overall, and that means that our regular SATA SSDs aren't going to make the cut. Although a lot of people use them in mid-range storage devices and/or handmade storage devices that host ZFS pools (for cache), some of these concepts have a life of their own in the latest generations of storage devices. These devices are fundamentally changing the way technology is working and redoing parts of modern IT history in terms of which protocols are used, how fast they are, how much lower latencies they have, and how they approach...

Summary

In this chapter, we introduced and configured various Open Source storage concepts for libvirt. We also discussed industry-standard approaches, such as iSCSI and NFS, as they are often used in infrastructures that are not based on KVM. For example, VMware vSphere-based environments can use FC, iSCSI, and NFS, while Microsoft-based environments can only use FC and iSCSI, from the list of subjects we covered in this chapter.

The next chapter will cover subjects related to virtual display devices and protocols. We'll provide an in-depth introduction to VNC and SPICE protocols. We will also provide a description of other protocols that are used for virtual machine connection. All that will help us to understand the complete stack of fundamentals that we need to work with our virtual machines, which we covered in the past three chapters.

Questions

1. What is a storage pool?
2. How does NFS storage work with libvirt?
3. How does iSCSI work with libvirt?
4. How do we achieve redundancy on storage connections?
5. What can we use for virtual machine storage except NFS and iSCSI?
6. Which storage backend can we use for object-based storage with libvirt?
7. How can we create a virtual disk image to use with a KVM virtual machine?
8. How does using NVMe SSDs and SCM devices change the way that we create storage tiers?
9. What are the fundamental problems of delivering tier-0 storage services for virtualization, cloud, and HPC environments?

Further reading

Please refer to the following links for more information regarding what was covered in this chapter:

• What's new with RHEL8 file systems and storage: https://www.redhat.com/en/blog/whats-new-rhel-8-file-systems-and-storage
• oVirt storage: https://www.ovirt.org/documentation/administration_guide/#chap-Storage
• RHEL 7 storage administration guide: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/index
• RHEL 8 managing storage devices: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_storage_devices/index
• OpenFabrics CCIX, Gen-Z, OpenCAPI (overview and comparison): https://www.openfabrics.org/images/eventpresos/2017presentations/213_CCIXGen-Z_BBenton.pdf