Chapter 5. Installing and Configuring Ceph

Ceph is a distributed, highly scalable storage system which provides block, object, and file-based storage in the same storage cluster. Ceph is open source and designed to run on off the shelf commodity hardware. Currently, Ceph RADOS Block Device (RBD) block storage is fully supported by Proxmox. The Ceph Reliable Autonomic Distributed Object Store (RADOS) provides features such as replication, snapshot, and other block storage abilities. There are numerous reasons to consider Ceph as a storage backend. The following are some of the highlights of why one should consider Ceph over other storage systems:

• Ceph is free
• Ceph is a highly scalable, reliable, distributed storage system
• Ceph RBD is seamlessly integrated with Proxmox clusters
• Ceph can be managed and monitored through a dedicated Ceph menu in the Proxmox GUI
• Ceph can tolerate multiple simultaneous drive failures
• As the Ceph cluster grows in size, so does the performance

Visit the official link to...

Ceph components

Before we dive in, let's take a look at some key components that make up a Ceph cluster. These components are what makes Ceph, and it is important to have a proper understanding of what they are. 

A physical node as cluster member

A physical node is the actual server hardware that holds one or more Ceph components.

Maps

In Ceph, maps hold information, such as a list of participating nodes in a cluster and their locations, and data paths, and a list of OSDs with certain data chunks. There are several maps in a Ceph cluster, such as a cluster map, an object storage daemon (OSD) map for a list of OSDs, a monitor map for known monitor nodes, a placement group (PG) map for the location of objects or data chunks, and a CRUSH map to determine how to store and retrieve data by computing the data storage location.

A cluster map

A cluster map is a map of devices and buckets that compose a Ceph cluster. Ceph uses a bucket hierarchy to define nodes or node locations, such as a room, rack,...

Virtual Ceph for training

It is possible to set up an entire Ceph cluster in a virtual environment. But this cluster should only be used for training and learning purposes. If you are learning Ceph for the first time and do not want to invest in the physical hardware, then a virtualized Ceph platform is certainly possible. This will eliminate the need to set up the physical hardware to set up Ceph nodes. The procedure to set up a virtual Ceph cluster is exactly the same as for a physical one. 

Installing a Ceph cluster

The following diagram is a basic representation of Proxmox and a Ceph cluster. Note that both the clusters are on separate subnets on separate switches:

A Ceph cluster should be set up with a separate subnet on a separate switch to keep it isolated from the Proxmox public subnet and for optimal Ceph cluster functioning. The Ceph Sync LAN is used by Ceph primarily to sync data between OSDs. The Ceph Public LAN is used primarily to serve user requests for data from Ceph into Proxmox VMs. The advantage of this practice is to keep Ceph's internal traffic isolated so that it does not interfere with the traffic of the running virtual machines. On a healthy Ceph cluster with the active+clean state, this is not an issue. However, when Ceph goes into self-healing mode due to an OSD or node failure, it rebalances itself by redistributing PGs among remaining OSDs, which causes very high bandwidth consumption. Separating two clusters ensures that the cluster does not slow down...

Summary

In this chapter, we learned what Ceph storage is and how to install and configure it to work with Proxmox cluster to store virtual disk images. We also learned various Ceph commands to manage a Ceph cluster. 

In the next chapter, we will learn details about KVM-based virtual machines. We will see how to create and manage KVM virtual machines and their advanced configurations.