Managing Storage

In this chapter, we'll look at how Kubernetes manages storage. Storage is very different from compute, but at a high level they are both resources. Kubernetes as a generic platform takes the approach of abstracting storage behind a programming model and a set of plugins for storage providers. First, we'll go into detail about the conceptual storage model and how storage is made available to containers in the cluster. Then, we'll cover the common cloud platform storage providers, such as Amazon Web Services (AWS), Google Compute Engine (GCE), and Azure. Then we'll look at a prominent open source storage provider, GlusterFS from Red Hat, which provides a distributed filesystem. We'll also look into an alternative solution – Flocker – that manages your data in containers as part of the Kubernetes cluster. Finally, we'll see how Kubernetes supports the integration of existing enterprise storage solutions.

At the...

Persistent volumes walkthrough

In this section, we will understand the Kubernetes storage conceptual model and see how to map persistent storage into containers so they can read and write. Let's start by understanding the problem of storage. Containers and pods are ephemeral.

Anything a container writes to its own filesystem gets wiped out when the container dies. Containers can also mount directories from their host node and read or write to them. These will survive container restarts, but the nodes themselves are not immortal. Also, if the pod itself is rescheduled to a different node, the container will not have access to the old node host's filesystem.

There are other problems, such as ownership for mounted hosted directories when the container dies. Just imagine a bunch of containers writing important data to various data directories on their host and then going away, leaving all that data all over the nodes with no direct way to tell what container wrote...

Public cloud storage volume types – GCE, AWS, and Azure

In this section, we'll look at some of the common volume types available in the leading public cloud platforms. Managing storage at scale is a difficult task that eventually involves physical resources, similar to nodes. If you choose to run your Kubernetes cluster on a public cloud platform, you can let your cloud provider deal with all these challenges and focus on your system. But it's important to understand the various options, constraints, and limitations of each volume type.

Many of the volume types we will go over are handled by in-tree plugins (part of core Kubernetes), but are in the process of migrating to out-of-tree CSI plugins. We will cover CSI later.

Amazon EBS

AWS provides Elastic Block Store (EBS) as persistent storage for EC2 instances. An AWS Kubernetes cluster can use AWS EBS as persistent storage with the following limitations:

• The pods must run on AWS EC2 instances...

GlusterFS and Ceph volumes in Kubernetes

GlusterFS and Ceph are two distributed persistent storage systems. GlusterFS is, at its core, a network filesystem. Ceph is, at its core, an object store. Both expose block, object, and filesystem interfaces. Both use the xfs filesystem under the hood to store the data and metadata as xattr attributes. There are several reasons why you may want to use GlusterFs or Ceph as persistent volumes in your Kubernetes cluster:

• You may have a lot of data and applications that access the data in GlusterFS or Ceph
• You have operational expertise managing GlusterFS or Ceph
• You run in the cloud, but the limitations of the cloud platform's persistent storage are a non-starter

Using GlusterFS

GlusterFS is intentionally simple, exposing the underlying directories as they are and leaving it to clients (or middleware) to handle high availability, replication, and distribution. Gluster organizes the data into logical volumes...

Flocker as a clustered container data volume manager

So far, we have discussed storage solutions that stored the data outside the Kubernetes cluster (except for emptyDir and HostPath, which are not persistent). Flocker is a little different. It is Docker-aware. It was designed to let Docker data volumes transfer with their container when the container is moved between nodes. You may want to use the Flocker volume plugin if you're migrating a Docker-based system that uses a different orchestration platform, such as Docker Compose or Mesos, to Kubernetes and you use Flocker for orchestrating storage. Personally, I feel that there is a lot of duplication between what Flocker does and what Kubernetes does to abstract storage.

Flocker has a control service and agents on each node. Its architecture is very similar to Kubernetes with its API server and the kubelet running on each node. The Flocker control service exposes a REST API and manages the configuration of the state across...

Integrating enterprise storage into Kubernetes

If you have an existing Storage Area Network (SAN) exposed over the iSCSI interface, Kubernetes has a volume plugin for you. It follows the same model as other shared persistent storage plugins we've seen earlier. It supports the following features:

• Connect to one portal
• Mount a device directly or via multipathd
• Format and partition any new device
• Authenticate via CHAP

You must configure the iSCSI initiator, but you don't have to provide any initiator information. All you need to provide is the following:

• The IP address of the iSCSI target and port (if not the default 3260)
• The target's iqn (an iSCSI-qualified name) – typically the reversed domain name
• LUN – the logical unit number
• The filesystem type
• A read-only Boolean flag

The iSCSI plugin supports ReadWriteOnce and ReadonlyMany. Note that you can't partition your device...

Projecting volumes

It's possible to project multiple volumes into a single directory, so they appear as a single volume. The supported volume types are Kubernetes-managed: secret, downwardAPI, and configMap. This is useful if you want to mount multiple sources of configuration into a pod. Instead of having to create a separate volume for each source, you can bundle all of them into a single projected volume. Here is an example:

apiVersion: v1
kind: Pod
metadata:
  name: the-pod
spec:
  containers:
  - name: the-container
    image: busybox
    volumeMounts:
    - name: all-in-one
      mountPath: "/projected-volume"
      readOnly: true
  volumes:
  - name: all-in-one
    projected:
      sources:
      - secret:
          name: the-secret
          items:
            - key: username
              path: the-group/the-user
      - downwardAPI:
          items:
            - path: "labels"
              fieldRef:
                fieldPath: metadata...

Using out-of-tree volume plugins with FlexVolume

FlexVolume became generally available in Kubernetes 1.8. It allows you to consume out-of-tree storage through a uniform API. Storage providers write a driver that you install on all nodes. The FlexVolume plugin can dynamically discover existing drivers. Here is an example of using a FlexVolume to bind to an external NFS volume:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-nfs
  namespace: default
spec:
  containers:
  - name: nginx-nfs
    image: nginx
    volumeMounts:
    - name: test
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: test
    flexVolume:
      driver: "k8s/nfs"
      fsType: "nfs"
      options:
        server: "172.16.0.25"
        share: "dws\_nas\_scratch"

However, at this point I highly recommend you avoid using the FlexVolume plugin and utilize CSI plugins instead.

The Container Storage Interface

The Container Storage Interface (CSI) is an initiative to standardize the interaction between container orchestrators and storage providers. It is driven by Kubernetes, Docker, Mesos, and Cloud Foundry. The idea is that storage providers implement just one CSI driver and container orchestrators need to support only the CSI. It is the equivalent of CNI for storage. There are several advantages over the FlexVolume approach:

• CSI is an industry-wide standard
• New capabilities are made available for CSI plugins only (such as volume snapshots and volume cloning)
• FlexVolume plugins require access to the node and master root filesystem to deploy drivers
• FlexVolume's storage driver often requires many external dependencies
• FlexVolume's EXEC-style interface is clunky

A CSI volume plugin was added in Kubernetes 1.9 as an alpha feature and is generally available since Kubernetes 1.13. FlexVolume will remain for...

Summary

In this chapter, we took a deep look into storage in Kubernetes. We've looked at the generic conceptual model based on volumes, claims, and storage classes, as well as the implementation of volume plugins. Kubernetes eventually maps all storage systems into mounted filesystems in containers or devices of raw block storage. This straightforward model allows administrators to configure and hook up any storage system from localhost directories through cloud-based shared storage all the way to enterprise storage systems. The transition of storage provisioners from in-tree to CSI-based out-of-tree drivers bodes well for the storage ecosystem. You should now have a clear understanding of how storage is modeled and implemented in Kubernetes and be able to make intelligent choices regarding how to implement storage in your Kubernetes cluster.

In Chapter 7, Running Stateful Applications with Kubernetes, we'll see how Kubernetes can raise the level of abstraction...