Running Kubernetes on Multiple Clouds and Cluster Federation

In this chapter, we'll take it to the next level by running Kubernetes on multiple clouds, multiple clusters, and cluster federation. A Kubernetes cluster is a closely knit unit where all of the components run in relative proximity and are connected by a fast network (typically, a physical data center or cloud provider availability zone). This is great for many use cases, but there are several important use cases where systems need to scale beyond a single cluster. One approach to address this use case is with Kubernetes federation, which is a methodical way to combine multiple Kubernetes clusters and interact with them as a single entity. Another approach taken by the Gardener (https://gardener.cloud/) project is to provide an abstraction around managing multiple separate Kubernetes clusters.

We will cover the following topics:

• The history of cluster federation on Kubernetes
• Understanding...

The history of cluster federation on Kubernetes

Before jumping into the details of cluster federation, let's get some historical context. It's funny to talk about the history of a project like Kubernetes that didn't even exist 5 years ago, but the pace of development and a large number of contributors took Kubernetes through accelerated evolution. This is especially relevant for Kubernetes federation.

In March 2015, the first revision of the Kubernetes Cluster Federation (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/federation.md) proposal was published. Back then, it was fondly nicknamed "Ubernetes." The basic idea was to reuse the existing Kubernetes APIs to manage multiple clusters. This proposal, now called Federation V1, went through several rounds of revision and implementation, but it never reached general availability and is considered deprecated at this point. The SIG cluster workgroup realized...

Understanding cluster federation

Cluster federation is conceptually simple. You aggregate multiple Kubernetes clusters and treat them as a single logical cluster. There is a federation control plane that presents to clients a single unified view of the system.

The following diagram demonstrates the big picture of the Kubernetes Cluster Federation. The pink box is a host cluster that runs the federation APIs and the green boxes are member clusters:

Figure 11.1: The Kubernetes Cluster Federation

The federation control plane consists of a federation API server and a federation controller manager that collaborate with each other. The federated API server forwards requests to all the clusters in the federation. In addition, the federated controller manager performs the duties of the controller manager across all of the clusters by routing requests to the individual federation cluster members' changes. In practice, cluster federation is not trivial and...

Managing a Kubernetes Cluster Federation

Managing a Kubernetes Cluster Federation involves many activities above and beyond managing a single cluster. You need to consider cascading resource deletion, load balancing across clusters, failover across clusters, federated service discovery, and federated discovery. Let's go over the various activities in detail. Note that due to the Alpha status of KubeFed, this should not be considered a step-by-step guide to follow. The goal here is to get a sense of what's involved in the management of multiple Kubernetes clusters as a federation.

Installing kubefedctl

The best way to interact with KubeFed is through the kubefedctl CLI. Here are the instructions to install the latest release of kubefedctl for macOS:

VERSION=0.3.0
OS=Darwin
ARCH=amd64
curl -LO https://github.com/kubernetes-sigs/kubefed/releases/download/v${VERSION}/kubefedctl-${VERSION}-${OS}-${ARCH}.tgz
tar -zxvf kubefedctl-*.tgz
chmod u+x kubefedctl
sudo mv kubefedctl...

Introducing the Gardener project

The Gardener (https://gardener.cloud/) project is an open source project developed by SAP. It lets you manage thousands (yes, thousands!) of Kubernetes clusters efficiently and economically. Gardener solves a very complex problem, and the solution is elegant but not simple. In this section, we will cover the terminology of Gardener, its conceptual model, dive deep into its architecture, and learn about its features of extensibility. The primary theme of Gardener is to use Kubernetes to manage Kubernetes clusters. A good way to think about Gardener is as Kubernetes-control-plane-as-a-service.

Understanding the terminology of Gardener

The Gardener project, as you may have guessed, uses botanical terminology to describe the world. There is a garden, which is a Kubernetes cluster that is responsible for managing seed clusters. A seed is a Kubernetes cluster that is responsible for managing a set of shoot clusters. A shoot cluster is a Kubernetes...

Summary

In this chapter, we've covered the important aspects of a Kubernetes Cluster Federation as well as the management of many Kubernetes clusters using the Gardener project. Cluster federation is still in beta and is a little raw, but it is already usable. There aren't a lot of deployments, and the officially supported target platforms are currently AWS and GCE/GKE, but there is a lot of momentum behind cloud federation. It is a very important piece for building massively scalable systems on Kubernetes. We've discussed the motivation and use cases for the Kubernetes Cluster Federation, the federation control plane components, and the federated Kubernetes objects. We also looked at the less supported aspects of a federation, such as custom scheduling, federated data access, and auto-scaling. We then looked at how to run multiple Kubernetes clusters, which includes setting up a Kubernetes Cluster Federation, adding and removing clusters to the federation along with...