Getting Started with Kubernetes, Second Edition: Orchestrate and manage large-scale Docker deployments, Second Edition

At the core of container technology are control groups (cgroups) and namespaces. Additionally, Docker uses union filesystems for added benefits to the container development process.

Cgroups work by allowing the host to share and also limit the resources each process or container can consume. This is important for both, resource utilization and security, as it prevents denial-of-service attacks on the host's hardware resources. Several containers can share CPU and memory while staying within the predefined constraints.

Namespaces offer another form of isolation for process interaction within operating systems. Namespaces limit the visibility a process has on other processes, networking, filesystems, and user ID components. Container processes are limited to see only what is in the same namespace. Processes from containers or the host processes are not directly accessible from within this container process. Additionally, Docker gives each container its own networking stack that protects the sockets and interfaces in a similar fashion.

Composition of a container

Union filesystems are also a key advantage of using Docker containers. Containers run from an image. Much like an image in the VM or Cloud world, it represents state at a particular point in time. Container images snapshot the filesystem, but tend to be much smaller than a VM. The container shares the host kernel and generally runs a much smaller set of processes, so the filesystem and boot strap period tend to be much smaller. Though those constraints are not strictly enforced. Second, the union filesystem allows for efficient storage, download, and execution of these images.

The easiest way to understand union filesystems is to think of them like a layer cake with each layer baked independently. The Linux kernel is our base layer; then, we might add an OS such as Red Hat Linux or Ubuntu. Next, we might add an application such as Nginx or Apache. Every change creates a new layer. Finally, as you make changes and new layers are added, you'll always have a top layer (think frosting) that is a writable layer.

Layered filesystem

What makes this truly efficient is that Docker caches the layers the first time we build them. So, let's say that we have an image with Ubuntu and then add Apache and build the image. Next, we build MySQL with Ubuntu as the base. The second build will be much faster because the Ubuntu layer is already cached. Essentially, our chocolate and vanilla layers, from the preceding Layered filesystem figure, are already baked. We simply need to bake the pistachio (MySQL) layer, assemble, and add the icing (the writable layer).

ThoughtWorks defines Continuous Integration as a development practice that requires developers to integrate code into a shared repository several times a day. By having a continuous process of building and deploying code, organizations are able to instill quality control and testing as part of the everyday work cycle. The result is that updates and bug fixes happen much faster and the overall quality improves.

However, there has always been a challenge in creating development environments that match that of testing and production. Often inconsistencies in these environments make it difficult to gain the full advantage of continuous delivery.

Using Docker, developers are now able to have truly portable deployments. Containers that are deployed on a developer's laptop are easily deployed on an in-house staging server. They are then easily transferred to the production server running in the cloud. This is because Docker builds containers up with build files that specify parent layers. One advantage of this is that it becomes very easy to ensure OS, package, and application versions are the same across development, staging, and production environments.

Because all the dependencies are packaged into the layer, the same host server can have multiple containers running a variety of OS or package versions. Further, we can have various languages and frameworks on the same host server without the typical dependency clashes we would get in a virtual machine (VM) with a single operating system.

The well-defined isolation and layer filesystem also make containers ideal for running systems with a very small footprint and domain-specific purposes. A streamlined deployment and release process means we can deploy quickly and often. As such, many companies have reduced their deployment time from weeks or months to days and hours in some cases. This development life cycle lends itself extremely well to small, targeted teams working on small chunks of a larger application.

Life cycle processes alone are an important piece of operations and management. How will we automatically recover when a container fails? Which upstream services are affected by such an outage? How will we patch our applications with minimal downtime? How will we scale up our containers and services as our traffic grows?

Networking and processing are also important concerns. Some processes are part of the same service and may benefit from the proximity to the network. Databases, for example, may send large amounts of data to a particular microservice for processing. How will we place containers near each other in our cluster? Is there common data that needs to be accessed? How will new services be discovered and made available to other systems?

Resource utilization is also a key. The small footprint of containers means that we can optimize our infrastructure for greater utilization. Extending the savings started in the elastic cloud will take us even further toward minimizing wasted hardware. How will we schedule workloads most efficiently? How will we ensure that our important applications always have the right resources? How can we run less important workloads on spare capacity?

Finally, portability is a key factor in moving many organizations to containerization. Docker makes it very easy to deploy a standard container across various operating systems, cloud providers, and on-premise hardware or even developer laptops. However, we still need tooling to move containers around. How will we move containers between different nodes on our cluster? How will we roll out updates with minimal disruption? What process do we use to perform blue-green deployments or canary releases?

Whether you are starting to build out individual microservices and separating concerns into isolated containers or if you simply want to take full advantage of the portability and immutability in your application development, the need for management and orchestration becomes clear. This is where orchestration tools such as Kubernetes offer the biggest value.

Open a browser and run the following code:

https://<your master ip>/ui/

The certificate is self-signed by default, so you'll need to ignore the warnings in your browser before proceeding. After this, we will see a login dialog. This is where we use the credentials listed during the K8s installation. We can find them at any time by simply using the config command:

$ kubectl config view

Now that we have credentials for login, use those, and we should see a dashboard like the following image:

Kubernetes UI dashboard

The main dashboard takes us to a page with not much display at first. There is a link to deploy a containerized app that will take you to a GUI for deployment. This GUI can be a very easy way to get started deploying apps without worrying about the YAML syntax for Kubernetes. However, as your use of containers matures, it's good practice to use the YAML definitions that are checked in to source control.

If you click on the Nodes link on the left-hand side menu, you will see some metrics on the current cluster nodes:

Kubernetes Node Dashboard

At the top, we see an aggregate of the CPU and memory usages followed by a listing of our cluster nodes. Clicking on one of the nodes will take us to a page with detailed information about that node, its health, and various metrics.

The Kubernetes UI has a lot of other views that will become more useful as we start launching real applications and adding configurations to the cluster.

Another service installed by default is Grafana. This tool will give us a dashboard to view metrics on the cluster nodes. We can access it using the following syntax in a browser: https://<your master ip>/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana

Kubernetes Grafana dashboard

From the main page, click on the Home dropdown and select Cluster. Here, Kubernetes is actually running a number of services. Heapster is used to collect the resource usage on the pods and nodes and stores the information in InfluxDB. The results, such as CPU and memory usage, are what we see in the Grafana UI. We will explore this in depth in Chapter 8, Monitoring and Logging.

The kubectl script has commands to explore our cluster and the workloads running on it. You can find it in the /kubernetes/client/bin folder. We will be using this command throughout the book, so let's take a second to set up our environment. We can do so by putting the binaries folder on our PATH, in the following manner:

$ export PATH=$PATH:/<Path where you downloaded K8s>/kubernetes/client/bin
$ chmod +x /<Path where you downloaded K8s>/kubernetes/client/bin

Now that we have kubectl on our path, we can start working with it. It has quite a few commands. Since we have not spun up any applications yet, most of these commands will not be very interesting. However, we can explore with two commands right away.

First, we have already seen the cluster-info command during initialization, but we can run it again at any time with the following command:

$ kubectl cluster-info

Another useful command is get. It can be used to see currently running services, pods, replication controllers, and a lot more. Here are the three examples that are useful right out of the gate:

$ kubectl get nodes

$ kubectl get events

$ kubectl get services

To start with, we will only see one service, named kubernetes. This service is the core API server for the cluster.

Let's dig a little bit deeper into our new cluster and its core services. By default, machines are named with the kubernetes- prefix. We can modify this using $KUBE_GCE_INSTANCE_PREFIX before a cluster is spun up. For the cluster we just started, the master should be named kubernetes-master. We can use the gcloud command-line utility to SSH into the machine. The following command will start an SSH session with the master node. Be sure to substitute your project ID and zone to match your environment. Also, note that you can launch SSH from the Google Cloud console using the following syntax:

$ gcloud compute ssh --zone "<your gce zone>" "kubernetes-master"

Once we are logged in, we should get a standard shell prompt. Let's run the docker command that filters for Image and Status:

$ sudo docker ps --format 'table {{.Image}}t{{.Status}}' 

Master container listing

Even though we have not deployed any applications on Kubernetes yet, we note that there are several containers already running. The following is a brief description of each container:

To exit the SSH session, simply type exit at the prompt.

We could SSH to one of the minions, but since Kubernetes schedules workloads across the cluster, we would not see all the containers on a single minion. However, we can look at the pods running on all the minions using the kubectl command:

$ kubectl get pods

Since we have not started any applications on the cluster yet, we don't see any pods. However, there are actually several system pods running pieces of the Kubernetes infrastructure. We can see these pods by specifying the kube-system namespace. We will explore namespaces and their significance later, but for now, the --namespace=kube-system command can be used to look at these K8s system resources, as follows:

$ kubectl get pods --namespace=kube-system

We should see something similar to the following:

etcd-empty-dir-cleanup-kubernetes-master 
etcd-server-events-kubernetes-master 
etcd-server-kubernetes-master 
fluentd-cloud-logging-kubernetes-master 
fluentd-cloud-logging-kubernetes-minion-group-xxxx
heapster-v1.2.0-xxxx 
kube-addon-manager-kubernetes-master 
kube-apiserver-kubernetes-master 
kube-controller-manager-kubernetes-master 
kube-dns-xxxx 
kube-dns-autoscaler-xxxx 
kube-proxy-kubernetes-minion-group-xxxx 
kube-scheduler-kubernetes-master 
kubernetes-dashboard-xxxx 
l7-default-backend-xxxx 
l7-lb-controller-v0.8.0-kubernetes-master 
monitoring-influxdb-grafana-xxxx 
node-problem-detector-v0.1-xxxx 
rescheduler-v0.2.1-kubernetes-master

The first six lines should look familiar. Some of these are the services we saw running on the master and will see pieces of these on the nodes. There are a few additional services we have not seen yet. The kube-dns option provides the DNS and service discovery plumbing, kubernetes-dashboard-xxxx is the user interface for Kubernetes, l7-default-backend-xxxx provides the default load balancing backend for the new Layer-7 load balancing capability, and heapster-v1.2.0-xxxx and monitoring-influx-grafana provide the Heapster database and user interface to monitor resource usage across the cluster. Finally, kube-proxy-kubernetes-minion-group-xxxx  is the proxy which directs traffic to the proper backing services and pods running on our cluster.

If we did SSH into a random minion, we would see several containers that run across a few of these pods. A sample might look like this image:

Minion container listing

Again, we saw a similar line up of services on the master. The services we did not see on the master include the following:

Alright, this is our first cluster on GCE, but let's explore some other providers. To keep things simple, we need to remove the one we just created on GCE. We can tear down the cluster with one simple command:

$ kube-down.sh

You just had a little taste of running the cluster on AWS. For the remainder of the book, I will be basing my examples on a GCE cluster. For the best experience following along, you can get back to a GCE cluster easily.

Simply tear down the AWS cluster, as follows:

$ kube-down.sh

Then, create a GCE cluster again using the following:

$ export KUBERNETES_PROVIDER=gce
$ kube-up.sh

It's worth getting to know the parameters used for the kube-up.sh script. Each provider under the kubernetes/cluster/ folder has its own su  folder which containers a config-default.sh  script.

For example, kubernetes/cluster/aws/config-default.sh has the default settings for using kube-up.sh with AWS. At the start of this script, you will see many of these values defined as well as environment variables that can be used to overrides the defaults.

In the following example, the ZONE variable is set for the script and it uses the value from the environment variable named KUBE_AWS_ZONE. If this variable is not set, it will use the default us-west-2a:

ZONE=${KUBE_AWS_ZONE:-us-west-2a}

Understanding these parameters will help you get a lot more mileage out of your kube-up.sh script.

The kube-up.sh script is still a pretty handy way to get started using Kubernetes on your platform of choice. However, it's not without flaws and can sometimes run aground when conditions are not just so.

Luckily, since K8's inception, a number of alternative methods for creating clusters have emerged. Two such GitHub projects are KOPs and kube-aws. While the later is tied to AWS, they both provide an alternative method to easily spinning up your new cluster:

Additionally, a number of managed services have arisen including Google Container Engine (GKE) and Microsoft Azure Container Service (ACS), which provide an automated install and some managed cluster operations. We will look at a brief demo of these in Chapter 12,Towards Production Ready.

Finally, there is the option to start from scratch. Luckily, starting in 1.4, the Kubernetes team has put a major focus in easing the cluster setup process. To that end they have introduced kubeadm for Ubuntu 16.04, CentOS 7, and HypriotOS v1.0.1+.

Let's take a quick look at spinning up a cluster on AWS from scratch using the kubeadm tool.

   $ apt-get update 
   $ apt-get install -y apt-transport-https

   $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg |
   apt-key add -

   $ vi /etc/apt/sources.list.d/kubernetes.list

   deb http://apt.kubernetes.io/ kubernetes-xenial main

   $ apt-get update

   $ apt-get install -y docker.io
   $ apt-get install -y kubelet kubeadm kubectl kubernetes-cni

$ kubeadm init

$ kubeadm join --token=<some token> <master ip address>

Networking

Our cluster will need a networking layer for the pods to communicate on. Note that kubeadm requires a CNI compatible network fabric. The list of plugins currently available can be found here:

http://kubernetes.io/docs/admin/addons/

For our example, we will use calico. We will need to create the calico components on our cluster using the following yaml. For convenience you can download it here:

http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/hosted/kubeadm/calico.yaml

Once you have this file on your master, create the components with the following command:

$ kubectl apply -f calico.yaml

Give this a minute to run setup and then list the kube-system nodes to check:

$ kubectl get pods --namespace=kube-system

You should get a listing similar to the following one with three new calico pods and one completed job that is not shown:

Calico setup

Joining the cluster

Now we need to run the join command we copied earlier, on each of our node instances:

$ kubeadm join --token=<some token> <master ip address>

Once you've finished that, you should be able to see all nodes from the master by running this command:

$ kubectl get nodes

If all went well, this will show three nodes and one master, as shown here:

Calico setup