This chapter will give a brief overview of containers and how they work as well as why management and orchestration is important to your business and/or project team. The chapter will also give a brief overview of how Kubernetes orchestration can enhance our container management strategy and how we can get a basic Kubernetes cluster up, running, and ready for container deployments.
This chapter will include the following topics:
Introducing container operations and management
Why container management is important
Advantages of Kubernetes
Downloading the latest Kubernetes
Installing and starting up a new Kubernetes cluster
Docker lies at the heart of the mass adoption and the excitement in the container space. As Malcom Mclean revolutionized the physical shipping world in 1957 by creating a standardized shipping container, which is used today for everything from ice cube trays to automobiles1, Linux containers are revolutionizing the software development world by making application environments portable and consistent across the infrastructure landscape. As an organization, Docker has taken the existing container technology to a new level by making it easy to implement and replicate across environments and providers.
Control groups (cGroups) work by allowing the host to share and also limit the resources each process or container can consume. This is important for both, resource utilization and security, as it prevents denial-of-service attacks on the host's hardware resources. Several containers can share CPU and memory while staying within the predefined constraints.
Namespaces offer another form of isolation in the way of processes. Processes are limited to see only the process ID in the same namespace. Namespaces from other system processes would not be accessible from a container process. For example, a network namespace would isolate access to the network interfaces and configuration, which allows the separation of network interfaces, routes, and firewall rules.
Union file systems are also a key advantage to using Docker containers. The easiest way to understand union file systems is to think of them like a layer cake with each layer baked independently. The Linux kernel is our base layer; then, we might add an OS like Red Hat Linux or Ubuntu. Next, we might add an application like Nginx or Apache. Every change creates a new layer. Finally, as you make changes and new layers are added, you'll always have a top layer (think frosting) that is a writable layer.
What makes this truly efficient is that Docker caches the layers the first time we build them. So, let's say that we have an image with Ubuntu and then add Apache and build the image. Next, we build MySQL with Ubuntu as the base. The second build will be much faster because the Ubuntu layer is already cached. Essentially, our chocolate and vanilla layers, from Figure 1.2, are already baked. We simply need to bake the pistachio (MySQL) layer, assemble, and add the icing (writable layer).
Wikipedia defines Continuous Integration as "the practice, in software engineering, of merging all developer working copies to a shared mainline several times a day." By having a continuous process of building and deploying code organizations are able to instill quality control and testing as part of the everyday work cycle. The result is that updates and bug fixes happen much faster and overall quality improves.
However, there has always been a challenge in setting development environments to match that of testing and production. Often inconsistencies in these environments make it difficult to gain the full advantage of continuous delivery.
Using Docker, developers are now able to have truly portable deployments. Containers that are deployed on a developer's laptop are easily deployed on an in-house staging server. They are then easily transferred to the production server running in the cloud. This is because Docker builds containers up with build files that specify parent layers. One advantage of this is that it becomes very easy to ensure OS, package, and application versions are the same across development, staging, and production environments.
Because all the dependencies are packaged into the layer, the same host server can have multiple containers running a variety of OS or package versions. Further, we can have various languages and frameworks on the same host server without the typical dependency clashes we would get in a Virtual Machine (VM) with a single operating system.
The well-defined isolation and layer filesystem also make containers ideal for running systems with a very small footprint and domain-specific purposes. A streamlined deployment and release process means we can deploy quickly and often. As such, many companies have reduced their deployment time from weeks or months to days and hours in some cases. This development life cycle lends itself extremely well to small, targeted teams working on small chunks of a larger application.
As we break down an application into very specific domains, we need a uniform way to communicate between all the various pieces and domains. Web services have served this purpose for years, but the added isolation and granular focus that containers bring have paved a way for what is being named microservices.
"In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies."
As the pivot to containerization and microservices evolves in an organization, they will soon need a strategy to maintain many containers and microservices. Some organizations will have hundreds or even thousands of containers running in the years ahead.
Life cycle processes alone are an important piece of operations and management. How will we automatically recover when a container fails? Which upstream services are affected by such an outage? How will we patch our applications with minimal downtime? How will we scale up our containers and services as our traffic grows?
Networking and processing are also important concerns. Some processes are part of the same service and may benefit from proximity on the network. Databases, for example, may send large amounts of data to a particular microservice for processing. How will we place containers near each other in our cluster? Is there common data that needs to be accessed? How will new services be discovered and made available to other systems?
Resource utilization is also a key. The small footprint of containers means that we can optimize our infrastructure for greater utilization. Extending the savings started in the elastic cloud world even further towards minimizing wasted hardware. How will we schedule workloads most efficiently? How will we ensure that our important applications always have the resources? How can we run less important workloads on spare capacity?
Finally, portability is a key factor in moving many organizations to containerization. Docker makes it very easy to deploy a standard container across various operating systems, cloud providers, and on-premise hardware, or even developer laptops. However, we still need tooling to move containers around. How will we move containers between different nodes on our cluster? How will we roll out updates with minimal disruption? What process do we use to perform blue-green deployments or canary releases?
Whether you are starting to build out individual microservices and separating concerns into isolated containers or if you simply want to take full advantage of the portability and immutability in your application development, the need for management and orchestration becomes clear.
This is where orchestration tools such as Kubernetes offer the biggest value. Kubernetes (K8s) is an open source project that was released by Google in June, 2014. Google released the project as part of an effort to share their own infrastructure and technology advantage with the community at large.
Google launches 2 billion containers a week in their infrastructure and has been using container technology for over a decade. Originally they were building a system named Borg, and now Omega, to schedule their vast quantities of workloads across their ever-expanding data center footprint. They took many of the lessons they learned over the years and rewrote their existing data center management tool for wide adoption by the rest of the world. The result was the Kubernetes open source project3.
Since its initial release in 2014, K8s has undergone rapid development with contributions all across the open source community, including Red Hat, VMware, and Canonical. The 1.0 release of Kubernetes went live in July, 2015. We'll be covering version 1.0 throughout the book. K8s gives organizations a tool to deal with some of the major operations and management concerns. We will explore how Kubernetes helps deal with resource utilization, high availability, updates, patching, networking, service discovery, monitoring, and logging.
Kubernetes is supported on a variety of platforms and OSes. For the examples in this book, I used an Ubuntu 14.04 Linux VirtualBox for my client and Google Compute Engine (GCE) with Debian for the cluster itself. We will also take a brief look at a cluster running on Amazon Web Services (AWS) with Ubuntu.
Most of the concepts and examples in this book should work on any installation of a Kubernetes cluster. To get more information on other platform setups, check the Kubernetes getting started page on the following GitHub link:
Start by updating packages:
$ sudo apt-get update
Install Python and curl if they are not present:
$ sudo apt-get install python $ sudo apt-get install curl
Install the gcloud SDK:
$ curl https://sdk.cloud.google.com | bash
$ gcloud auth login
If you have problems with login or want to use another browser, you can optionally use the
--no-launch-browser command. Copy and paste the URL to the machine and/or browser of your choice. Log in with your Google Cloud credentials and click on Allow on the permissions page. Finally, you should receive an authorization code that you can copy and paste back into the shell where the prompt is waiting.
$ gcloud config list project
We can modify this and set a new default project with this command. Make sure to use project ID and not project name, as follows:
$ gcloud config set project <PROJECT ID>
We can find our project ID in the console at:
Alternatively, we can list active projects:
$ gcloud alpha projects list
Now that we have our environment set up, installing the latest Kubernetes version is done in a single step as follows:
$ curl -sS https://get.k8s.io | bash
It may take a minute or two to download Kubernetes depending on your connection speed. After this, it will automatically call the
kube-up.sh script and start building our cluster. By default, it will use the Google Cloud and GCE.
If something fails during the cluster setup and you need to start again, you can simply run the
kube-up.sh script. Go to the folder where you ran the previous
curl command. Then, you can kick off the cluster build with the following command:
After Kubernetes is downloaded and the
kube-up.sh script has started, we will see quite a few lines roll past. Let's take a look at them one section at a time.
The preceding section (Figure 1.3) shows the checks for prerequisites as well as makes sure that all components are up to date. This is specific to each provider. In the case of GCE, it will check that the SDK is installed and that all components are up to date. If not, you will see a prompt at this point to install or update.
Now the script is turning up the cluster. Again, this is specific to the provider. For GCE, it first checks to make sure that the SDK is configured for a default project and zone. If they are set, you'll see those in the output.
Next, it uploads the server binaries to Google Cloud storage, as seen in the Creating gs:\\... lines.
It then checks for any pieces of a cluster already running. Then, we finally start creating the cluster. In the output in Figure 1.5, we see it creating the master server, IP address, and appropriate firewall configurations for the cluster.
Finally, it creates the minions or nodes for our cluster. This is where our container workloads will actually run. It will continually loop and wait while all the minions start up. By default, the cluster will have four node (minions), but K8s supports having upwards of 100 (and soon beyond 1000). We will come back to scaling the nodes later on in the book.
Now that everything is created, the cluster is initialized and started. Assuming that everything goes well, we will get an IP address for the master. Also, note that configuration along with the cluster management credentials are stored in
Then, the script will validate the cluster. At this point, we are no longer running provider-specific code. The validation script will query the cluster via the
kubectl.sh script. This is the central script for managing our cluster. In this case, it checks the number of minions found, registered, and in a ready state. It loops through giving the cluster up to 10 minutes to finish initialization.
After a successful startup, a summary of the minions and the cluster component health is printed to the screen:
<your master ip>
The certificate is self-signed by default, so you'll need to ignore the warnings in your browser before proceeding. After this, we will see a login dialog. This is where we use the credentials listed during the K8s installation. We can find them at any time by simply using the
$ kubectl config view
Now that we have credentials for login, use those, and we should see a dashboard like the following image:
The main dashboard page gives us a summary of the minions (or slave nodes). We can also see the CPU, memory, and used disk space on each minion as well the IP address.
The UI has a number of built-in views listed under the Views dropdown menu on the top right of the screen. However, most of them will be empty by default. Once workloads and services are spun up, these views will become a lot more interesting.
<your master ip>
Here, Kubernetes is actually running a number of services. Heapster is used to collect resource usage on the pods and nodes and stores the information in InfluxDB. The results, like CPU and memory usage, are what we see in the Grafana UI. We will explore this in depth in Chapter 6, Monitoring and Logging.
Swagger (http://swagger.io/) is a tool to add a higher level of interaction and easy discovery to an API.
Through this interface, you can learn a lot about the Kubernetes RESTful API. The bulk of the interesting endpoints are listed under
v1. If we look at
/api/v1/nodes, we can see the structure of the JSON response as well as details of possible parameters for the request. In this case, we see that the first parameter is
pretty, which toggles whether the JSON is returned with pretty indentation for easier reading.
We can try this out by using
<your master ip>
By default, we'll see a JSON response with pretty indentation enabled. The response should have a list of all the nodes currently in our cluster.
Now, let's try tweaking the
pretty request parameter you just learned about. Use
<your master ip>
Now we have the same response output, but with no indentation. This is a great resource for exploring the API and learning how to use various function calls to get more information and interact with your cluster programmatically.
kubectl.sh script has commands to explore our cluster and the workloads running on it. We will be using this command throughout the book, so let's take a second to set up our environment. We can do so by making the script executable and putting it on our
PATH, in the following manner:
$ cd /home/<Username>/kubernetes/cluster $ chmod +x kubectl.sh $ export PATH=$PATH:/home/<Username>/kubernetes/cluster $ ln -s kubectl.sh kubectl
You may choose to download the
kubernetes folder outside your home folder, so modify the preceding commands as appropriate.
It is also a good idea to make the changes permanent by adding the
export command to the end of your
.bashrc file in your home directory.
Now that we have
kubectl on our path, we can start working with it. It has quite a few commands. Since we have not spun up any applications yet, most of these commands will not be very interesting. However, we can explore with two commands right away.
First, we have already seen the
cluster-info command during initialization, but we can run it again at any time with the following:
$ kubectl cluster-info
Another useful command is
get command can be used to see currently running services, pods, replication controllers, and a lot more. Here are the three examples that are useful right out of the gate:
Listing the nodes in our cluster:
$ kubectl get nodes
List cluster events:
$ kubectl get events
Finally, we can see any services that are running in the cluster as follows:
$ kubectl get services
To start with, we will only see one service, named
kubernetes. This service is the core API server, monitoring and logging services for the pods and cluster.
Let's dig a little bit deeper into our new cluster and its core services. By default, machines are named with the
kubernetes- prefix. We can modify this using
$KUBE_GCE_INSTANCE_PREFIX before a cluster is spun up. For the cluster we just started, the master should be named
kubernetes-master. We can use the
gcloud command-line utility to SSH into the machine. The following command will start an SSH session with the master node. Be sure to substitute your project ID and zone to match your environment. Also, note that you can launch SSH from the Google Cloud console using the following syntax:
$ gcloud compute --project "<Your project ID>" ssh --zone "<your gce zone>" "kubernetes-master"
Once we are logged in, we should get a standard shell prompt. Let's run the familiar
sudo docker ps command.
fluentd-gcp: This container collects and sends the cluster logs file to the Google Cloud Logging service.
kube-ui: This is the UI that we saw earlier.
kube-controller-manager: The controller manager controls a variety of cluster functions. Ensuring accurate and up-to-date replication is one of its vital roles. Additionally, it monitors, manages, and discovers new nodes. Finally, it manages and updates service endpoints.
kube-apiserver: This container runs the API server. As we explored in the Swagger interface, this RESTful API allows us to create, query, update, and remove various components of our Kubernetes cluster.
kube-scheduler: The scheduler takes unscheduled pods and binds them to nodes based on the current scheduling algorithm.
etcd: This runs the etcd software built by CoreOS.
etcdis a distributed and consistent key-value store. This is where the Kubernetes cluster state is stored, updated, and retrieved by various components of K8s.
Pausecontainer is often referred to as the pod infrastructure container and is used to set up and hold the networking namespace and resource limits for each pod.
To exit the SSH session, simply type
exit at the prompt.
We could SSH to one of the minions, but since Kubernetes schedules workloads across the cluster, we would not see all the containers on a single minion. However, we can look at the pods running on all the minions using the
$ kubectl get pods
Since we have not started any applications on the cluster yet, we don't see any pods. However, there are actually several system pods running pieces of the Kubernetes infrastructure. We can see these pods by specifying the
kube-system namespace. We will explore namespaces and their significance later, but for now, the
--namespace=kube-system command can be used to look at these K8s system resources as follows:
$ kubectl get pods --namespace=kube-system
We should see something similar to the following:
etcd-server fluentd-cloud-logging kube-apiserver kube-controller-manager kube-scheduler kube-ui kube-dns monitoring-heapster monitoring-influx-grafana
The first six should look familiar. These are additional pieces of the services we saw running on the master. The final three are services we have not seen yet.
kube-dns provides the DNS and service discovery plumbing.
monitoring-heapster is the system used to monitor resource usage across the cluster.
monitoring-influx-grafana provides the database and user interface we saw earlier for monitoring the infrastructure.
If we did SSH into a random minion, we would see several containers that run across a few of these pods. A sample might look like the image here:
skydns: This uses DNS to provide a distributed service discovery utility that works with
kube2Sky: This is the connector between
kubernetes. Services in the API are monitored for changes and updated in
heapster: This does resource usage and monitoring.
exechealthz: This performs health checks on the pods.
By default, Kubernetes uses the GCE provider for Google Cloud. We can override this default by setting the
KUBERNETES_PROVIDER environment variable. The following providers are supported with values listed in Table 1.1:
Google Compute Engine
Google Container Engine
Amazon Web Services
Virtual development environment
Private cloud / on-premise virtualization
Libvirt running CoreOS
Virtualization management tool
Canonical Juju (folks behind Ubuntu)
OS service orchestration tool
Table 1.1. Kubernetes providers
Let's try setting up the cluster on AWS. As a prerequisite, we need to have the AWS Command Line Interface (CLI) installed and configured for our account. AWS CLI Installation and configuration documentation can be found here:
Installation documentation: http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os
Configuration documentation: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
Then, it is a simple environment variable setting as follows:
$ export KUBERNETES_PROVIDER=aws
Again, we can use the
kube-up.sh command to spin up the cluster as follows:
As with GCE, the setup activity will take a few minutes. It will stage files in S3, create the appropriate instances, Virtual Private Cloud (VPC), security groups, and so on in our AWS account. Then, the Kubernetes cluster will be set up and started. Once everything is finished and started, we should see the cluster validation at the end of the output.
Once again, we will SSH into master. This time, we can use the native SSH client. We'll find the key files in
$ ssh -v -i /home/<username>/.ssh/kube_aws_rsa [email protected]<Your master IP>
sudo docker ps to explore the running containers. We should see something like the following:
<your master ip>
That is a little taste of running the cluster on AWS. For the remainder of the book, I will be basing my examples on a GCE cluster. For the best experience following along, you can get back to a GCE cluster easily.
Simply tear down the AWS cluster as follows:
Then, create a GCE cluster again using following:
$ export KUBERNETES_PROVIDER=gce $ kube-up.sh
We took a very brief look at how containers work and how they lend themselves to the new architecture patterns in microservices. You should now have a better understanding of how these two forces will require a variety of operations and management tasks and how Kubernetes offers strong features to address these challenges. Finally, we created two different clusters on both GCE and AWS and explored the startup script as well as some of the built-in features of Kubernetes.
In the next chapter, we will explore the core concept and abstractions K8s provides to manage containers and full application stacks. We will also look at basic scheduling, service discovery, and health checking.
1Malcom McLean entry on Wikipedia: https://en.wikipedia.org/wiki/Malcom_McLean
2Martin Fowler on microservices: http://martinfowler.com/articles/microservices.html
3Kubernetes GitHub project page: https://github.com/kubernetes/kubernetes