It's no mystery that Docker is one of the open-source technologies that have the most traction nowadays. The reasons are easy to understand, Docker makes the container technology available for all, and it comes with an included battery that is removable and is blessed by a vibrant community.
In the early days, users started working with Docker after being fascinated with this easy-to-use tool, which allowed them to sort out many challenges: pulling, packing, isolating, and making applications portable across systems with almost no burden.
A simplified Docker Ecosystem
You may notice a swarm of whales here plays nice with others. However, since the advent of containers, people have been looking for tools to efficiently orchestrate a huge number of them. The Docker team addressed this necessity with the release of Docker Swarm, hereinafter Swarm, one of the pieces of the Docker ecosystem, in 2015, alongside with Docker Machine and Docker Compose. The preceding image shows the simplified Docker Ecosystem, which consists of Docker Machine provisioning a new Docker-ready machine, then a set of machines will be formed into a Docker Swarm cluster. Later we will be able to use Docker Compose to deploy containers to the cluster, as if it were a normal Docker Engine.
The plan to make a cluster management system, natively, for Docker started in early 2014, as a communication protocol project called Beam. Later, it was implemented as a daemon to control heterogeneous distributed systems with the Docker API. The project had been renamed to libswarm and Swarmd is its daemon. Keeping the same concept of allowing any Docker client to connect to a pool of Docker Engines, the third generation of the project had been re-designed to use the same set of Docker Remote APIs and renamed to "Swarm" in November 2014. Basically, the most important part of Swarm are its remote APIs; the maintainers work hard to keep them 100% compatible with every version of Docker Engine. We'll call the first generation of Swarm as "Swarm v1".
In February 2016, after the core team found the scaling limitation of the centralized service, Swarm has been internally redesigned again as swarm.v2. This time, a decentralized cluster design has been taken into account. In June 2016, SwarmKit had been released as the orchestration toolkit for distributed service at any scale. Docker announced that SwarmKit was merged into Docker Engine at DockerCon 2016. We'll refer to this version of Swarm as "Swarm v2" or "Swarm mode".
As we'll see later, these three musketeers (Docker Swarm, Docker Machine, and Docker Compose) operate best when together and they are so seamlessly intertwined with each other that it is almost impossible to think of them as single pieces.
However, even despite this Machine and Compose are really direct in their goals and easy to use and understand, Swarm is a tool that indeed deserves a book for itself.
With Docker Machine, you can provision machines, both virtual and physical, on a number of cloud platforms as well as bare metal machines to run Docker containers. With Docker Compose, you can define Dockerfiles on steroids, by describing behaviors with the easy yet powerful syntax of YAML and launch applications by just "composing up" these files. Swarm is a powerful clustering tool that requires to be studied more in depth.
In this chapter, we will be taking a look at the following topics:
What is container orchestration
Docker Swarm fundamentals and architecture
Differences with other open source orchestrators
The "old" Swarm, v1
The "new" Swarm, Swarm Mode
A clustering tool is software that allows an operator to talk to a single end point and to command and orchestrate a set of resources, in our case containers. Instead of manually distributing workloads (containers) on a cluster, a clustering tool is used to automate this and many other tasks. It's the clustering tool that will decide where to start jobs (containers), how to store them, when to eventually restart them, and so on. The operator needs to only configure some behaviors, decide the cluster topology and size, tune settings, and enable or disable advanced features. Docker Swarm is an example of clustering tool for containers.
Beyond clustering tools, there is also a choice of container manager platforms. They do not provide container hosting, but interact with one or more existing systems; this kind of software usually offer good web interfaces, monitoring tools, and other visual or higher-level functionalities. Examples of container manager platforms are Rancher or Tutum (acquired in 2015 by Docker Inc.).
Swarm is described by Docker itself as:
Docker Swarm is native clustering for Docker. It turns a pool of Docker hosts into a single, virtual Docker host.
Swarm is a tool that gives you the illusion to manage a single-huge Docker host made of many Docker hosts, as they were one and had one command-entry point. It allows you to orchestrate and operate a certain number of containers on these hosts, using the routine Docker tools, either using the Docker native or the python-docker client, even curl with the Docker Remote APIs.
This is what a minimal Swarm cluster in production looks similar to:
There are many reasons to use a clustering solution for your containers. As you will see your applications growing, you will face new mandatory requirements, such as scalability, manageability, and high availability.
There are many tools available out there; picking up Docker Swarm gives us some immediate advantages:
Native clustering: Swarm is a native in Docker and made by the Docker team and community. Its original creators are Andrea Luzzardi and Victor Vieux, who are the early implementers of Docker Engine Remote API itself. Swarm integrates, with no additional requirements, with Machine, Compose, and the other tools from the ecosystem.
Production grade: Swarm v1 was declared mature in November 2015 and is ready to be used in production. The team already demonstrated that Swarm can scale-up to control Engines that are as large as 1,000 nodes. Swarm v2 allows forming clusters with multi-thousand nodes, as it uses a decentralized discovery.
Work out-of-the-box: Swarm does not require you to re-architect your app to adapt to another orchestration tool. You can use your Docker images and configurations with no changes and deploy them at a scale.
Easy to setup and use: Swarm is easy to operate. Effective deployments can be done by just adding some flags to Machine commands or using Docker commands since Docker 1.12. Discovery service is integrated into Swarm Mode, making it quick to install: There is no need to set up external Consul, Etcd, or Zookeeper clusters.
Active community: Swarm is a vibrant project, with a very active community and is under heavy development.
Available on Hub: You don't need to install Swarm, it comes ready as a Docker image (Swarm v1), and so you just pull and run it from the Hub or integrated into the Docker Engine. While Swarm Mode is already integrated into Docker 1.12+. That's all.
Docker Swarm is the choice of several projects, for example:
Rackspace Carina is built atop Docker Swarm: Rackspace offers hosted container environment, which is internally based on Docker Swarm
Zenly is using Swarm across Google Cloud Platform and bare metal servers
ADP uses Docker and Swarm to give velocity to their legacy deployments
Swarms can be deployed with Amazon AWS and Microsoft Azure templates directly on their public clouds
There are two opposite approaches when creating and utilizing infrastructures: pet versus cattle.
In the pet model, the administrator deploys servers or virtual machines or, in our case, containers and takes care of them. She or he logs in, installs software, configures it, and ensures that everything is working fine. As a result, this is her or his pet.
By contrast, the administrator doesn't really care about the destiny of his infrastructural components, when thinking of them as cattles. She or he doesn't log in to every single unit or handle it manually, rather, uses a bulk approach, deployment, configuration, and management are done with automation tools. If a server or container dies, it's automatically resurrected, or another is generated to substitute for the defunct. As a result, the operator is handling cattle.
In this book, we'll use the pet model in the very first chapter to introduce some basic concepts to the reader. But we'll follow the cattle pattern later, when it will be the time to do serious things.
The main purpose of Swarm was already defined, but how does it accomplish its goals? Here are its key features:
Swarm v1 supports Docker Engine of version 1.6.0 or more recent. Swarm v2 has been in built for Docker Engine since version 1.12.
APIs of each release of Swarm will be compatible with Docker APIs on the same release train. API compatibility is maintained for one version backward.
In Swarm v1, the leader-election mechanism is implemented for multiple Swarm masters using the leadership library (only supported when deploying Swarm with a discovery service, such as Etcd, Consul, or Zookeeper).
In Swarm v2, leader election has been built using the decentralized mechanism. Swarm v2 does not need a dedicated set of discovery services anymore because it integrates Etcd, an implementation of the Raft consensus algorithm (see Chapter 2, Discover the Discovery Services).
In the Swarm v1 terminology, the leader Swarm master is called primary, where others are called replica. In Swarm v2, there is a concept of Master and Worker nodes. While the leader nodes are managed automatically by the cluster using Raft.
Basic and advanced scheduling options. The scheduler is an algorithm that decides the hosts on which the containers must be physically placed. Swarm comes with a set of built-in schedulers.
Constraints and affinities to let the operator take decisions on scheduling; for example, one wants to keep the database containers geographically near and suggest the scheduler to do that. Constraints and affinities use Docker Swarm labels.
In Swarm v2, in-cluster load balancing is implemented with the built-in DNS Round-Robin, while it supports external load balancing via the routing mesh mechanism, which is implemented over IPVS.
High-availability and failover mechanism means that you can create a Swarm with more than one master; so if they go down, there will be other master/s ready to take control. Swarm v2 is available, by default, when we form a cluster of at least 3 nodes. All nodes can be the master nodes. Also, Swarm v2 includes health indicator information.
We have more than only Docker Swarm out there to clusterize containers. For completeness, we will briefly review the most widely known open source alternatives, before diving completely into Swarm.
Kubernetes (http://kubernetes.io), also known as k8s, aims at the same goal of Docker Swarm; it's a manager for cluster of containers. Started originally as project Borg in Google laboratories, it was later open sourced and released as a stable version in 2015, supporting Google Cloud Platform, CoreOS, Azure, and vSphere.
Kubernetes so far runs containers in Docker, which is commanded via API by a so called Kubelet, a service that registers and manages Pods. Architecturally, Kubernetes divides its clusters, logically, not into bare containers but into Pods. A Pod is the smallest deployable unit and is physically a representation of an application made by a group of one or more containers, usually collocated, that share resources such as storage and networking (users can simulate Pods in Docker using Compose and starting from Docker 1.12 create Docker DABs (Distributed Application Bundles)).
Kubernetes includes some expected basic clustering features, such as labels, health checkers, Pods registry, has configurable schedulers, and services such as ambassadors or load balancers.
In practice, the Kubernetes user utilizes the kubectl client to interface to the Kubernetes master, the cluster controlling unit that commands the Kubernetes nodes doing some work, called minions. Minions run Pods and everything is glued by Etcd.
On a Kubernetes node, you will find a running Docker Engine, which runs a kube-api container, and a system service called kubelet.service.
There are a of kubectl commands that are pretty intuitive, such as
kubectl cluster-info,kubectl get pods, andkubectl get nodesto retrieve information about the cluster and its healthkubectl create -f cassandra.yamland any derivative Pod commands, to create, manage, and destroy Podskubectl scale rc cassandra --replicas=2to scale Pods and applicationskubectl label pods cassandra env=prodto configure Pod labels
This is just a high level panoramic of Kubernetes. The main differences between Kubernetes and Docker Swarm are:
Swarm has a more straightforward architecture to understand. Kubernetes requires more focus, just to grasp its fundamentals. But studying is always good!
Again on architecture: Kubernetes is based on Pods, Swarm on containers, and DABs.
You need to install Kubernetes. By either deploying on GCE, using CoreOS, or on the top of OpenStack, you must take care of it. You must deploy and configure a Kubernetes cluster, and this is some little extra effort. Swarm is integrated into Docker, and requires no extra installations.
Kubernetes has an additional concept of Replication Controllers, a technology that ensure that all the Pods described by some templates are running at a given time.
Both Kubernetes and Swarm use Etcd. But while in Kubernetes it's treated as an external facility service, in Swarm it's integrated and runs on manager nodes.
A performance comparison between Kubernetes and Swarm might take form of holy wars and we want to subtract to this practice. There are benchmarks showing how fast is Swarm in starting containers and other benchmarks showing how fast is Kubernetes in running its workloads. We are of the opinion that benchmark results must always be taken cum grano salis. That said, both Kubernetes and Swarm are suitable for running big, fast, and scalable containers clusters.
Fleet (https://github.com/coreos/fleet) is another possible choice amongst container orchestrators. It comes from the family of CoreOS container products (which includes CoreOS, Rocket, and Flannel) and is basically different from Swarm, Kubernetes, and Mesos in that it's architected as an extension to system. Fleet operates through schedulers to distribute resources and tasks across the cluster nodes. Hence, its goal is not only to provide a pure containers clusterization rather to be a distributed more general elaboration system. It's possible, for example, to run Kubernetes on the top of Fleet.
A Fleet cluster is made of engines responsible for scheduling jobs, other management operations, and agents, running on each host, that are physically executing the jobs they're assigned and reporting the status continuously to engines. Etcd is the discovery services that keeps everything glued.
You interact through a Fleet cluster with its main command fleetctl, with the list, start, and stop containers and services options.
So, summarising, Fleet is different from Docker Swarm:
It's a higher-level abstraction that distributes tasks, it's not a mere container orchestrator.
Think of Fleet as more of a distributed init system for your cluster. Systemd is for one host, Fleet for a cluster of hosts.
Fleet clusterizes specifically a bunch of CoreOS nodes
You can run Kubernetes on the top of Fleet to exploit Fleet features of resiliency and high availability
There are no known stable and robust ways to integrate Fleet and Swarm v1 automatically.
Currently, Fleet is not tested to run clusters with more than 100 nodes and 1000 containers (https://github.com/coreos/fleet/blob/master/Documentation/fleet-scaling.md) while we were able to run Swarms with 2300 and later 4500 nodes.
Whether you can see Fleet as a distributed init system for your cluster, you can think of Mesos (https://mesos.apache.org/) in terms of a distributed kernel. With Mesos, you can make available all your nodes resources as if they were one and, for the scope of this book, run containers clusters on them.
Mesos, originally started at the University of Berkeley in 2009, is a mature project and has been used in production with success, for example by Twitter.
It's even more general purpose than Fleet, being multi-platform (you can run it on Linux, OS X or Windows nodes) and capable of running heterogeneous jobs. You can typically have clusters of containers running on Mesos just aside of pure Big Data jobs (Hadoop or Spark) and others, including continuous integration, real-time processing, web applications, data storage, and even more.
A Mesos cluster is made of one Master, slaves, and frameworks. As you would expect, the master allocates resources and tasks on the slaves, it is responsible for the system communications and runs a discovery service (ZooKeeper). But what are frameworks? Frameworks are applications. A framework is made of a scheduler and an executor, the first one distributes tasks and the second executes them.
For our interest, typically containers are run on Mesos through a framework named Marathon (https://mesosphere.github.io/marathon/docs/native-docker.html).
A comparison between Mesos and Docker Swarm does not make sense here, since they may very well run complementarily, that is Docker Swarm v1 can run on Mesos and a portion of Swarm source code is just dedicated to this. Swarm Mode and SwarmKit, instead, are very similar to Mesos since they abstract jobs in tasks and group them in services, to distribute loads on the cluster. We'll discuss better of SwarmKit features in Chapter 3, Meeting Docker Swarm Mode.
Kubernetes, Fleet and Mesos try to address a similar problem; they provide a layer abstraction for your resources and allow you to interface to a cluster manager. Then you can launch jobs and tasks and the project of your choice will sort it out. The difference can be seen in the features provided out-of-the-box and on how much you can customize the precision of allocating and scaling resources and jobs. Of the three, Kubernetes is more automatic, Mesos more customizable so, from a certain point of view, powerful (if you need all that power, of course).
Kubernetes and Fleet abstract and make default many details that for Mesos are needed to be configured, for example a scheduler. On Mesos, you can use the Marathon or Chronos scheduler or even write your own. If you don't require, don't want or even can't dig deep into those technicalities, you can pick up Kubernetes or Fleet. It depends on your actual and/or forecasted workload.
So, what solution should you adopt? As always, you have a problem and open source is generous enough to make many technologies available that can often intersect on to each other, to help you successfully reach a goal. The problem is how and what to choose to resolve your problem. Kubernetes, Fleet, and Mesos are all powerful and interesting projects and so is Docker Swarm.
In a hypothetic standing of how automatic and simple to understand these four guys are, Swarm is a winner. This is not an advantage always, but in this book we'll show how Docker Swarm can help you to make real things work, bearing in mind that in one of the DockerCon keynotes Solomon Hykes, CTO and Founder of Docker, suggested that Swarm would be a tier that could provide a common interface onto the many orchestration and scheduling frameworks.
This section discusses the overview architecture of Docker Swarm. The internal structure of Swarm is described in Figure 3.
The internal structure of Docker Swarm v1
Starting with the MANAGER part, you will see a block labeled with Docker Swarm API on the left-side of the diagram. As mentioned previously, Swarm exposes the set of remote APIs similar to Docker, which allows you to use any Docker clients to connect to Swarm. However, the Swarm APIs are slightly different from the standard Docker Remote APIs, as Swarm APIs contains cluster-related information too. For example, running docker info against Docker Engine will give you the information of the single Engine, but when we call docker info against a Swarm cluster, we'll also get the number of nodes in the cluster as well as each node's information and health.
The block next to Docker Swarm API is Cluster Abstraction. It is an abstraction layer to allow different kinds of cluster to be implemented as backend of Swarm and share the same set of Docker Remote APIs. Currently we have two cluster backend, the built-in Swarm cluster implementation and the Mesos cluster implementation. Swarm Cluster and Built-in Scheduler blocks represent the built-in Swarm cluster implementation, while the blocks denoted by Mesos Cluster is the Mesos cluster implementation.
The Built-in Scheduler of the Swarm backend comes with a number of Scheduling Strategies. Two strategies are Spread and BinPack, which will be explained in the later chapters. If you're familiar with Swarm, you will note that the Random strategy is missing here. The Random strategy is excluded from the explanation as it is for testing purpose only.
Along with Scheduling Strategies, Swarm employs a set of Scheduling Filters to help screening criteria-unmet nodes out. There are currently six kinds of filter namely, Health, Port, Container Slots, Dependency, Affinity, and Constraint. They are applied to filter when one is scheduling the newly created container in exactly this order.
On the AGENTS part, there are Swarm agents trying to register address of their Engines into the discovery service.
Finally, the centralized piece, DISCOVERY, is to coordinate addresses of the Engines between AGENTS and MANAGER. The agent-based Discovery Service currently uses LibKV, which delegates the discovery function to your key-value store of choices, Consul, Etcd, or ZooKeeper. In contrast, we also can use only Docker Swarm manager without any key-value store. This mode is called agent-less discovery, which are File and Nodes (specify address on the command line).
We will use the agent-less model later in this chapter to create a minimal local Swarm cluster. We'll meet the other discovery services starting in Chapter 2, Discover the Discovery Services and the Swarm Mode architecture in Chapter 3, Meeting Docker Swarm Mode.
Before continuing to other sections, we review some Docker-related terminologies to recall Docker concepts and introduce Swarm keywords.
A Docker Engine is a Docker daemon running on a host machine. Sometimes in the book we'll just refer to it as Engine. We usually start an Engine by calling
docker daemonvia systemd or other start up services.Docker Compose is a tool to describe in YAML how multi-container services must be architected.
Docker stacks are the binary result of creating images of multiple-containers app (described by Compose) instead of single containers.
A Docker daemon is an interchangeable term with Docker Engine.
A Docker client is the client program packed in the same docker executable. For example, when we do
docker run, we are using the Docker client.Docker networking is a Software-defined Network that links a set of containers in the same network together. By default, we'll use the libnetwork (https://github.com/docker/libnetwork) implementation came with Docker Engine. But you can optionally deploy third-party network drivers of your choices using plugins.
Docker Machine is a tool used to create hosts capable of running Docker Engines called machines .
A Swarm node in Swarm v1 is a machine that is a pre-installed Docker Engine and has a Swarm agent program running alongside. A Swarm node will register itself into a Discovery service.
A Swarm master in Swarm v1 is a machine that is running a Swarm manager program. A Swarm master reads addresses of Swarm nodes from its Discovery service.
A Discovery service is a token-based service offered by Docker or a self-hosted one. For the self-hosted ones, you can run HashiCorp Consul, CoreOS Etcd, or Apache ZooKeeper as key-value stores to serve as the discovery service.
Leader Election is a mechanism done by Swarm Masters to find the primary node. Other master nodes will be in the replica role until the primary node goes down and then the leader election process will start again. As we'll see, the number of Swarm masters should be an odd number.
SwarmKit is a new Kit released by Docker to abstract orchestration. Theoretically, it should be able run any kind of service but in practice so far it orchestrates only containers and sets of containers.
Swarm Mode is the new Swarm, available since Docker 1.12, that integrates SwarmKit into the Docker Engine.
Swarm Master (in Swarm Mode) is a node that manages the cluster: It schedules services, keeps the cluster configuration (nodes, roles, and labels) and ensures that there is a cluster leader.
Swarm Worker (in Swarm Mode) is a node which runs tasks, for example, hosts containers.
Services are abstractions of workloads. For example, we can have a service "nginx" replicated 10 times, meaning that you will have 10 tasks (10 nginx containers) distributed on the cluster and load balanced by Swarm itself
Tasks are the unit of work of Swarms. A task is a container.
We'll now proceed with the installation of two small Swarm v1 and v2 proof of concept clusters, the first on local and the second on Digital Ocean. In order to execute the recipes, check the list of ingredients, ensure that you have everything, and then begin.
To follow the example, you'll need:
Either a Windows, Mac OS X, or Linux desktop
A Bash or Bash-compatible shell. On Windows you can either useCygwin or Git Bash.
The latest version of VirtualBox, installed for the local example
At least 4GB of memory for 4 VirtualBox instances of 1G of memory each for the local example
A Docker client, at least version 1.6.0 for Swarm v1 and 1.12 for Swarm v2
The latest version of Docker Machine, which is currently 0.8.1
Docker announced the desktop version of Docker for Mac and Docker for Windows early in 2016. It's better than the Docker Toolbox, since it includes the Docker CLI tools you expect but doesn't use boot2docker and VirtualBox anymore (it uses unikernels instead, which we'll introduce in Chapter 11, What is Next?) and it's fully integrated into the operating system (Mac OS X Sierra or Windows 10 with Hyper-V enabled).
You can download the Docker desktop from https://www.docker.com/products/overview#/install_the_platform and install it easily.
Just drag and drop the Docker beta icon to your applications folder if you're using Mac OS X. Input your beta registration code, if any, and it's done.
On OS X, you will have the Docker whale in the system tray, which you can open and also configure your settings. A Docker host will be running natively on your desktop.
In the case of Docker for Windows, it requires Windows 10 with Hyper-V enabled. Basically, Hyper-V comes with Windows 10 Professional or higher versions. After double-clicking on the setup program, you'll see that the first screen, showing the License Agreement, looks similar to the following screenshot. The setup program will request you for a key similar to that of Docker for Mac.
If the installation process goes smoothly, you will see that the finish screen is ready for you to launch Docker for Windows, as shown:
At the time of launch, Docker will initialize itself to the Hyper-V. Once the process is done, you can just open PowerShell and start using Docker.
If something goes wrong you can open the logging Windows from the tray icon's menu, as well as check with Hyper-V Manager.
We'll extensively use Machine in this book, so ensure that you have it installed through Docker for Mac or Windows or Docker Toolbox. If you use Linux on your desktop, install the Docker client with your package system (apt or rpm). You will also have to download the bare machine binary, just curl it and assign it the execution permissions; follow the instructions at https://docs.docker.com/machine/install-machine/. The current stable version is 0.8.1.
$ curl -L https://github.com/docker/machine/releases/download/v0.8.1/docker- machine-uname -s-uname -m > /usr/local/bin/docker-machine $ chmod +x /usr/local/bin/docker-machine`
You can check that the machine is ready to be used with the following command from the command line:
$ docker-machine --version docker-machine version 0.8.1, build 41b3b25
If you have problems, please control the system paths or download the correct binary for your architecture.
For the very first example, we'll run the easiest possible configuration of a Swarm v1 cluster locally to get a taste of how "old" Swarms worked (and still works). This tiny cluster will have the following features:
Made of four nodes of 1CPU, 1GB of memory each, it will consist of an infrastructure of four CPUs and 4GB of memory available in total
Each node will run on VirtualBox
Each node is connected to each other on the local VirtualBox network
No discovery service is involved: a static
nodes://mechanism will be usedNo security is configured, in other words TLS is disabled
Our cluster will look similar to the following diagram. Four Engines will be connected to each other through port 3376 in a mesh. Beyond the Docker engine, in fact, each of them will run a Docker container exposing port 3376 (Swarm) on host and redirecting it into itself. We, operators, will be able to connect to (any of) the hosts by setting the environment variable DOCKER_HOST to IP:3376. Everything will become clearer if you follow the example step-by-step.
To begin, we must create four Docker hosts with Docker Machine. Docker Machine automates these steps with one command instead of manually creating a Linux VM, generating and uploading certificates, logging into it via SSH, and installing and configuring the Docker daemon.
Machine will perform the following steps:
Spin up a VirtualBox VM starting from the boot2docker image.
Assign the VM an IP on the VirtualBox internal network.
Upload and configure certificates and keys.
Install Docker daemon on this VM.
Configure the Docker daemon and expose it so it will be remotely reachable.
As a result, we'll have a VM running Docker and ready to be accessed to run containers.
Built with Tiny Core Linux, Boot2Docker is a lightweight distribution, which is especially designed for running Docker containers. It's runs completely on RAM and the boot time is extremely fast, around five seconds from start to the console. When starting the Engine, Boot2Docker starts Docker Engine at the secure port 2376 by default.
Boot2Docker is by no mean for the production workload. It's designed for development and testing purpose only. We'll start with using boot2docker then later move on to the production in subsequent chapters. At the time of writing, Boot2Docker supports Docker 1.12.3 and uses Linux Kernel 4.4. It comes with AUFS 4 as the default storage driver for Docker Engine.
If we execute:
$ docker-machine ls
on our new installation to list the available machines, we see that we have no running ones.
So, let's start by creating one, with this command:
$ docker-machine create --driver virtualbox node0
This command specifically asks to use the VirtualBox driver (-d for short) and to name the machine node0. Docker Machines can provision machines on dozens of different public and private providers, such as AWS, DigitalOcean, Azure, OpenStack, and has lots of options. For now, we go with the standard settings. The first cluster node will be ready after some time.
At this point, issue the following command to get a control of this host (so to remotely gain access):
$ docker-machine env node0
This will print some shell variables. Just copy the last row, the one with eval, paste it and issue enter. With those variables configured, you are not operating the local daemon anymore (if any), but the Docker daemon of node0.
If you check the list of machines again, you will see a * next to the image name, to indicate that it's the machine currently in use. Alternatively, you can type the following command to print the currently active machine:
$ docker-machine active
The daemon is running on this machine, with some standard settings (such as on port tcp/2376 enabled TLS). You can ensure that by SSHing to the node and verify the running processes:
$ docker-machine ssh node0 ps aux | grep docker 1320 root /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp://0.0.0.0:2376 --label provider=virtualbox -- tlsverify --tlscacert=/var/lib/boot2docker/ca.pem -- tlscert=/var/lib/boot2docker/server.pem -- tlskey=/var/lib/boot2docker/server-key.pem -s aufs
So, you can immediately this Docker daemon by, for example, starting containers and checking the Docker status:
Perfect! Now we provision the other three hosts, in the same exact way, by naming them node1, node2, and node3:
$ docker-machine create --driver virtualbox node1 $ docker-machine create --driver virtualbox node2 $ docker-machine create --driver virtualbox node3
When they finish, you will have four Docker hosts available. Check with Docker machine.
We're now ready to start a Swarm cluster. But, before, for this very first example in order to keep it as simple as possible, we'll go with disabling TLS for running engines. Our plan is: Run the Docker daemon on port 2375, without TLS.
Let's make a bit of order and explain all ports combinations in detail.
|
Insecure |
Secure |
|
Engine: 2375 |
Engine: 2376 |
|
Swarm: 3375 |
Swarm: 3376 |
|
Swarm v2 uses 2377 for node discovery among nodes |
Port 2377 is for Swarm v2 node to discover each other nodes in the cluster.
To understand where the TLS configuration is, we'll do some exercises by turning off the TLS of all our Docker hosts. Also turning it off here is intended to motivate the readers to learn how the swarm manage command works by invoking it ourselves.
We have four hosts running Docker on port tcp/2376 and with TLS, as Docker Machine creates them by default. We must reconfigure them to change the daemon port to tls/2375 and remove TLS. So, we log in into each of them, with this command:
$ docker-machine ssh node0
Then, we gain root privileges:
$ sudo su -
And configure boot2docker, by modifying the file /var/lib/boot2docker/profile:
# cp /var/lib/boot2docker/profile /var/lib/boot2docker/profile-bak # vi /var/lib/boot2docker/profile
We delete the rows with CACERT, SERVERKEY, and SERVERCERT and configure the daemon port to tcp/2375 and DOCKER_TLS to no. In practice this will be our configuration:
After this log out from the SSH session and restart the machine:
$ docker-machine restart node0
Docker is now running on port tcp/2375 with no security. You can check this with the following command:
$ docker-machine ssh node0 ps aux | grep docker 1127 root /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp://0.0.0.0:2375 --label provider=virtualbox -s aufs
Finally, on your local desktop computer, unset DOCKER_TLS_VERIFY and re-export DOCKER_HOST in order to use the daemon listening on tcp/2375 with no TLS:
$ unset DOCKER_TLS_VERIFY $ export DOCKER_HOST="tcp://192.168.99.103:2375"
We must repeat these steps for each of our four nodes that will be part of our first Swarm.
To get started with Swarm v1 (no surprise), one must pull the swarm image from the Docker hub. Open the four terminals, source the environment variables for each of your machines in each one in the first one, source node0 (docker-machine env node0, and copy and paste the env variable to the shell), in second node1, and so on -, and after completing the steps for changing the standard port and disabling TLS described some lines above, on each of them do:
$ docker pull swarm
We'll use no discovery service for the first example, but the simplest of the mechanisms, such as the nodes://. With nodes://, the Swarm cluster nodes are connected manually, to form a grid of peers. What the operator has to do is simply define a list of nodes IPs and the daemon port, separated by commas, as shown:
nodes://192.168.99.101:2375,192.168.99.102:2375,192.168.99.103:2375,192.168.99.107:2375
To use Swarm, you simply run the swarm container with some arguments. To show the help online, you type:
$ docker run swarm --help
As you see, Swarm has basically four commands:
Create is used to create clusters with a discovery service, for example
token://List shows the list of the cluster nodes
Manage allows you to operate a Swarm cluster
Join, in combination with a discovery service, is used for joining new nodes to an existing cluster
For now, we'll use the manage command. This is the command with most of the options (which you can investigate by issuing docker run swarm manage --help). We limit now to connect nodes. The following is the strategy on each node:
Expose the Swarm service through the swarm container.
Run this container in
daemon(-d) mode.Forward the standard Swarm port
tcp/3376to the internal (on container) porttcp/2375.Specify the list of hosts part of the cluster, with
nodes://- each host has to be a pairIP:portwhere the port is the Docker engine port (tcp/2375).
So, in each terminal you're connected to every machine, execute this:
$ docker run \ -d \ -p 3376:2375 \ swarm manage \ nodes://192.168.99.101:2375,192.168.99.102:2375, 192.168.99.103:2375,192.168.99.107:2375
Tip
When using the nodes:// mechanism, you can use Ansible-like host range patterns, so compact the syntax of three contiguous IPs like nodes://192.168.99.101:2375,192.168.99.102:2375,192.168.99.103:2375
In nodes://192.168.99.[101:103]:2375
Now, as the next step, we'll connect to it and inspect its information before starting using for running containers. For convenience, open a new terminal. We connect now not anymore to the Docker engine on one of our nodes, but to the Docker Swarm. So we will connect to tcp/3376 and not to tcp/2375 anymore. For the purpose of showing in detail what we're doing, let's start by sourcing node0 variables:
$ docker-machine env node0
Copy and paste the eval line, as you already know, and check what shell variables are exported with the following command:
$ export | grep DOCKER_
We now need to do the following:
Change the
DOCKER_HOSTto connect to Swarm porttcp/3376instead of Enginetcp/2375Disable
DOCKER_TLS_VERIFY.Disable
DOCKER_CERT_PATH.
You should have a configuration similar to this:
If we now connect to the Docker swarm at 3376, and show some info, we see that we're running Swarm:
Congratulations! You just started your first Docker cluster with Swarm. We can see that we still have no containers running on our cluster apart from the four swarms, but the Server Version is swarm/1.2.3, the scheduling strategy is spread, and, most importantly, we have four healthy nodes in our swarm (details of each Swarm node follow).
Also, you can get some extra information regarding the scheduler behavior of this Swarm cluster:
Strategy: spread Filters: health, port, containerslots, dependency, affinity, constraint
A spread scheduling strategy means that Swarm will attempt to place containers on the less utilized host and the listed filters are available when you create containers, thus allowing you to decide to manually suggest some options. For example, you might want to make your Galera cluster containers geographically near but on different hosts.
But, what's the size of this Swarm? You can see it at the very end of this output:
It means that on this tiny Swarm you have the total availability of these resources: four CPUs and 4GB of memory. That's just what we expected, by merging the computational resources of 4 VirtualBox hosts with a CPU and 1GB of memory each.
Now that we have a Swarm cluster, it's time to start using it. We'll show that the spread strategy algorithm will decide to place containers to the less loaded hosts. In this example, it's really easy, as we start with four empty nodes. So, we're connected to Swarm and Swarm will put containers on hosts. We start one nginx container, mapping its port tcp/80 to the host (machine) port tcp/80.
$ docker run -d -p 80:80 nginx 2c049db55f9b093d19d575704c28ff57c4a7a1fb1937bd1c20a40cb538d7b75c
In this example, we see that the Swarm scheduler decided to place this container onto node1:
Since we have to bind a port tcp/80 to any host, we will have only four chances, four containers on four different hosts. Let's create new nginx containers and see what happens:
$ docker run -d -p 80:80 nginx 577b06d592196c34ebff76072642135266f773010402ad3c1c724a0908a6997f $ docker run -d -p 80:80 nginx 9fabe94b05f59d01dd1b6b417f48155fc2aab66d278a722855d3facc5fd7f831 $ docker run -d -p 80:80 nginx 38b44d8df70f4375eb6b76a37096f207986f325cc7a4577109ed59a771e6a66d
Now we have 4 nginx containers placed on our 4 Swarm hosts:
Now we try to create a new nginx:
$ docker run -d -p 80:80 nginx docker: Error response from daemon: Unable to find a node that satisfies the following conditions [port 80 (Bridge mode)]. See 'docker run --help'.
What happened is just that Swarm wasn't able to find a suitable host to place a new container on, because on all hosts, port tcp/80 are all occupied. After running these 4 nginx containers, plus the four swarm containers (for the infrastructure management), as we expected, we have eight running containers on this Swarm cluster:
This is how Swarm v1 was intended to work (and still does its job).
In this section, we'll set up a small cluster with the new Swarm mode built in Docker Engine 1.12 or later.
At the DockerCon16, among the big announcements, two drew big attention regarding containers orchestration:
The integration between the Engine and Swarm, called the Docker Swarm mode.
SwarmKit
In practice, the Docker daemon, starting from version 1.12, adds the possibility to run a so-called Swarm Mode. New CLI commands were added to the docker client, such as node, service, stack, deploy, alongside with, of course, swarm.
We'll cover Swarm Mode and SwarmKit in more detail starting fromChapter 3, Meeting Docker Swarm Mode, but now that we finished the example with Swarm v1, we're going to give the reader a taste on how Swarm v2 has a much simpler user experience than v1. The only requirement to use Swarm v2 is to have a daemon version of at least version 1.12-rc1. But with Docker Machine 0.8.0-rc1+, you can provision Docker hosts fulfilling this requirement with the usual procedure.
Docker also announced Docker for AWS and Docker for Azure at DockerCon 2016. Not only AWS and Azure, but actually we're also fans of DigitalOcean, so we created a new tool that wraps around doctl the DigitalOcean command line interface, to help provision Docker cluster in the new massively way. The tool is called belt and now available from http://github.com/chanwit/belt. You can pull belt with this command:
go get github.com/chanwit/belt
or download the binary from the Release tab of the project.
First, we'll prepare a template file for provisioning on DigitalOcean. Your .belt.yaml will look like this:
$ cat .belt.yaml --- digitalocean: region: sgp1 image: 18153887 ssh_user: root ssh_key_fingerprint: 816630
Please note that my image number 18153887 is the snapshot containing Docker 1.12. DigitalOcean usually makes the latest Docker image available after every release. To make you able to control your cluster, SSH key needs to be there. For the field ssh_key_fingerprint, you can either put the finger print as well as the key ID.
Do not forget to set your DIGITALOCEAN_ACCESS_TOKEN environment variable. Also, Belt recognizes the same set of Docker Machine shell variables. If you are familiar with Docker Machine you'll know how to set them. To refresh, these are the shell variables we introduced in the previous section:
export DOCKER_TLS_VERIFY="1"export DOCKER_HOST="tcp://<IP ADDRESS>:2376"export DOCKER_CERT_PATH="/Users/user/.docker/machine/machines/machine"export DOCKER_MACHINE_NAME="machine"
So, now let's see how to use Belt:
$ export DIGITALOCEAN_ACCESS_TOKEN=1b207 .. snip .. b6581c
Now we create a Swarm of four nodes each with 512M of memory:
$ belt create 512mb node[1:4] ID Name Public IPv4 Memory VCPUs Disk 18511682 node1 512 1 20 18511683 node4 512 1 20 18511684 node3 512 1 20 18511681 node2 512 1 20
You can see that we can specify a set of nodes with similar syntax node[1:4]. This command created four nodes on DigitalOcean. Please wait for about 55 seconds for all nodes to be provisioned. Then you can list them:
$ belt ls ID Name Public IPv4 Status Tags 18511681 node2 128.199.105.119 active 18511682 node1 188.166.183.86 active 18511683 node4 188.166.183.103 active 18511684 node3 188.166.183.157 active
Their status now has changed from "new" to "active". All IP addresses are assigned. Everything is good to go for now.
We now can start Swarm.
Before that make sure we are running Docker 1.12. We check this on node1.
$ belt active node1 node1 $ belt docker version Client: Version: 1.12.0-rc2 API version: 1.24 Go version: go1.6.2 Git commit: 906eacd Built: Fri Jun 17 21:02:41 2016 OS/Arch: linux/amd64 Experimental: true Server: Version: 1.12.0-rc2 API version: 1.24 Go version: go1.6.2 Git commit: 906eacd Built: Fri Jun 17 21:02:41 2016 OS/Arch: linux/amd64 Experimental: true
The belt docker command is just a thin wrapper command that sends the whole command line going through SSH to your Docker host. So this tool will not get in the way and your Docker Engines is always in-control.
Now we will initialize the first node with Swarm Mode.
$ belt docker swarm init Swarm initialized: current node (c0llmsc5t1tsbtcblrx6ji1ty) is now a manager.
Then we'll join other three nodes to this newly formed cluster. Joining a large cluster is a tedious task. Instead of going through every node and do docker swarm join manually, we'll let belt do this for us:
$ belt swarm join node1 node[2:4] node3: This node joined a Swarm as a worker. node2: This node joined a Swarm as a worker. node4: This node joined a Swarm as a worker.
Tip
You can of course be able to run: belt --host node2 docker swarm join <node1's IP>:2377 to manually join node2 to your cluster.
And you'll get this view of cluster:
$ belt docker node ls ID NAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS 4m5479vud9qc6qs7wuy3krr4u node2 Accepted Ready Active 4mkw7ccwep8pez1jfeok6su2o node4 Accepted Ready Active a395rnht2p754w1beh74bf7fl node3 Accepted Ready Active c0llmsc5t1tsbtcblrx6ji1ty * node1 Accepted Ready Active Leader
Congratulations! You just installed a Swarm cluster on DigitalOcean.
We now create a service for nginx. This command creates an Nginx service with 2 instances of containers published at port 80.
$ belt docker service create --name nginx --replicas 2 -p 80:80 nginx d5qmntf1tvvztw9r9bhx1hokd
Here we go:
$ belt docker service ls ID NAME REPLICAS IMAGE COMMAND d5qmntf1tvvz nginx 2/2 nginx
Now let's scale it to 4 nodes.
$ belt docker service scale nginx=4 nginx scaled to 4 $ belt docker service ls ID NAME REPLICAS IMAGE COMMAND d5qmntf1tvvz nginx 4/4 nginx
Similar to Docker Swarm, you can now use belt ip to see where the node runs. You can use any IP address to browse the NGINX service. It's available on every node.
$ belt ip node2 128.199.105.119
This is how Swarm Mode looks like starting from Docker 1.12.
In this chapter, we met Docker Swarm, defined its aims, features, and architecture. We also reviewed some of the other possible open source alternatives to Swarm, and their relationships with it. Finally, we installed and started using Swarm by creating a simple local cluster made of four hosts on Virtualbox and on Digital Ocean.
Clusterize containers with Swarm will be the main topic for the whole book, but before we start using Swarm in production, we'll understand some theory before, beginning with the discovery services, the topic of Chapter 2, Discover the Discovery Services.
