The DevOps 2.1 Toolkit: Docker Swarm: The next level of building reliable and scalable software unleashed

Every Continuous Integration process starts with a code that is checked out from a repository. We'll use the GitHub repository vfarcic/go-demo (https://github.com/vfarcic/go-demo) throughout the book. It contains the code of the service we'll use throughout the book. The service is written in Go (https://golang.org/). Fear not! Even though I consider it one of the best currently available languages, you will not be required to learn Go. We'll use the go-demo service only as a demonstration of the processes explained throughout the book. Even though I strongly recommend learning Go, the book does not assume any knowledge of the language. All the examples will be programming language agnostic.

Let's get going and check out the go-demo code:

git clone https://github.com/vfarcic/go-demo.git

cd go-demo

Some of the files will be shared between the host file system and Docker Machines we'll start creating soon. Docker Machine makes the whole directory that belongs to the current user available inside the VM. Therefore, please make sure that the code is checked out inside one of the user's sub-folders.

Now that we have the code checked out from the repository, we need a server that we'll use to build and run tests. For now, we'll use Docker Machine, since it provides an easy way to create a "Docker ready" VMs on our laptops.

The Docker Machine (https://docs.docker.com/machine/overview/) is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with the docker-machine commands. You can use Machine to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like AWS or DigitalOcean.

Using docker-machine commands, you can start, inspect, stop, and restart a managed host, upgrade the Docker client and daemon, and configure a Docker client to talk to your host.

Machine was the only way to run Docker on Mac or Windows previous to Docker v1.12. Starting with the beta program and Docker v1.12, Docker for Mac and Docker for Windows are available as native apps and the better choice for this use case on newer desktops and laptops. I encourage you to try out these new apps. The installers for Docker for Mac and Docker for Windows include Docker Machine, along with Docker Compose.

Let's create our first server called go-demo use the following command:

docker-machine create -d virtualbox go-demo

The command should be self-explanatory. We specified virtualbox as the driver (or Hyper-V if you're running Docker for Windows) and named the machine go-demo:

Now that the machine is running, we should instruct our local Docker Engine to use it, use the following command:

docker-machine env go-demo

The docker-machine env go-demo command outputs environment variables required for the local engine to find the server we'd like to use. In this case, the remote engine is inside the VM we created with the docker-machine create command.

The output is as follows:

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/vfarcic/.docker/machine/machines/go-demo"
export DOCKER_MACHINE_NAME="go-demo"

We can envelop the env command into an eval that will evaluate the output and, in this case, create the environment variables using the following command:

eval $(docker-machine env go-demo)

From now on, all the Docker commands we execute locally will be channeled to the engine running inside the go-demo machine.

Now we are ready to run the first two steps in the CI flow. We'll execute unit tests and build the service binary.

We'll use Docker Compose for the CI flow. As you will see soon, Docker Compose has little, if any, value when operating the cluster. However, for operations that should be performed on a single machine, Docker Compose is still the easiest and the most reliable way to go.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application's services. Then, using a single command, you create and start all the services from your configuration. Compose is great for development, testing, and staging environments, as well as CI workflows.

The repository that we cloned earlier, already has all the services we'll need defined inside the docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-loc) file.

Let's take a look at the content of the docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-local.yml) file:

cat docker-compose-test-local.yml

The service we'll use for our unit tests is called unit. It is as follows:

unit:
    image:golang:1.6
    volumes:
      - .:/usr/src/myapp
      - /tmp/go:/go
    working_dir: /usr/src/myapp
    command: bash -c "go get -d -v -t && go test --cover -v \
./... && go build -v-o go-demo"

It is a relatively simple definition. Since the service is written in Go, we are using the golang:1.6 image.

Next, we are exposing a few volumes. Volumes are directories that are, in this case, mounted on the host. They are defined with two arguments. The first argument is the path to the host directory while the second represents a directory inside the container. Any file already inside the host directory will be available inside the container and vice versa.

The first volume is used for the source files. We are sharing the current host directory . with the container directory /usr/src/myapp. The second volume is used for Go libraries. Since we want to avoid downloading all the dependencies every time we run unit tests, they will be stored inside the host directory /tmp/go. That way, dependencies will be downloaded only the first time we run the service.

Volumes are followed with the working_dir instruction. When the container is run, it will use the specified value as the starting directory.

Finally, we are specifying the command we want to run inside the container. I won't go into details since they are specific to Go. In short, we download all the dependencies go get -d -v -t, run unit tests go test --cover -v ./..., and build the go-demo binary go build -v -o go-demo. Since the directory with the source code is mounted as a volume, the binary will be stored on the host and available for later use.

With this single Compose service, we defined two steps of the CI flow. It contains unit tests and build of the binary.

Please note that even though we run the service called unit, the real purpose of this CI step is to run any type of tests that do not require deployment. Those are the tests we can execute before we build the binary and, later on, Docker images.

Let's run the following code:

docker-compose \ 
    -f docker-compose-test-local.yml \ 
    run --rm unit

We specified that Compose should use docker-compose-test-local.yml file (default is docker-compose.yml) and run the service called unit. The --rm argument means that the container should be removed once it stops. The run command should be used for services that are not meant to run forever. It is perfect for batch jobs and, as in this case, for running tests.

As you can see from the output, we pulled the golang image, downloaded service dependencies, successfully ran the tests, and built the binary.

We can confirm that the binary is indeed built and available on the host by listing the files in the current directory using the following command. For brevity, we'll filter the result:

ls -l *go-demo*

Now that we passed the first round of tests and have the binary, we can proceed and build the Docker images.

Docker images are built through a definition stored in a Dockerfile. With few exceptions, it takes a similar approach as if we would define a simple script. We will not explore all the options we can use when defining a Dockerfile, but only those used for the go-demo service. Please consult the Dockerfile reference (https://docs.docker.com/engine/reference/builder/) page for more info. The go-demo Dockerfile is as follows:

FROM alpine:3.4
MAINTAINER Viktor Farcic <viktor@farcic.com>

RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

EXPOSE 8080
ENV DB db
CMD ["go-demo"]
HEALTHCHECK --interval=10s CMD wget -qO- localhost:8080/demo/hello

COPY go-demo /usr/local/bin/go-demo
RUN chmod +x /usr/local/bin/go-demo

Each of the statements will be built as a separate image. A container is a collection of images stacked one on top of the other.

Every Dockerfile starts with the FROM statement. It defines the base image that should be used. In most cases, my preference is to use alpine Linux. With its size being around 2MB it is probably the smallest distribution we can use. That is aligned with the idea that containers should have only things that are needed and avoid any extra overhead.

MAINTAINER is for informational purposes only.

The RUN statement executes any command set as its argument. I won't explain this one since it is very specific to the service we're building.

The EXPOSE statement defines the port the service will be listening to. It is followed by the definition of the environment variable DB that tells the service the address of the database. The default value is db and, as you'll see soon, it can be changed at runtime. The CMD statement represents the command that will be run when containers start.

The HEALTHCHECK instruction tells Docker how to test a container to check that it is still working. This can detect cases such as a web server that is stuck in an infinite loop and unable to handle new connections, even though the server process is still running. When a container has a healthcheck specified, it has a health status in addition to its normal status. This status is initially starting. Whenever a health check passes, it becomes healthy (from whatever state it was previously in). After a certain number of consecutive failures, it becomes unhealthy. In our case, the healthcheck will be executed every ten seconds. The command sends a simple request to one of the API endpoints. If the service responds with status 200, the wget command will return 0 and Docker will consider the service healthy. Any other response will be considered as unhealthy and Docker Engine will perform certain actions to fix the situation.

Finally, we copy the go-demo binary from the host to the /usr/local/bin/ directory inside the image and give it executable permissions with the chmod command.

To some, the order of the statements might not look logical. However, there is a good reason behind such declarations and their order. Those that are less likely to change are defined before those that are prone to changes. Since go-demo will be a new binary every time we build the images, it is defined last.

The reasons behind such order lie in the way Docker Engine creates images. It starts from the top-most definition and checks whether it changed since the last time the build was run. If it didn't, it moves to the next statement. As soon as it finds a statement that would produce a new image, it, and all the statements following it are built into Docker images. By placing those that are less likely to change closer to the top, we can reduce the build time, disk usage, and bandwidth.

Now that we understand the Dockerfile behind the go-demo service, we can build the images.

The command is very straightforward and is as follows:

docker build -t go-demo .

As an alternative, we can define build arguments inside a Docker Compose file. The service defined in docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-local.yml) file is as follows:

  app:
    build: .
    image: go-demo

In both cases, we specified that the current directory should be used for the build process . and that the name of the image is go-demo.

We can run the build through Docker compose with the command that is as follows:

docker-compose \ 
    -f docker-compose-test-local.yml \ 
    build app

We'll use the latter method throughout the rest of the book.

We can confirm that the image was indeed built, by executing the docker images command as follows:

docker images

The output is as follows:

REPOSITORY TAG    IMAGE ID     CREATED        SIZE
go-demo    latest 5e90126bebf1 49 seconds ago 23.61 MB
golang     1.608a89f0a4ee5 11 hours ago   744.2 MB
alpine     latest 4e38e38c8ce0 9 weeks ago    4.799 MB

As you can see, go-demo is one of the images we have inside the server.

Now that the images are built, we can run staging tests that depend on the service and its dependencies to be deployed on a server.

Please note that the real purpose of this step in the CI flow is to run the tests that require the service and its dependencies to be running. Those are still not integration tests that require production or production-like environment. The idea behind those tests is to run the service together with its direct dependencies, run the tests, and, once they're finished, remove everything and free the resources for some other task. Since these are still not integration tests, some, if not all, dependencies can be mocks.

Due to the nature of these tests, we need to split the task into three actions:

The dependencies are defined as the staging-dep service inside the docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-local.yml) file. The definition is as follows:

  staging-dep:
    image: go-demo
    ports:
      -8080:8080
    depends_on:
      - db

  db:
    image: mongo:3.2.10

The image is go-demo, and it exposes the port 8080 (both on the host and inside the container). It depends on the service db which is a mongo image. Services defined as depends_on will be run before the service that defines the dependency. In other words, if we run the staging-dep target, Compose will run the db first.

Let's run the dependencies as shown in the following code:

docker-compose \ 
    -f docker-compose-test-local.yml \
    up -d staging-dep

Once the command is finished, we will have two containers running (go-demo and db). We can confirm that by listing all the processes:

docker-compose \ 
    -f docker-compose-test-local.yml \
    ps

The output is as follows:

Name                 Command               State Ports
---------------------------------------------------------------------
godemo_db_1          /entrypoint.sh mongod Up    27017/tcp
godemo_staging-dep_1 go-demo               Up    0.0.0.0:8080->8080/tcp

Now that the service and the database it depends on are running, we can execute the tests. They are defined as the service staging. The definition is as follows:

  staging:
    extends:
      service: unit
    environment:
      - HOST_IP=localhost:8080
    network_mode: host
command: bash -c"go get -d -v -t && go test --tags integration -v"

Since the definition of the staging tests is very similar to those we run as unit tests, the staging service extends unit. By extending a service, we inherit its full definition. Further on, we defined an environment variable HOST_IP. The tests code uses that variable to determine the location of the service under test. In this case, since the go-demo service is running on the same server as tests, the IP is server's localhost. Since, by default, localhost inside a container is not the same as the one on the host, we had to define network_mode as host. Finally, we defined the command that should be executed. It will download tests dependencies go get -d -v -t and run the tests go test --tags integration -v.

Let's run the following commands:

docker-compose \ 
    -f docker-compose-test-local.yml \ 
    run --rm staging

All the tests passed, and we are one step closer to the goal of having full confidence that the service is indeed safe to be deployed to production.

We don't have any use for keeping the service and the database running so let's remove them and free the resources for some other task:

docker-compose \ 
    -f docker-compose-test-local.yml \ 
    down

The down command stops and removes all services defined in that Compose file. We can verify that by running the following  ps command:

docker-compose \ 
    -f docker-compose-test-local.yml \ 
    ps

The output is as follows:

Name   Command   State   Ports
------------------------------

There is only one thing missing for the CI flow to be complete. At this moment we have the go-demo image that is usable only inside the go-demo server. We should store it in a registry so that it can be accessed from other servers as well.

Before we push our go-demo image, we need a place to push to. Docker offers multiple solutions that act as a registry. We can use Docker Hub (https://hub.docker.com/), Docker Registry (https://docs.docker.com/registry/), and Docker Trusted Registry (https://docs.docker.com/docker-trusted-registry/). On top of those, there are many other solutions from third party vendors.

Which registry should we use? Docker Hub requires a username and password, and I do not trust you enough to provide my own. One of the goals I defined before I started working on the book is to use only open source tools so Docker Trusted Registry, while being an excellent choice under different circumstances, is also not suitable. The only option left (excluding third party solutions), is Docker Registry (https://docs.docker.com/registry/).

The registry is defined as one of the services inside the docker-compose-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-local.yml) Compose file. The definition is as follows:

  registry:
    container_name: registry
    image: registry:2.5.0
    ports:
      -5000:5000
    volumes:
      - .:/var/lib/registry
    restart: always

We set registry as an explicit container name, specified the image, and opened the port 5000 (both on the host and inside the container).

Registry stores the images inside the /var/lib/registry directory, so we mounted it as a volume on the host. That way, data will not be lost if the container fails. Since this is a production service that could be used by many, we defined that it should always be restarted on failure.

Let's run the following commands:

docker-compose \ 
    -f docker-compose-local.yml \ 
    up -d registry

Now that we have the registry, we can do a dry-run. Let's confirm that we can pull and push images to it:

docker pull alpine

docker tag alpine localhost:5000/alpine

docker push localhost:5000/alpine

Docker uses a naming convention to decide where to pull and push images from. If the name is prefixed with an address, the engine will use it to determine the location of the registry. Otherwise, it assumes that we want to use Docker Hub. Therefore, the first command pulled the alpine image from Docker Hub.

The second command created a tag of the alpine image. The tag is a combination of the address of our registry localhost:5000 and the name of the image. Finally, we pushed the alpine image to the registry running on the same server.

Before we start using the registry in a more serious fashion, let's confirm that the images are indeed persisted on the host:

ls -1 docker/registry/v2/repositories/alpine/

The output is as follows:

_layers
_manifests
_uploads

I won't go into details what each of those sub-directories contains. The important thing to note is that registry persists the images on the host so no data will be lost if it fails or, in this case, even if we destroy the VM since that Machine directory is mapped to the same directory on our laptop.

We were a bit hasty when we declared that this registry should be used in production. Even though data is persisted, if the whole VM crashes, there would be a downtime until someone brings it up again or creates a new one. Since one of the goals is to avoid downtime whenever possible, later on, we should look for a more reliable solution. The current setup should do for now.

Now we are ready to push the go-demo image to the registry:

docker tag go-demo localhost:5000/go-demo:1.0

docker push localhost:5000/go-demo:1.0

As with the Alpine example, we tagged the image with the registry prefix and pushed it to the registry. We also added a version number 1.0.

The push was the last step in the CI flow. We run unit tests, built the binary, built the Docker image, ran staging tests, and pushed the image to the registry. Even though we did all those things, we are not yet confident that the service is ready for production. We never tested how it would behave when deployed to a production (or production-like) cluster. We did a lot, but not enough.

If CI were our final objective, this would be the moment when manual validations should occur. While there is a lot of value in manual labor that requires creativity and critical thinking, we cannot say the same for repetitive tasks. Tasks required for converting this Continuous Integration flow into Continuous Delivery and, later on, deployment are, indeed repetitive.

We have the CI process done, and it is time to do the extra mile and convert it into Continuous Delivery.

Before we move into the steps required for the Continuous Integration process to become Continuous Delivery, we need to take a step back and explore cluster management. After all, in most cases, there is no production environment without a cluster.

We'll destroy the VMs at the end of each chapter. That way, you can come back to any of part of the book and do the exercises without the fear that you might need to do some steps from one of the earlier chapters. Also, such a procedure will force us to repeat a few things. Practice makes perfect. To reduce your waiting times, I did my best to keep things as small as possible and keep download times to a minimum. Execute the following command:

docker-machine rm -f go-demo

The next chapter is dedicated to the setup and operation of a Swarm cluster.