It is paradoxical, yet true, to say, that the more we know, the more ignorant we become in the absolute sense, for it is only through enlightenment that we become conscious of our limitations. Precisely one of the most gratifying results of intellectual evolution is the continuous opening up of new and greater prospects. —Nikola Tesla
To fully understand the challenges and benefits that Docker Swarm brings, we need to start from the beginning. We need to go back to a code repository and decide how are we going to build, test, run, update, and monitor the services we're developing. Even though the objective is to implement continuous deployment to a Swarm cluster, we need to step back and explore Continuous Integration (CI) first. The steps we'll define for the CI process will dictate how we proceed towards Continuous Delivery (CD), from there towards Continuous Deployment (CDP), and, finally, how we ensure that our services are monitored and able to self-heal. This chapter explores Continuous Integration as a prerequisite for the more advanced processes.
A note to The DevOps 2.0 Toolkit readersThe text that follows is identical to the one published in The DevOps 2.0 Toolkit. If it is still fresh in your mind, feel free to jump to the sub-section Defining a fully Dockerized manual Continuous Integration flow . Since I wrote the 2.0, I discovered a few better ways to implement CI processes. I hope you'll benefit from this chapter even if you consider yourself a veteran CI practitioner.
To understand Continuous Deployment we should first define its predecessors, Continuous Integration and Continuous Delivery. Integration phase of a project development tended to be one of the most painful stages of Software Development Life Cycle (SDLC). We would spend weeks, months or even years working in separate teams dedicated to separate applications and services. Each of those teams would have their set of requirements and tried their best to meet them. While it wasn't hard to periodically verify each of those applications and services in isolation, we all dreaded the moment when team leads would decide that the time has come to integrate them into a unique delivery. Armed with the experience from previous projects, we knew that integration would be problematic. We knew that we would discover problems, unmet dependencies, interfaces that do not communicate with each other correctly and that managers will get disappointed, frustrated, and nervous. It was not uncommon to spend weeks or even months in this phase. The worse part of all that was that a bug found during the integration phase could mean going back and redoing days or weeks worth of work. If someone asked me how I felt about integration back then, I'd say that it was closest I could get to becoming permanently depressed. Those were different times. We thought that was the right way to develop applications.
A lot changed since then. Extreme Programming (XP) and other agile methodologies became familiar, automated testing became frequent, and Continuous Integration started to take ground. Today we know that the way we developed software back then was wrong. The industry moved a long way since those days.
Continuous Integration usually refers to integrating, building, and testing code within the development environment. It requires developers to integrate code into a shared repository often. How often is often can be interpreted in many ways and it depends on the size of the team, the size of the project and the number of hours we dedicate to coding. In most cases, it means that coders either push directly to the shared repository or merge their code with it. No matter whether we're pushing or merging, those actions should, in most cases, be done at least a couple of times a day. Getting code to the shared repository is not enough, and we need to have a pipeline that, as a minimum, checks out the code and runs all the tests related, directly or indirectly, to the code corresponding to the repository. The result of the execution of the pipeline can be either red or green. Something failed, or everything was run without any problems. In the former case, minimum action would be to notify the person who committed the code.
The Continuous Integration pipeline should run on every commit or push. Unlike Continuous Delivery, Continuous Integration does not have a clearly defined goal of that pipeline. Saying that one application integrates with others does not tell us a lot about its production readiness. We do not know how much more work is required to get to the stage when the code can be delivered to production. All we are striving for is the knowledge that a commit did not break any of the existing tests. Nevertheless, CI is a vast improvement when done right. In many cases, it is a very hard practice to implement, but once everyone is comfortable with it, the results are often very impressive.
Integration tests need to be committed together with the implementation code, if not before. To gain maximum benefits, we should write tests in Test-Driven Development (TDD) fashion. That way, not only that tests are ready for commit together with implementation, but we know that they are not faulty and would not pass no matter what we do. There are many other benefits TDD brings to the table and, if you haven't already, I strongly recommend to adopt it. You might want to consult the Test-Driven Development (http://technologyconversations.com/category/test-driven-development/) section of the Technology Conversations (http://technologyconversations.com/) blog.
Tests are not the only CI prerequisite. One of the most important rules is that when the pipeline fails, fixing the problem has higher priority than any other task. If this action is postponed, next executions of the pipeline will fail as well. People will start ignoring the failure notifications and, slowly, CI process will begin losing its purpose. The sooner we fix the problem discovered during the execution of the CI pipeline, the better we are. If corrective action is taken immediately, knowledge about the potential cause of the problem is still fresh (after all, it's been only a few minutes between the commit and the failure notification) and fixing it should be trivial.
Every Continuous Integration process starts with a code that is checked out from a repository. We'll use the GitHub repository
vfarcic/go-demo (https://github.com/vfarcic/go-demo) throughout the book. It contains the code of the service we'll use throughout the book. The service is written in Go (https://golang.org/). Fear not! Even though I consider it one of the best currently available languages, you will not be required to learn Go. We'll use the go-demo service only as a demonstration of the processes explained throughout the book. Even though I strongly recommend learning Go, the book does not assume any knowledge of the language. All the examples will be programming language agnostic.
All the commands from this chapter are available in the
01-continuous-integration.sh (https://gist.github.com/vfarcic/886ae97fe7a98864239e9c61929a3c7c) Gist.
A note to Windows users Please make sure that your Git client is configured to check out the code AS-IS. Otherwise, Windows might change carriage returns to the Windows format.
Let's get going and check out the
git clone https://github.com/vfarcic/go-demo.git cd go-demo
Some of the files will be shared between the host file system and Docker Machines we'll start creating soon. Docker Machine makes the whole directory that belongs to the current user available inside the VM. Therefore, please make sure that the code is checked out inside one of the user's sub-folders.
Now that we have the code checked out from the repository, we need a server that we'll use to build and run tests. For now, we'll use Docker Machine, since it provides an easy way to create a "Docker ready" VMs on our laptops.
The Docker Machine (https://docs.docker.com/machine/overview/) is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with the
docker-machine commands. You can use Machine to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like AWS or DigitalOcean.
docker-machine commands, you can start, inspect, stop, and restart a managed host, upgrade the Docker client and daemon, and configure a Docker client to talk to your host.
Machine was the only way to run Docker on Mac or Windows previous to Docker v1.12. Starting with the beta program and Docker v1.12, Docker for Mac and Docker for Windows are available as native apps and the better choice for this use case on newer desktops and laptops. I encourage you to try out these new apps. The installers for Docker for Mac and Docker for Windows include Docker Machine, along with Docker Compose.
The examples that follow assume that you have Docker Machine version v0.9 (https://www.docker.com/products/docker-machine) that includes Docker Engine v1.13+ (https://www.docker.com/products/docker-engine). The installation instructions can be found in the Install Docker Machine (https://docs.docker.com/machine/install-machine/) page.
A note to Windows users The recommendation is to run all the examples from Git Bash (installed through Docker Toolbox as well as Git). That way the commands you'll see throughout the book will be same as those that should be executed on OS X or any Linux distribution.
A note to Linux users Docker Machine on Linux might not be able to mount a host volume inside VMs. The problem is related to the fact that both host and Docker Machine OSes use
/home directory. Mounting
/home from the host would overwrite some of the required files. If you experience problems with mounting of the host volume, please export the
export VIRTUALBOX_SHARE_FOLDER="$PWD:$PWD" If machines are already created, you'll have to destroy them and create them again. Please note that this problem should be fixed in newer Docker Machine versions so use this workaround only if you notice that the volume is not mounted (files from the host are not available inside VMs).
Let's create our first server called
go-demo use the following command:
docker-machine create -d virtualbox go-demo
A note to Windows users If you're using Docker for Windows instead of Docker Toolbox, you will need to change the driver from virtualbox to Hyper-V. The problem is that Hyper-V does not allow mounting host volumes, so it is still highly recommended to use Docker Toolbox when working with Docker Machine. The reason behind the choice of running Docker inside Docker Machines instead natively on the host lies in the need to run a cluster (coming in the next chapter). Docker Machine is the easiest way to simulate a multi-node cluster.
The command should be self-explanatory. We specified virtualbox as the driver (or Hyper-V if you're running Docker for Windows) and named the machine
A note to Windows users In some cases, Git Bash might think that it is still running as BAT. If you experience a problem with the
docker-machine env commands, please export the
Now that the machine is running, we should instruct our local Docker Engine to use it, use the following command:
docker-machine env go-demo
docker-machine env go-demo command outputs environment variables required for the local engine to find the server we'd like to use. In this case, the remote engine is inside the VM we created with the
docker-machine create command.
The output is as follows:
export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://192.168.99.100:2376" export DOCKER_CERT_PATH="/Users/vfarcic/.docker/machine/machines/go-demo" export DOCKER_MACHINE_NAME="go-demo"
We can envelop the
env command into an
eval that will evaluate the output and, in this case, create the environment variables using the following command:
eval $(docker-machine env go-demo)
From now on, all the Docker commands we execute locally will be channeled to the engine running inside the
Now we are ready to run the first two steps in the CI flow. We'll execute unit tests and build the service binary.
We'll use Docker Compose for the CI flow. As you will see soon, Docker Compose has little, if any, value when operating the cluster. However, for operations that should be performed on a single machine, Docker Compose is still the easiest and the most reliable way to go.
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application's services. Then, using a single command, you create and start all the services from your configuration. Compose is great for development, testing, and staging environments, as well as CI workflows.
The repository that we cloned earlier, already has all the services we'll need defined inside the
docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-loc) file.
Let's take a look at the content of the
docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-local.yml) file:
The service we'll use for our unit tests is called
unit. It is as follows:
unit: image:golang:1.6 volumes: - .:/usr/src/myapp - /tmp/go:/go working_dir: /usr/src/myapp command: bash -c "go get -d -v -t && go test --cover -v \ ./... && go build -v-o go-demo"
It is a relatively simple definition. Since the service is written in Go, we are using the
Next, we are exposing a few volumes. Volumes are directories that are, in this case, mounted on the host. They are defined with two arguments. The first argument is the path to the host directory while the second represents a directory inside the container. Any file already inside the host directory will be available inside the container and vice versa.
The first volume is used for the source files. We are sharing the current host directory
. with the container directory
/usr/src/myapp. The second volume is used for Go libraries. Since we want to avoid downloading all the dependencies every time we run unit tests, they will be stored inside the host directory
/tmp/go. That way, dependencies will be downloaded only the first time we run the service.
Volumes are followed with the
working_dir instruction. When the container is run, it will use the specified value as the starting directory.
Finally, we are specifying the command we want to run inside the container. I won't go into details since they are specific to Go. In short, we download all the dependencies
go get -d -v -t, run
go test --cover -v ./..., and build the go-demo binary
go build -v -o go-demo. Since the directory with the source code is mounted as a volume, the binary will be stored on the host and available for later use.
With this single Compose service, we defined two steps of the CI flow. It contains unit tests and build of the binary.
Please note that even though we run the service called
unit, the real purpose of this CI step is to run any type of tests that do not require deployment. Those are the tests we can execute before we build the binary and, later on, Docker images.
Let's run the following code:
docker-compose \ -f docker-compose-test-local.yml \ run --rm unit
A note to Windows users You might experience a problem with volumes not being mapped correctly. If you see an
Invalid volume specification error, please export the environment variable
COMPOSE_CONVERT_WINDOWS_PATHS set to
export COMPOSE_CONVERT_WINDOWS_PATHS=0 If that fixed the problem with volumes, please make sure that the variable is exported every time you run
We specified that Compose should use
docker-compose-test-local.yml file (default is
docker-compose.yml) and run the service called
--rm argument means that the container should be removed once it stops. The run command should be used for services that are not meant to run forever. It is perfect for batch jobs and, as in this case, for running tests.
As you can see from the output, we pulled the
golang image, downloaded service dependencies, successfully ran the tests, and built the binary.
We can confirm that the binary is indeed built and available on the host by listing the files in the current directory using the following command. For brevity, we'll filter the result:
ls -l *go-demo*
Now that we passed the first round of tests and have the binary, we can proceed and build the Docker images.
Docker images are built through a definition stored in a Dockerfile. With few exceptions, it takes a similar approach as if we would define a simple script. We will not explore all the options we can use when defining a Dockerfile, but only those used for the
go-demo service. Please consult the Dockerfile reference (https://docs.docker.com/engine/reference/builder/) page for more info. The
go-demo Dockerfile is as follows:
FROM alpine:3.4 MAINTAINER Viktor Farcic <firstname.lastname@example.org> RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2 EXPOSE 8080 ENV DB db CMD ["go-demo"] HEALTHCHECK --interval=10s CMD wget -qO- localhost:8080/demo/hello COPY go-demo /usr/local/bin/go-demo RUN chmod +x /usr/local/bin/go-demo
Each of the statements will be built as a separate image. A container is a collection of images stacked one on top of the other.
Every Dockerfile starts with the
FROM statement. It defines the base image that should be used. In most cases, my preference is to use
alpine Linux. With its size being around 2MB it is probably the smallest distribution we can use. That is aligned with the idea that containers should have only things that are needed and avoid any extra overhead.
MAINTAINER is for informational purposes only.
RUN statement executes any command set as its argument. I won't explain this one since it is very specific to the service we're building.
EXPOSE statement defines the port the service will be listening to. It is followed by the definition of the environment variable
DB that tells the service the address of the database. The default value is
db and, as you'll see soon, it can be changed at runtime. The
CMD statement represents the command that will be run when containers start.
HEALTHCHECK instruction tells Docker how to test a container to check that it is still working. This can detect cases such as a web server that is stuck in an infinite loop and unable to handle new connections, even though the server process is still running. When a container has a healthcheck specified, it has a health status in addition to its normal status. This status is initially starting. Whenever a health check passes, it becomes healthy (from whatever state it was previously in). After a certain number of consecutive failures, it becomes unhealthy. In our case, the healthcheck will be executed every ten seconds. The command sends a simple request to one of the API endpoints. If the service responds with status
wget command will return
0 and Docker will consider the service healthy. Any other response will be considered as unhealthy and Docker Engine will perform certain actions to fix the situation.
Finally, we copy the
go-demo binary from the host to the
/usr/local/bin/ directory inside the image and give it executable permissions with the
To some, the order of the statements might not look logical. However, there is a good reason behind such declarations and their order. Those that are less likely to change are defined before those that are prone to changes. Since
go-demo will be a new binary every time we build the images, it is defined last.
The reasons behind such order lie in the way Docker Engine creates images. It starts from the top-most definition and checks whether it changed since the last time the build was run. If it didn't, it moves to the next statement. As soon as it finds a statement that would produce a new image, it, and all the statements following it are built into Docker images. By placing those that are less likely to change closer to the top, we can reduce the build time, disk usage, and bandwidth.
Now that we understand the Dockerfile behind the
go-demo service, we can build the images.
The command is very straightforward and is as follows:
docker build -t go-demo .
As an alternative, we can define build arguments inside a Docker Compose file. The service defined in
docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-local.yml) file is as follows:
app: build: . image: go-demo
In both cases, we specified that the current directory should be used for the build process
. and that the name of the image is
We can run the build through Docker compose with the command that is as follows:
docker-compose \ -f docker-compose-test-local.yml \ build app
We'll use the latter method throughout the rest of the book.
We can confirm that the image was indeed built, by executing the
docker images command as follows:
The output is as follows:
REPOSITORY TAG IMAGE ID CREATED SIZE go-demo latest 5e90126bebf1 49 seconds ago 23.61 MB golang 1.608a89f0a4ee5 11 hours ago 744.2 MB alpine latest 4e38e38c8ce0 9 weeks ago 4.799 MB
As you can see,
go-demo is one of the images we have inside the server.
Now that the images are built, we can run staging tests that depend on the service and its dependencies to be deployed on a server.
Please note that the real purpose of this step in the CI flow is to run the tests that require the service and its dependencies to be running. Those are still not integration tests that require production or production-like environment. The idea behind those tests is to run the service together with its direct dependencies, run the tests, and, once they're finished, remove everything and free the resources for some other task. Since these are still not integration tests, some, if not all, dependencies can be mocks.
Due to the nature of these tests, we need to split the task into three actions:
- Run the service and all the dependencies.
- Run the tests.
- Destroy the service and all the dependencies.
The dependencies are defined as the
staging-dep service inside the
docker-compose-test-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-test-local.yml) file. The definition is as follows:
staging-dep: image: go-demo ports: -8080:8080 depends_on: - db db: image: mongo:3.2.10
The image is
go-demo, and it exposes the port
8080 (both on the host and inside the container). It depends on the service
db which is a
mongo image. Services defined as
depends_on will be run before the service that defines the dependency. In other words, if we run the
staging-dep target, Compose will run the
Let's run the dependencies as shown in the following code:
docker-compose \ -f docker-compose-test-local.yml \ up -d staging-dep
Once the command is finished, we will have two containers running (
db). We can confirm that by listing all the processes:
docker-compose \ -f docker-compose-test-local.yml \ ps
The output is as follows:
Name Command State Ports --------------------------------------------------------------------- godemo_db_1 /entrypoint.sh mongod Up 27017/tcp godemo_staging-dep_1 go-demo Up 0.0.0.0:8080->8080/tcp
Now that the service and the database it depends on are running, we can execute the tests. They are defined as the service staging. The definition is as follows:
staging: extends: service: unit environment: - HOST_IP=localhost:8080 network_mode: host command: bash -c"go get -d -v -t && go test --tags integration -v"
Since the definition of the staging tests is very similar to those we run as unit tests, the staging service extends unit. By extending a service, we inherit its full definition. Further on, we defined an environment variable
HOST_IP. The tests code uses that variable to determine the location of the service under test. In this case, since the
go-demo service is running on the same server as tests, the IP is server's localhost. Since, by default, localhost inside a container is not the same as the one on the host, we had to define
host. Finally, we defined the command that should be executed. It will download tests dependencies
go get -d -v -t and run the tests
go test --tags integration -v.
Let's run the following commands:
docker-compose \ -f docker-compose-test-local.yml \ run --rm staging
All the tests passed, and we are one step closer to the goal of having full confidence that the service is indeed safe to be deployed to production.
We don't have any use for keeping the service and the database running so let's remove them and free the resources for some other task:
docker-compose \ -f docker-compose-test-local.yml \ down
down command stops and removes all services defined in that Compose file. We can verify that by running the following
docker-compose \ -f docker-compose-test-local.yml \ ps
The output is as follows:
Name Command State Ports ------------------------------
There is only one thing missing for the CI flow to be complete. At this moment we have the
go-demo image that is usable only inside the go-demo server. We should store it in a registry so that it can be accessed from other servers as well.
Before we push our
go-demo image, we need a place to push to. Docker offers multiple solutions that act as a registry. We can use Docker Hub (https://hub.docker.com/), Docker Registry (https://docs.docker.com/registry/), and Docker Trusted Registry (https://docs.docker.com/docker-trusted-registry/). On top of those, there are many other solutions from third party vendors.
Which registry should we use? Docker Hub requires a username and password, and I do not trust you enough to provide my own. One of the goals I defined before I started working on the book is to use only open source tools so Docker Trusted Registry, while being an excellent choice under different circumstances, is also not suitable. The only option left (excluding third party solutions), is Docker Registry (https://docs.docker.com/registry/).
The registry is defined as one of the services inside the
docker-compose-local.yml (https://github.com/vfarcic/go-demo/blob/master/docker-compose-local.yml) Compose file. The definition is as follows:
registry: container_name: registry image: registry:2.5.0 ports: -5000:5000 volumes: - .:/var/lib/registry restart: always
We set registry as an explicit container name, specified the image, and opened the port
5000 (both on the host and inside the container).
Registry stores the images inside the
/var/lib/registry directory, so we mounted it as a volume on the host. That way, data will not be lost if the container fails. Since this is a production service that could be used by many, we defined that it should always be restarted on failure.
Let's run the following commands:
docker-compose \ -f docker-compose-local.yml \ up -d registry
Now that we have the registry, we can do a dry-run. Let's confirm that we can pull and
push images to it:
docker pull alpine docker tag alpine localhost:5000/alpine docker push localhost:5000/alpine
Docker uses a naming convention to decide where to pull and push images from. If the name is prefixed with an address, the engine will use it to determine the location of the registry. Otherwise, it assumes that we want to use Docker Hub. Therefore, the first command pulled the alpine image from Docker Hub.
The second command created a tag of the alpine image. The tag is a combination of the address of our registry
localhost:5000 and the name of the image. Finally, we pushed the
alpine image to the registry running on the same server.
Before we start using the registry in a more serious fashion, let's confirm that the images are indeed persisted on the host:
ls -1 docker/registry/v2/repositories/alpine/
The output is as follows:
_layers _manifests _uploads
I won't go into details what each of those sub-directories contains. The important thing to note is that registry persists the images on the host so no data will be lost if it fails or, in this case, even if we destroy the VM since that Machine directory is mapped to the same directory on our laptop.
We were a bit hasty when we declared that this registry should be used in production. Even though data is persisted, if the whole VM crashes, there would be a downtime until someone brings it up again or creates a new one. Since one of the goals is to avoid downtime whenever possible, later on, we should look for a more reliable solution. The current setup should do for now.
Now we are ready to push the
go-demo image to the registry:
docker tag go-demo localhost:5000/go-demo:1.0 docker push localhost:5000/go-demo:1.0
As with the Alpine example, we tagged the image with the registry prefix and pushed it to the registry. We also added a version number
The push was the last step in the CI flow. We run unit tests, built the binary, built the Docker image, ran staging tests, and pushed the image to the registry. Even though we did all those things, we are not yet confident that the service is ready for production. We never tested how it would behave when deployed to a production (or production-like) cluster. We did a lot, but not enough.
If CI were our final objective, this would be the moment when manual validations should occur. While there is a lot of value in manual labor that requires creativity and critical thinking, we cannot say the same for repetitive tasks. Tasks required for converting this Continuous Integration flow into Continuous Delivery and, later on, deployment are, indeed repetitive.
We have the CI process done, and it is time to do the extra mile and convert it into Continuous Delivery.
Before we move into the steps required for the Continuous Integration process to become Continuous Delivery, we need to take a step back and explore cluster management. After all, in most cases, there is no production environment without a cluster.
We'll destroy the VMs at the end of each chapter. That way, you can come back to any of part of the book and do the exercises without the fear that you might need to do some steps from one of the earlier chapters. Also, such a procedure will force us to repeat a few things. Practice makes perfect. To reduce your waiting times, I did my best to keep things as small as possible and keep download times to a minimum. Execute the following command:
docker-machine rm -f go-demo
The next chapter is dedicated to the setup and operation of a Swarm cluster.