Docker Multi-Host Networking Experiments on Amazon AWS

Felix Rabe

December 30th, 2015

So far (August 2015), Docker only provided single-host networking configuration. This means that Docker containers could only be configured to interact with each other when running on the same host, connected via the linking mechanism.

This is about to change. The current experimental builds also include multi-host networking via the overlay driver. This allows to find and connect containers across multiple hosts by their name, resolved via /etc/hosts as in the current linking mechanism.

This article will guide you to your first overlay networking hello world experiment on Amazon AWS. We'll set up one Consul server instance for service discovery, and two further instances that we then can connect to each other via overlay networking.

Please note that experimental Docker features such as overlay networking are under active development and subject to change. Docker Machine is also relatively new and has its rough edges. If you encounter any problems with it, remove machines with docker-machine rm, and add the --debug argument to its commands, e.g. docker-machine --debug create.

Prerequisites: Docker Tools

We will make use of two tools from the Docker Toolbox:

  • Docker Machine to set up the Amazon AWS EC2 instances.
  • Docker Engine to interface with the Docker daemons in the EC2 instances.

You can either install Docker Machine as part of the Docker Toolbox, or install it separately by downloading a recent binary release from GitHub.

To make use of the new Docker networking features, you will need to install the experimental Docker Engine. Get the latest binary according to your operating system and install (or symlink) it into your binary search path as docker (docker.exe on Windows):

I am no Windows user and will use the Bash shell (found on OS X and Linux) for this article. Commands may thus have a different syntax on Windows.

Prerequisites: Amazon AWS

You will require an account with Amazon AWS.

Optionally, you can install and configure the Amazon AWS Command Line Interface according to the setup docs.

Environment Variables

(If you'd rather read a Bash script than prose for this section, head over to https://gist.github.com/felixrabe/cf076655f1d0235e28a3.)

Go to your Amazon AWS VPC console (https://console.aws.amazon.com/vpc/home), take note of the region you are in (this article uses us-west-2 in Oregon), and list your VPCs under "Your VPCs". You should have exactly one listed, if not, you probably know which one to pick. Note its "VPC ID", you will need it later on.

As the machine image, we'll use ami-5189a661 (Ubuntu Server 14.04 LTS (HVM), SSD Volume Type). If you choose to run in a different region, the AMI ID will be different.

Now is the time to open a Terminal and put a few environment variables in place for the Docker Machine AWS driver. (All of these can also be specified as command line arguments.)

First, your Amazon AWS access keys:

export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx

Or if you have the AWS CLI installed, you can directly read these keys from your ~/.aws/credentials file:

export AWS_ACCESS_KEY_ID=$(   grep -e ^aws_access_key_id     ~/.aws/credentials | sed 's/.*= *//g')
export AWS_SECRET_ACCESS_KEY=$(grep -e ^aws_secret_access_key ~/.aws/credentials | sed 's/.*= *//g')

Then the regional settings (insert your own VPC ID here):

export AWS_DEFAULT_REGION=us-west-2
export AWS_AMI=ami-5189a661
export AWS_VPC_ID=vpc-xxx

Consul

At this moment, Docker overlay networking requires an external key value store for service discovery — see the docs here and here (plus this issue) to find out if that is still the case. We'll be using Consul in the following step.

Create your first (in this session at least) Docker machine to run Consul for service discovery:

docker-machine create -d amazonec2 --amazonec2-root-size 8 \
   --engine-install-url "https://experimental.docker.com" \
   consul

You will later need to connect to this machine via its private IP address, which you can obtain using:

consul_private=$(docker-machine inspect consul --format '{{.Driver.PrivateIPAddress}}')
echo "$consul_private"

(Be aware that this article will not provide you a bullet-proof setup: Once Consul gets restarted, AWS will (likely) give it a new IP address, which then needs to be communicated via some configuration file editing to the following instances.)

Start the Consul server using:

docker $(docker-machine config consul) run -d --restart always \
   --name consul -p 8500:8500 -h consul \
   progrium/consul -server -bootstrap

(This command substitution here is not so robust either, though it works (most likely). The better (and often-documented) solution would be to use docker-machine env.)

Ports

Docker Machine creates a security group called docker-machine. It defines what network ports are available between instances and to the Internet. Docker Machine already opens TCP ports 22 (SSH) and 2376 (Docker). We need to add UDP port 4789 (VXLAN used by the overlay networking driver) and TCP ports 7946 and 8500 (Consul). And because it's nice for debugging, let's add ICMP (for ping) too.

If you don't have the AWS CLI installed, you can do this in your web console:

Docker Machine Security Group

Or, with AWS CLI installed, you can run:

aws ec2 --region "$AWS_DEFAULT_REGION" authorize-security-group-ingress \
   --group-name docker-machine --source-group docker-machine \
   --protocol icmp --port -1

aws ec2 --region "$AWS_DEFAULT_REGION" authorize-security-group-ingress \
   --group-name docker-machine --source-group docker-machine \
     --protocol udp --port 4789

aws ec2 --region "$AWS_DEFAULT_REGION" authorize-security-group-ingress \
   --group-name docker-machine --source-group docker-machine \
     --protocol tcp --port 7946

aws ec2 --region "$AWS_DEFAULT_REGION" authorize-security-group-ingress \
   --group-name docker-machine --source-group docker-machine \
     --protocol tcp --port 8500

Machine One

This is the first EC2 instance that we'll use directly in our experiment:

docker-machine create -d amazonec2 --amazonec2-root-size 8 \
   --engine-install-url "https://experimental.docker.com" \
   --engine-opt         "default-network=overlay:multihost" \
   --engine-opt         "kv-store=consul:$consul_private:8500" \
   --engine-label       "com.docker.network.driver.overlay.bind_interface=eth0" \
   aws-1

The overlay network driver currently requires Linux 3.16. (I tried to use Ubuntu 15.04 (ami-efd2c6df) instead of 14.04 LTS to avoid this step, but that broke docker-machine.) Installing a newer kernel on the instance will satisfy this requirement:

docker-machine ssh aws-1 'sudo DEBIAN_FRONTEND=noninteractive apt-get install -qqy linux-image-generic-lts-vivid'
docker-machine restart aws-1

We also need this instance's private IP address:

aws_1_private=$(docker-machine inspect aws-1 --format '{{.Driver.PrivateIPAddress}}')
echo "$aws_1_private"

Machine Two

The setup of the second instance works almost the same way, but for a different name and one additional argument:

docker-machine create -d amazonec2 --amazonec2-root-size 8 \
   --engine-install-url "https://experimental.docker.com" \
   --engine-opt         "default-network=overlay:multihost" \
   --engine-opt         "kv-store=consul:$consul_private:8500" \
   --engine-label       "com.docker.network.driver.overlay.bind_interface=eth0" \
   --engine-label       "com.docker.network.driver.overlay.neighbor_ip=$aws_1_private" \
   aws-2

docker-machine ssh aws-2 'sudo DEBIAN_FRONTEND=noninteractive apt-get install -qqy linux-image-generic-lts-vivid'
docker-machine restart aws-2

Hello World

Finally, we can run two Docker containers on two hosts that know about each other:

docker $(docker-machine config aws-1) run --name one -d -ti busybox
docker $(docker-machine config aws-2) run --name two --rm -ti busybox cat /etc/hosts
docker $(docker-machine config aws-2) run --name two --rm -ti busybox ping -c 3 one

If you enjoyed this exploration of an experimental Docker feature, make sure to frequent the Docker Blog to stay up-to-date.