Advanced Container Resource Analysis

In this article by Russ McKendrick author of the book Monitoring Docker, we will see how you can use the API built into Docker to gain an insight to what resources your containers are running. Now, we are to see how we can take it to the next level by using cAdvisor from Google. In this article, you will cover the following topics:

  • How to install cAdvisor and start collecting metrics
  • Learn all about the web interface and real-time monitoring

What your options are for shipping metrics to a remote Prometheus database for long-term storage and trend analysis

(For more resources related to this topic, see here.)

What is cAdvisor?

Google describes cAdvisor as follows:

cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers. It is a running daemon that collects, aggregates, processes, and exports information about running containers. Specifically, for each container, it keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage, and network statistics. This data is exported by a container and is machine-wide.

The project started off life as an internal tool at Google for gaining an insight into containers that had been launched using their own container stack.

cAdvisor is written in Go (https://golang.org); you can either compile your own binary or you can you the pre-compiled binary that is supplied via a container, which is available from Google's own Docker Hub account. You can find this at http://hub.docker.com/u/google/.

Once installed, cAdvisor will sit in the background and capture metrics that is similar to that of the docker stats command.

cAdvisor takes these metrics along with those for the host machine and exposes them via a simple and easy-to-use built-in web interface.

Running cAdvisor using a container

There are a number of ways to install cAdvisor; the easiest way to get started is to download and run the container image that contains a copy of a precompiled cAdvisor binary.

Before running cAdvisor, lets launch a fresh vagrant host:

[russ@mac ~]$ cd ~/Documents/Projects/monitoring-docker/vagrant-centos/
[russ@mac ~]$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==>default: Importing base box 'russmckendrick/centos71'...
==>default: Matching MAC address for NAT networking...
==>default: Checking if box 'russmckendrick/centos71' is up to date...
.....
==>default: => Installing docker-engine ...
==>default: => Configuring vagrant user ...
==>default: => Starting docker-engine ...
==>default: => Installing docker-compose ...
==>default: => Finished installation of Docker
[russ@mac ~]$ vagrantssh

Using a backslash
As we have a lot options to pass to the docker run command, we are using \ to split the command over multiple lines so its easier to follow what is going on.

Once you have access to the host machine, run the following command:

docker run \
  --detach=true \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --privileged=true \
  --name=cadvisor \
google/cadvisor:latest

You should now have a cAdvisor container up and running on your host machine. Before we start, let's look at cAdvisor in more detail by discussing why we have passed all the options to the container.

The cAdvisor binary is designed to run on the host machine alongside the Docker binary, so by launching cAdvisor in a container, we are actually isolating the binary in its down environment. To give cAdvisor access to the resources on the host machine it needs, we have to mount several partitions and also give the container privileged access to let the cAdvisor binary to think it being executed on the host machine.

Compiling cAdvisor from source

As mentioned in the previous section, cAdvisor really ought to be executed on the host machine; this means, you may have a use case to compile your own cAdvisor binary and run it directly on the host.

To compile cAdvisor, you will need to perform the following steps:

  1. Install Go and Mercurial on the host machine—version 1.3 or higher of Go is needed to compile cAdvisor.
  2. Set the path for Go to work from.
  3. Grab the source code for cAdvisor and godep.
  4. Set the path for your Go binaries.
  5. Build the cAdvisor binary using godep to source the dependencies for us.
  6. Copy the binary to /usr/local/bin/.
  7. Download either an Upstart or Systemd script and launch the process.

If you followed the instructions in the previous section, you will already have a cAdvisor process running. Before compiling from source, you should start with a clean host; let's log out of the host and launch a fresh copy:

[vagrant@centos7 ~]$ exit
logout
Connection to 127.0.0.1 closed.
[russ@mac ~]$ vagrant destroy
default: Are you sure you want to destroy the 'default' VM? [y/N] y
==>default: Forcing shutdown of VM...
==>default: Destroying VM and associated drives...
==>default: Running cleanup tasks for 'shell' provisioner...
[russ@mac ~]$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==>default: Importing base box 'russmckendrick/centos71'...
==>default: Matching MAC address for NAT networking...
==>default: Checking if box 'russmckendrick/centos71' is up to date...
.....
==>default: => Installing docker-engine ...
==>default: => Configuring vagrant user ...
==>default: => Starting docker-engine ...
==>default: => Installing docker-compose ...
==>default: => Finished installation of Docker
[russ@mac ~]$ vagrantssh

To build cAdvisor on the CentOS 7 host, run the following command:

sudo yum install -y golanggit mercurial
export GOPATH=$HOME/go
go get -d github.com/google/cadvisor
go get github.com/tools/godep
export PATH=$PATH:$GOPATH/bin
cd $GOPATH/src/github.com/google/cadvisor
godep go build .
sudocpcadvisor /usr/local/bin/
sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96771/raw/86b01a044006f85eebbe395d3857de1185ce4701/cadvisor.service -O /lib/systemd/system/cadvisor.service
sudosystemctl enable cadvisor.service
sudosystemctl start cadviso

On the Ubuntu 14.04 LTS host, run the following command:

sudo apt-get -y install software-properties-common
sudo add-apt-repository ppa:evarlast/golang1.4
sudo apt-get update
sudo apt-get -y install golang mercurial
export GOPATH=$HOME/go
go get -d github.com/google/cadvisor
go get github.com/tools/godep
export PATH=$PATH:$GOPATH/bin
cd $GOPATH/src/github.com/google/cadvisor
godep go build .
sudocpcadvisor /usr/local/bin/
sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96771/raw/e12c100d220d30c1637bedd0ce1c18fb84beff77/cadvisor.conf -O /etc/init/cadvisor.conf
sudo start cadvisor

You should now have a running cAdvisor process. You can check this by running ps aux | grepcadvisor and you should see a process with a path of /usr/local/bin/cadvisor running.

Collecting metrics

Now, you have cAdvisor running; what do you need to do to configure the service in order to start collecting metrics? The short answer is, nothing at all. When you started the cAdvisor process, it instantly started polling your host machine to find out what containers are running and gathered information on both of them and your host machine.

The Web interface

cAdvisor should be running on the 8080 port; if you were to open http://192.168.33.10:8080/, you should be greeted with the cAdvisor logo and an overview of your host machine:

This initial page streams live stats about the host machine, though each section is repeated when you start to drill down and view the containers. To start with, let's look at each section using the host information.

Overview

This overview section gives you a bird's-eye view of your system; it uses gauges so you can quickly get an idea of which resources are reaching their limits. In the following screenshot, there is very little in the way of CPU utilization and the file system usage is relatively low; however, we are using 64% of the available RAM:

Processes

The following screenshot displays a combined view of the output of the ps aux, dockerps and top commands:

Here is what each column heading means:

  • User: This shows which user is running the process.
  • PID: This is the unique process ID
  • PPID: This is the PID of the parent process
  • Start Time: This shows what time the process started
  • CPU %: This is the percentage of the CPU the process is currently consuming
  • MEM %: This is the percentage of the RAM the process is currently consuming
  • RSS: This shows how much of the the main memory the process is using
  • Virtual Size: This shows how much of the the virtual memory the process is using
  • Status: This shows the current status of the process; these are the standard Linux process state codes
  • Running Time: This shows for how long the process has been running
  • Command: This is shows which command the process is running
  • Container: This shows which container the process is attached to; the container listed as / is the host machine

As there could be several hundred processes active, this section is split into pages; you can navigate to these with the buttons on the bottom-left. Also, you can sort the processes by clicking on any of the headings.

CPU

The following graph shows the CPU utilization over the last minute:

Here is what each term means:

  • Total Usage: This shows an aggregate usage across all cores.
  • Usage per Core: This graph breaks down the usage per core.
  • Usage Breakdown (not shown in the preceding screenshot): This shows aggregate usage across all cores, but breaks it down to what is being used by kernel and what is being used by the user-owned processes.

Memory

The Memory section is split into two parts. The graph tells you the total amount of memory used by all the processes for the host or container; this is the total of the hot and cold memory. The Hot memory is the current working set; these are pages that have been touched by the kernel recently. The Cold memory is the page that hasn't been touched for a while and could be reclaimed if needed.

The Usage Breakdown gives a visual representation of the total memory in the host machine, or allowance in the container, alongside the total and hot usage:

Network

This section shows the incoming and outgoing traffic over the last minute. You can change the interface using the drop-down box on the top-left. There is also a graph that shows any networking errors. Typically, this graph should be flat; if it isn't, then you will be seeing performance issues with your host machine or container:

Summary

In this article, we learned how to take the viewing real-time statistics of our containers off the command line and into the web browser. We explored a few different methods to install Google's cAdvisor application and also how to use its web interface to keep an eye on our running containers. We also learned how to capture metrics from cAdvisor and store them using Prometheus, a modern time series database.

Resources for Article:


Further resources on this subject:


You've been reading an excerpt of:

Monitoring Docker

Explore Title