Chapter 9. Convolutional Neural Networks

In this chapter, we are discussing the convolutional neural networks (CNNs). At first we are going to discuss all components with examples in Swift just to develop an intuition about the algorithm and what is going on under the hood. However, in the real life you most likely will not develop CNN from scratch, because you will use some ready available and battle-tested deep learning framework.

So, in the second part of the chapter we will show a full development cycle of deep learning mobile application. We are going to take the photos of people's faces labeled with their emotions, train a CNN on a GPU workstation, and then integrate it into an iOS application using Keras, Vision, and Core ML frameworks.

To the end of this chapter you will have learned about:

• Affective computing
• Computer vision, its tasks, and its methods
• CNNs, their anatomy, and core concepts behind them
• Applications of CNNs in computer vision
• How to train CNNs using a GPU workstation and...

Understanding users emotions

While voice input is undoubtedly a useful feature, we all well know how the actual meaning of the sentence can be opposite to the literal one, depending on the speaker's intonation, facial expression, and context. Try this simple sentence: Oh, really? Depending on the conditions, this can mean: I doubt, I didn't know, I'm impressed, I don't care, This is obvious, and so on. The problem is that speech is not the only mode of conversation for human beings, and that's why much research is focused these days on teaching computers to understand (and also simulate) gestures, facial expressions, sentiments in a text, eye movements, sarcasm, and other affect manifestations. An interdisciplinary field that emerges around the question of emotional and compassionate AI is known as affective computing. It integrates knowledge from the computer and cognitive sciences, as well as psychology and robotics. The aim is the creation of computer systems that will adapt themselves...

Introducing computer vision problems

In this book, we mentioned computer vision several times, but since this chapter is focused on this particular domain, we will look at it in more detail now. There are several practical tasks related to image and video processing, which are referred to as computer vision domain. While working on some computer vision task, it's important to know these names, to be able to find what you need in the vast ocean of computer vision publications:

• Object recognition: The same as classification. Assigning labels to the images. This is a cat. Age estimation. Facial expression recognition.
• Object localization: Finding frame of object in the image. The cat is in this frame.
• Object detection: Finding frames of objects in the image. The cat is in this frame.
• Semantic segmentation: Each point in the picture is assigned to one class. If the picture contains several cats, each cat's pixel would be assigned to the cat class.
• Instance segmentation: Each point in the picture...

Introducing convolutional neural networks

CNNs, or ConvNets have gotten a lot of attention in the last few years, mainly due to their major successes in the domain of computer vision. They are at the core of most computer vision systems nowadays, including self-driving cars and large-scale photo classification systems.

In some sense, CNNs are very similar to multilayer perceptron, which we have discussed in the previous chapter. These networks also build from the layers, but unlike MLP, which usually has all layers similar to each other, CNNs usually include many layers of different types. And the most important type of the layer is (surprise, surprise) the convolutional layer. Modern CNNs can be really deep—hundreds of different layers. Nevertheless, you can still see the whole network as one differentiable function that takes some input (usually raw values of image pixels), and produces some output (for example, class probabilities: 0.8 cat, 0.2 dog).

Pooling operation

Pooling or subsampling is a simple operation of input size decreasing (Figure 9.2). If you have a black and white image, and you want to decrease its size, you can do it in the following way: chose a sliding window of size n × m and stride s. Go through the image, applying sliding window and shifting on the s pixels every time you want to move your window. At each position calculate an average (for average pooling) or maximum (for max pooling) and record this value into the destination matrix. Now, there are two common ways to handle borders of the image:

Figure 9.2. Pooling operation. Grey window in the source image corresponds to the grey cell in the destination image

The pooling is used in the CNNs to reduce the size of the data, as it travels down the network.

Convolution operation

Convolution is one of the most important operations in the image processing. Blurring, sharpening, edge detection, denoising, embossing and many other familiar operations in image editors are actually convolutions. It is similar to the pooling operation in some way, because it is also a sliding window operation, but instead of taking the average over the window, it performs element-wise multiplication by the kernel – matrix of size n × n and sums the result. The result of the operation depends on the kernel (also known as convolution filter) – a matrix, which is usually square, but not necessarily, see Figure 9.3. The notions of the stride and padding are the same as in the pooling case:

Figure 9.3: Different convolution filters have different effects on the picture

Convolution operation works in the following way (see the following diagram):

• The convolution kernel (filter) slides over the image from left to right, and from top to bottom
• At each position, we calculate an...

Building the network

When you first encounter a variety of architectures of CNNs, you feel overwhelmed by the abundance of new terms, different layers, and their hyperparameters. In fact, at the moment, only a few architectures have found broad application, and the number of designs suitable for mobile development is even smaller.

There are five basic types of layers plus an input layer, which usually does nothing except passing data forward:

• Input layer: The first layer in the neural network. It does nothing, only takes the input and passes it downstream.
• Convolution layers: Where convolutions happen
• Fully: Connected or dense layers
• Nonlinearity layers: These are layers which apply activation functions to the output of the previous layer: sigmoid, ReLU, tanh, softmax and so on.
• Pooling layers: Downsample their input.
• Regularization layers: layers to fight an overfitting.

Note

Modern deep learning frameworks contain much more different types of layers for all needs, but these are the most commonly...

Loss functions

Loss function is a necessary part, because it is what we want to minimize during the training. You can find a few popular loss functions in the table:

Name	Formula	Usually used for
Mean squared error or L2-loss		Regression
Mean absolute error or L1-loss		Regression
Categorical cross entropy		Softmax multiclass classification

 

Where y is a ground-truth vector and ŷ is a vector of predictions of length n.

Training the network

Stochastic gradient descent (SGD) is an effective way of training deep neural networks. SGD seeks such parameters Θ of the network, which minimize the loss function ℒ.

Where 

 is a training dataset.

Training happens in steps. At every step, we choose a subset of our training set of size m (mini-batch) and use it to approximate loss function gradient with respect to parameters Θ:

Mini-batch training advantages are as follows:

• Gradient of the loss function over a mini-batch is a better approximation of the gradient over the whole training set then calculated over only one sample
• Thanks to the GPU you can perform computations in parallel on every sample in the batch, which is faster, then processing them one-by-one

 

Training the CNN for facial expression recognition

For the demonstration of the CNNs we will implement a simple neural network for emotion recognition. We will use the dataset of face expressions fer2013 from the ICML 2013 contest Facial Expression Recognition Challenge [1].

Note

The dataset can be downloaded from the kaggle site:

https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data

You will be asked to register and accept the terms and conditions.

The archive fer2013.tar.gz contains fer2013.csv with the dataset itself and some supplementary information files. The .csv file contains 35,887 samples, of which 28,709 marked as training set, 3,589 as public test, and 3,589 private test. There are three columns in the table: emotion, pixels and usage. Every sample is a grayscale 48 × 48 pixels face photo in a form of pixel array. The faces were cropped in an automatic way, so there are some false-positives in the dataset (non-faces and cartoon...

Environment setup

To train the deep CNN, you will need a computer with a CUDA-compatible GPU. I used an Ubuntu 16.x machine with NVidia GTX980 GPU for model training, and a macOS machine to convert the model to Core ML format. If you don't have CUDA-compatible GPU, you can try to train the model on CPU; but be aware that this will take a lot of time. Also, the trained model for this chapter is available in the supplementary materials, so if you prefer not to contribute to the global warming by retraining the model from scratch, it's also possible.

Here's a list of what should be installed on your system to train the network:

• Latest NVIDIA drivers
• CUDA 8.0
• cuDNN 5.1
• Python 2.7
• tensorflow-gpu (or TensorFlow for CPU-only mode)
• Keras
• Keras-viz
• Matplotlib, Pandas

Please, refer to the official sites for the installation instructions.

Deep learning frameworks

There are a plenty of deep learning toolkits and libraries for different kinds of platforms. For a long time, the three most popular of them were Theano (Python), Torch (Lua), and Caffe (C++). Somehow, Caffe became an industrial standard, while Theano and Torch were mostly used among researchers. I call these three libraries the first generation of deep learning frameworks. Most of the pre-trained neural networks that are available on the internet are still in Caffe format. They had their own problems, so the next generation of frameworks followed in several years. If the first generation was created mainly by efforts of individual researchers, the second generation was pushed by big IT companies. Today, apart from Apple, every internet giant has its own open source deep learning framework: Google has TensorFlow and Keras, Microsoft has CNTK, Facebook released Caffe 2, and Torch was reborn as PyTorch, thanks to Twitter and Facebook. Amazon has chosen MXNet as its...

Loading the data

As usual, first we add some magic to display images inline in the Jupyter:

%matplotlib inline 

We're using Pandas to handle our data:

import pandas 

Please, visit the Kaggle site and download the dataset: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge

Load the dataset into the memory:

data = pandas.read_csv("fer2013/fer2013.csv") 

Dataset consists of gray scale face photos encoded as pixel intensities. 48 x 48 gives 2304 pixels for each. Every image is marked according to the emotion on the face.

data.head() 
emotion  pixels   Usage 
0  0  70 80 82 72 58 58 60 63 54 58 60 48 89 115 121...  Training 
1  0  151 150 147 155 148 133 111 140 170 174 182 15...  Training 
2  2  231 212 156 164 174 138 161 173 182 200 106 38...  Training 
3  4  24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1...  Training 
4  6  4 0 0 0 0 0 0 0 0 0 0 0 3 15 23 28 48 50 58 84...  Training 
How many faces of each class do we have? 
 
data.emotion.value_counts...

Splitting the data

Do not forget to split your data into training and test sets before training the model as shown in the following:

train_set = data[(data.Usage == 'Training')] 
test_set = data[(data.Usage != 'Training')] 
X_train = np.array(map(str.split, train_set.pixels), np.float32) 
X_test = np.array(map(str.split, test_set.pixels), np.float32) 
(X_train.shape, X_test.shape) 
((28273, 2304), (7067, 2304)) 
48*48 
2304 
X_train = X_train.reshape(28273, 48, 48, 1) 
X_test = X_test.reshape(7067, 48, 48, 1) 
(X_train.shape, X_test.shape) 
((28273, 48, 48, 1), (7067, 48, 48, 1)) 
num_train = X_train.shape[0] 
num_test = X_test.shape[0] 
(num_train, num_test) 
(28273, 7067) 

Converting labels to categorical:

from keras.utils import np_utils # utilities for one-hot encoding of ground truth values 
Using TensorFlow backend. 
y_train = train_set.emotion 
y_train = np_utils.to_categorical(y_train, num_classes) 
y_test = test_set.emotion 
y_test = np_utils.to_categorical(y_test, num_classes) 

Data augmentation

In the deep learning applications, generally, the more data you have, the better. Deep neural networks usually have a lot of parameters, so on the small datasets they overfit easily. We can generate more training samples from the samples we already have by using the technique called data augmentation. The idea is to change samples at random. With the face photos, we could, for example, flip faces horizontally, shift them a bit, or add some rotations:

from keras.preprocessing.image import ImageDataGenerator  
datagen = ImageDataGenerator( 
    rotation_range=25, 
    width_shift_range=0.2, 
    height_shift_range=0.2, 
    horizontal_flip=True) 

Compute quantities required for featurewise normalization (std, mean, and principal components, if ZCA whitening is applied):

 datagen.fit(X_train)
 batch_size = 32 

At each iteration, we will consider 32 training examples at once, in other words, our batch size is 32. Let's see our images after augmentation:

from matplotlib import pyplot...

Creating the network

Keras allows building the deep neural networks by adding new layers one by one. Note, that all layers should be familiar to you to this moment.

from keras.models import Sequential 
from keras.layers import Activation, Dropout, Flatten, Dense, BatchNormalization, Conv2D, MaxPool2D 
model = Sequential() 
 
model.add(Conv2D(16, (3, 3), padding='same', activation='relu', input_shape=(height, width, depth))) 
model.add(Conv2D(16, (3, 3), padding='same')) 
model.add(BatchNormalization()) 
model.add(Activation('relu')) 
model.add(MaxPool2D((2,2))) 
 
model.add(Conv2D(32, (3, 3), padding='same', activation='relu')) 
model.add(Conv2D(32, (3, 3), padding='same')) 
model.add(BatchNormalization()) 
model.add(Activation('relu')) 
model.add(MaxPool2D((2,2))) 
 
model.add(Conv2D(64, (3, 3), padding='same', activation='relu')) 
model.add(Conv2D(64, (3, 3), padding='same')) 
model.add(BatchNormalization()) 
model.add(Activation('relu')) 
model.add(MaxPool2D((2,2))) 
 
model.add(Flatten...

Plotting the network structure

Perhaps, the more convenient way to explore the structure of the network is to draw a picture. Let's do that:

from IPython.display import SVG 
from keras.utils.vis_utils import model_to_dot 
 
SVG(model_to_dot(model, show_shapes=True).create(prog='dot', format='svg')) 
 
from IPython.display import Image 
from keras.utils import plot_model 
plot_model(model, show_shapes=True, show_layer_names=True, to_file='model.png') 

See the Figure 9.10 for the result.

Training the network

First, we have to define how long we want to train out network. One epoch is one full pass over the training set. The number of steps in the epoch depends on the batch size and the number of samples in the training set. Let's say we want to pass over the training set 100 times:

num_epochs = 100

Fit the model on batches with real-time data augmentation:

num_epochs = 100 # we iterate 200 times over the entire training set 
history = model.fit_generator(train_flow, 
                    steps_per_epoch=len(X_train) / batch_size, 
                    epochs=num_epochs,  
                    verbose=1,  
                    validation_data=test_flow,  
                    validation_steps=len(X_test) / batch_size) 
Epoch 1/100 
883/883 [==============================] - 15s - loss: 1.7065 - acc: 0.2836 - val_loss: 1.8536 - val_acc: 0.1822 
Epoch 2/100 
883/883 [==============================] - 14s - loss: 1.4980 - acc: 0.4008 - val_loss: 1.5688 - val_acc: 0.3891 
... 
883/883...

Plotting loss

Loss values on training and validation sets allows to see, how our model improves over the time and decide when to stop training:

from matplotlib import pyplot as plt 
history.history.keys() 
['acc', 'loss', 'val_acc', 'val_loss'] plt.plot(history.history['loss']) 
plt.plot(history.history['val_loss']) 
plt.title('model loss') 
plt.ylabel('loss') 
plt.xlabel('epoch') 
plt.legend(['train', 'test'], loc='upper left') 
plt.show() 

Figure 9.11: Loss on training and test sets over the training epochs

Making predictions

First, let's prepare data to make predictions about the images:

array = np.mat(data.pixels[1]).reshape(48, 48) 
image = scipy.misc.toimage(array, cmin=0.0) 
display(image) 
print(emotion_labels[data.emotion[1]]) 
 
<Image> 
 

Let us input an angry emotion image:

input_img = np.array(array).reshape(1,48,48,1)

Okay, we have an angry face. Now let's make prediction and check if the network can recognize it correctly:

prediction = model.predict(input_img) 
print(prediction) 
[[ 0.05708674  0.35863262  0.03299783  0.17862292  0.00069717  0.37196276]] 
emotion_labels[prediction.argmax()] 
'Neutral' 

Note those array of 6 float numbers. These are probabilities of belonging to each class. In other words, the model predicts, that this face can be of an angry person only with the probability of 5%. The full table would look like this:

Angry	Fear	Happy	Sad	Surprise	Neutral
0.05708674	0.35863262	0.03299783	0.17862292	0.00069717	0.37196276

for i in xrange(1, 100): 
    array = np.mat(data.pixels...

Saving the model in HDF5 format

Saving the model is really easy as shown in the following:

model.save('Emotions.h5') 

Converting to Core ML format

The easiest way to use pre-trained CNN on iOS is by converting it to the Core ML format:

from keras.models import load_model
model = load_model('Emotions.h5')
coreml_model = convert(model, 
                       image_input_names = 'image', 
                       class_labels = emotion_labels)
...
coreml_model.save('Emotions.mlmodel')

Visualizing convolution filters

Debugging CNNs is notoriously difficult. One of the ways to check if the convolutional layers learned anything meaningful is to visualize their outputs using Keras-vis package:

from vis.utils import utils 
from vis.visualization import visualize_class_activation, get_num_filters 

We have to convert grayscale images to rgb to use them with keras-vis:

def to_rgb(im): 
    # I think this will be slow 
    w, h = im.shape 
    ret = np.empty((w, h, 3), dtype=np.uint8) 
    ret[:, :, 0] = im 
    ret[:, :, 1] = im 
    ret[:, :, 2] = im 
    return ret 

Names of the layers we want to visualize (consult model structure for exact layer names):

layer_names = ['conv2d_1', 'conv2d_2',  
               'conv2d_3', 'conv2d_4',  
               'conv2d_5', 'conv2d_6'] 
 
layer_sizes = [(80, 20), (80, 20),  
               (80, 40), (80, 40),  
               (80, 80), (80, 80)] 
 
stitched_figs = [] 
 
for (layer_name, layer_size) in zip(layer_names, layer_sizes): 
    layer_idx...

Deploying CNN to iOS

You need to drag-and-drop the Core ML file generated in the previous section into your project to start working with the model.

Imports:

import Foundation
import Vision
import AVFoundation
import UIKit

At first, let's define some data structures. An enumeration for possible classification results:

enum FaceExpressions: String {
 case angry = "angry"
 case anxious = "anxious"
 case neutral = "neutral"
 case happy = "happy"
 case sad = "sad"
}

An enum for errors of the classifier:

enum ClassifierError: Error {
 case unableToResizeBuffer
 case noResults
}

Classifier is a wrapper singleton for Core ML model:

class Classifier {
 public static let shared = Classifier()

 private let visionModel: VNCoreMLModel
 var visionRequests = [VNRequest]()
 var completion: ((_ label: [(FaceExpressions, Double)], _ error: Error?)->())?

private init() {
 guard let visionModel = try? VNCoreMLModel(for: Emotions().model) else {
 fatalError("Could not load model")
 }

 self.visionModel = visionModel...

Summary

In this chapter, we built a deep learning CNN, and trained it using Keras to recognize facial expressions on photos. Then we ported it for the mobile application using Core ML. The model can work in real time. We've also become acquainted with the Apple Vision framework.

CNNs are powerful tools that can be applied for many computer vision tasks, as well as for time-series prediction, natural language processing, and others. They are built around the concept of convolution—a mathematical operation that can be used for defining many types of image transformations. CNNs learn convolution filters in the similar manner as usual neural networks learn weights using the same stochastic gradient descent. Convolution requires less computations than usual matrix multiplications, which is why they can be effectively used on mobile devices. Apart from convolutional layers, CNNs usually include other types of layers like pooling, fully-connected, nonlinearity, regularization, and so on. Over the...

Bibliography

1. Challenges in Representation Learning: A report on three machine learning contests, I Goodfellow, D Erhan, PL Carrier, A Courville, M Mirza, B Hamner, W Cukierski, Y Tang, DH Lee, Y Zhou, C Ramaiah, F Feng, R Li, X Wang, D Athanasakis, J Shawe-Taylor, M Milakov, J Park, R Ionescu, M Popescu, C Grozea, J Bergstra, J Xie, L Romaszko, B Xu, Z Chuang, and Y. Bengio. arXiv 2013. Site of the competition http://deeplearning.net/icml2013-workshop-competition.
2. Affective Computing, Rosalind Picard. MIT Technical Report #32, 1995 http://affect.media.mit.edu/pdfs/95.picard.pdf.
3. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Sergey Ioffe, Christian Szegedy, 2015.

 

The rest of the chapter is locked

Saving the model in HDF5 format

Saving the model is really easy as shown in the following:

model.save('Emotions.h5') 

Converting to Core ML format

The easiest way to use pre-trained CNN on iOS is by converting it to the Core ML format:

from keras.models import load_model
model = load_model('Emotions.h5')
coreml_model = convert(model, 
                       image_input_names = 'image', 
                       class_labels = emotion_labels)
...
coreml_model.save('Emotions.mlmodel')

Visualizing convolution filters

Debugging CNNs is notoriously difficult. One of the ways to check if the convolutional layers learned anything meaningful is to visualize their outputs using Keras-vis package:

from vis.utils import utils 
from vis.visualization import visualize_class_activation, get_num_filters 

We have to convert grayscale images to rgb to use them with keras-vis:

def to_rgb(im): 
    # I think this will be slow 
    w, h = im.shape 
    ret = np.empty((w, h, 3), dtype=np.uint8) 
    ret[:, :, 0] = im 
    ret[:, :, 1] = im 
    ret[:, :, 2] = im 
    return ret 

Names of the layers we want to visualize (consult model structure for exact layer names):

layer_names = ['conv2d_1', 'conv2d_2',  
               'conv2d_3', 'conv2d_4',  
               'conv2d_5', 'conv2d_6'] 
 
layer_sizes = [(80, 20), (80, 20),  
               (80, 40), (80, 40),  
               (80, 80), (80, 80)] 
 
stitched_figs = [] 
 
for (layer_name, layer_size) in zip(layer_names, layer_sizes): 
    layer_idx...

Deploying CNN to iOS

You need to drag-and-drop the Core ML file generated in the previous section into your project to start working with the model.

Imports:

import Foundation
import Vision
import AVFoundation
import UIKit

At first, let's define some data structures. An enumeration for possible classification results:

enum FaceExpressions: String {
 case angry = "angry"
 case anxious = "anxious"
 case neutral = "neutral"
 case happy = "happy"
 case sad = "sad"
}

An enum for errors of the classifier:

enum ClassifierError: Error {
 case unableToResizeBuffer
 case noResults
}

Classifier is a wrapper singleton for Core ML model:

class Classifier {
 public static let shared = Classifier()

 private let visionModel: VNCoreMLModel
 var visionRequests = [VNRequest]()
 var completion: ((_ label: [(FaceExpressions, Double)], _ error: Error?)->())?

private init() {
 guard let visionModel = try? VNCoreMLModel(for: Emotions().model) else {
 fatalError("Could not load model")
 }

 self.visionModel = visionModel...

Summary

In this chapter, we built a deep learning CNN, and trained it using Keras to recognize facial expressions on photos. Then we ported it for the mobile application using Core ML. The model can work in real time. We've also become acquainted with the Apple Vision framework.

CNNs are powerful tools that can be applied for many computer vision tasks, as well as for time-series prediction, natural language processing, and others. They are built around the concept of convolution—a mathematical operation that can be used for defining many types of image transformations. CNNs learn convolution filters in the similar manner as usual neural networks learn weights using the same stochastic gradient descent. Convolution requires less computations than usual matrix multiplications, which is why they can be effectively used on mobile devices. Apart from convolutional layers, CNNs usually include other types of layers like pooling, fully-connected, nonlinearity, regularization, and so on. Over the...

Bibliography

1. Challenges in Representation Learning: A report on three machine learning contests, I Goodfellow, D Erhan, PL Carrier, A Courville, M Mirza, B Hamner, W Cukierski, Y Tang, DH Lee, Y Zhou, C Ramaiah, F Feng, R Li, X Wang, D Athanasakis, J Shawe-Taylor, M Milakov, J Park, R Ionescu, M Popescu, C Grozea, J Bergstra, J Xie, L Romaszko, B Xu, Z Chuang, and Y. Bengio. arXiv 2013. Site of the competition http://deeplearning.net/icml2013-workshop-competition.
2. Affective Computing, Rosalind Picard. MIT Technical Report #32, 1995 http://affect.media.mit.edu/pdfs/95.picard.pdf.
3. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Sergey Ioffe, Christian Szegedy, 2015.