Chapter 7. Learning to Recognize Emotions on Faces

We previously familiarized ourselves with the concepts of object detection and object recognition, but we never combined them to develop an app that can do both end-to-end. For the final chapter in this book, we will do exactly that.

The goal of this chapter is to develop an app that combines both face detection and face recognition, with a focus on recognizing emotional expressions in the detected face.

For this, we will touch upon two other classic algorithms that come bundled with OpenCV: Haar Cascade Classifiers and multi-layer peceptrons (MLPs). While the former can be used to rapidly detect (or locate, answering the question: where?) objects of various sizes and orientations in an image, the latter can be used to recognize them (or identify, answering the question: what?).

The end goal of the app will be to detect your own face in each captured frame of a webcam live stream and label your emotional expression. To make this task feasible...

Planning the app

The final app will consist of a main script that integrates the process flow end-to-end, from face detection to facial expression recognition, as well as some utility functions to help along the way.

Thus, the end product will require several components:

• chapter7: The main script and entry-point for the chapter.

• chapter7.FaceLayout: A custom layout based on gui.BaseLayout that operates in two different modes:

  • Training mode: In the training mode, the app will collect image frames, detect a face therein, assign a label depending on the facial expression, and upon exiting, save all the collected data samples in a file, so that it can be parsed by datasets.homebrew.

  • Testing mode: In the testing mode, the app will detect a face in each video frame and predict the corresponding class label by using a pre-trained MLP.

• chapter3.main: The main function routine to start the GUI application.

• detectors.FaceDetector: A class for face detection.

  • detect: A method to detect faces in a grayscale...

Face detection

OpenCV comes preinstalled with a range of sophisticated classifiers for general-purpose object detection. Perhaps, the most commonly known detector is the cascade of Haar-based feature detectors for face detection, which was invented by Paul Viola and Michael Jones.

Haar-based cascade classifiers

Every book on OpenCV should at least mention the Viola–Jones face detector. Invented in 2001, this cascade classifier disrupted the field of computer vision, as it finally allowed real-time face detection and face recognition.

The classifier is based on Haar-like features (similar to Haar basis functions), which sum up the pixel intensities in small regions of an image, as well as capture the difference between adjacent image regions. Some example rectangle features are shown in the following figure, relative to the enclosing (light gray) detection window:

Here, the top row shows two examples of an edge feature, either vertically oriented (left) or oriented at a 45 degree angle (right...

Facial expression recognition

The facial expression recognition pipeline is encapsulated by chapter7.py. This file consists of an interactive GUI that operates in two modes (training and testing), as described earlier.

In order to arrive at our end-to-end app, we need to cover the following three steps:

1. Load the chapter7.py GUI in the training mode to assemble a training set.

2. Train an MLP classifier on the training set via train_test_mlp.py. Because this step can take a long time, the process takes place in its own script. After successful training, store the trained weights in a file, so that we can load the pre-trained MLP in the next step.

3. Load the chapter7.py GUI in the testing mode to classify facial expressions on a live video stream in real-time. This step involves loading several pre-trained cascade classifiers as well as our pre-trained MLP classifier. These classifiers will then be applied to every captured video frame.

Assembling a training set

Before we can train an MLP, we need to...

Putting it all together

In order to run our app, we will need to execute the main function routine (in chapter7.py) that loads the pre-trained cascade classifier and the pre-trained multi-layer perceptron, and applies them to each frame of the webcam live stream.

However, this time, instead of collecting more training samples, we will select the radio button that says Test. This will trigger an EVT_RADIOBUTTON event, which binds to FaceLayout._on_testing, disabling all training-related buttons in the GUI and switching the app to the testing mode. In this mode, the pre-trained MLP classifier is applied to every frame of the live stream, trying to predict the current facial expression.

As promised earlier, we now return to FaceLayout._process_frame:

def _process_frame(self, frame):
    """ detects face, predicts face label in testing mode """

Unchanged from what we discussed earlier, the method begins by detecting faces in a downscaled grayscale version of the current frame:

    success, frame...

Summary

The final chapter of this book has really rounded up our experience and made us combine a variety of our skills to arrive at an end-to-end app that consists of both object detection and object recognition. We became familiar with a range of pre-trained cascade classifiers that OpenCV has to offer, collected our very own training dataset, learned about multi-layer perceptrons, and trained them to recognize emotional expressions in faces. Well, at least my face.

The classifier was undoubtedly able to benefit from the fact that I was the only subject in the dataset, but with all the knowledge and experience that you have gathered with this book, it is now time to overcome these limitations! You can start small and train the classifier on images of you indoors and outdoors, at night and day, during summer and winter. Or, maybe, you are anxious to tackle a real-world dataset and be part of Kaggle's Facial Expression Recognition challenge (see https://www.kaggle.com/c/challenges-in-representation...