Introduction to Neural Networks with OpenCV

This chapter introduces a family of machine learning models called artificial neural networks (ANNs), or sometimes just neural networks. A key characteristic of these models is that they attempt to learn relationships among variables in a multi-layered fashion; they learn multiple functions to predict intermediate results before combining these into a single function to predict something meaningful (such as the class of an object). Recent versions of OpenCV contain an increasing amount of functionality related to ANNs – and, in particular, ANNs with many layers, called deep neural networks (DNNs). We will experiment with both shallower ANNs and DNNs in this chapter.

We have already gained some exposure to machine learning in other chapters – especially in Chapter 7, Building Custom Object Detectors, where we developed a...

Technical requirements

This chapter uses Python, OpenCV, and NumPy. Please refer to Chapter 1, Setting Up OpenCV, for installation instructions.

The completed code and sample videos for this chapter can be found in this book's GitHub repository, https://github.com/PacktPublishing/Learning-OpenCV-4-Computer-Vision-with-Python-Third-Edition, in the chapter10 folder.

Understanding ANNs

Let's define ANNs in terms of their basic role and components. Although much of the literature on ANNs emphasizes the idea that they are biologically inspired by the way neurons connect in a brain, we don't need to be biologists or neuroscientists to understand the fundamental concepts of an ANN.

First of all, an ANN is a statistical model. What is a statistical model? A statistical model is a pair of elements, namely the space S (a set of observations) and the probability, P, where P is a distribution that approximates S (in other words, a function that would generate a set of observations that is very similar to S).

Here are two different ways to think of P:

• P is a simplification of a complex scenario.
• P is the function that generated S in the first place, or at the very least a set of observations very similar to S.

Thus, ANNs are models that...

Training a basic ANN in OpenCV

OpenCV provides a class, cv2.ml_ANN_MLP, that implements an ANN as a multi-layer perceptron (MLP). This is exactly the kind of model we described earlier, in the Understanding neurons and perceptrons section.

To create an instance of cv2.ml_ANN_MLP, and to format data for this ANN's training and use, we rely on functionality in OpenCV's machine learning module, cv2.ml. As you may recall, this is the same module that we used for SVM-related functionality in Chapter 7, Building Custom Object Detectors. Moreover, cv2.ml_ANN_MLP and cv2.ml_SVM share a common base class called cv2.ml_StatModel. Therefore, you will find that OpenCV provides similar APIs for ANNs and SVMs.

Let's examine a dummy example as a gentle introduction to ANNs. This example will use completely meaningless data, but it will show us the basic API for training and...

Training an ANN classifier in multiple epochs

Let's create an ANN that attempts to classify animals based on three measurements: weight, length, and number of teeth. This is, of course, a mock scenario. Realistically, no one would describe an animal with just these three statistics. However, our intent is to improve our understanding of ANNs before we start applying them to image data.

Compared to the minimal example in the previous section, our animal classification mock-up will be more sophisticated in the following ways:

• We will increase the number of neurons in the hidden layer.
• We will use a larger training dataset. For convenience, we will generate this dataset pseudorandomly.
• We will train the ANN in multiple epochs, meaning that we will train and retrain it multiple times with the same dataset each time.

The number of neurons in the hidden layer is an important...

Recognizing handwritten digits with an ANN

A handwritten digit is any of the 10 Arabic numerals (0 to 9), written manually with a pen or pencil, as opposed to being printed by a machine. The appearance of handwritten digits can vary significantly. Different people have different handwriting, and – with the possible exception of a skilled calligrapher – a person does not produce identical digits every time he or she writes. This variability means that the visual recognition of handwritten digits is a non-trivial problem for machine learning. Indeed, students and researchers in machine learning often test their skills and new algorithms by attempting to train an accurate recognizer for handwritten digits. We will approach this challenge in the following manner:

1. Load data from a Python-friendly version of the MNIST database. This is a widely used database containing...

Using DNNs from other frameworks in OpenCV

OpenCV can load and use DNNs that have been trained in any of the following frameworks:

• Caffe (http://caffe.berkeleyvision.org/)
• TensorFlow (https://www.tensorflow.org/)
• Torch (http://torch.ch/)
• Darknet (https://pjreddie.com/darknet/)
• ONNX (https://onnx.ai/)
• DLDT (https://github.com/opencv/dldt/)

The Deep Learning Deployment Toolkit (DLDT) is part of Intel's OpenVINO Toolkit (https://software.intel.com/openvino-toolkit/) for computer vision. DLDT provides tools for optimizing DNNs from other frameworks and for converting them into a common format. A collection of DLDT-compatible models is freely available in a repository called the Open Model Zoo (https://github.com/opencv/open_model_zoo/). DLDT, the Open Model Zoo, and OpenCV have some of the same people on their development teams; all three of these projects are sponsored by...

Detecting and classifying objects with third-party DNNs

For this demo, we are going to capture frames from a webcam in real-time and use a DNN to detect and classify 20 kinds of objects that may be in any given frame. Yes, a single DNN can do all this in real-time on a typical laptop that a programmer might use!

Before delving into the code, let's introduce the DNN that we will use. It is a Caffe version of a model called MobileNet-SSD, which uses a hybrid of a framework from Google called MobileNet and another framework called Single Shot Detector (SSD) MultiBox. The latter framework has a GitHub repository at https://github.com/weiliu89/caffe/tree/ssd/. The training technique for the Caffe version of MobileNet-SSD is provided by a project on GitHub at https://github.com/chuanqi305/MobileNet-SSD/. Copies of the following MobileNet-SSD files can be found in this book&apos...

Detecting and classifying faces with third-party DNNs

For this demonstration, we are going to use one DNN to detect faces and two other DNNs to classify the age and gender of each detected face. Specifically, we will use pre-trained Caffe models that are stored in the following files in the chapter10/faces_data folder of this book's GitHub repository.

Here is an inventory of the files in this folder, and of the files' origins:

• detection/res10_300x300_ssd_iter_140000.caffemodel: This is the DNN for face detection. The OpenCV team has provided this file at https://github.com/opencv/opencv_3rdparty/blob/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel. This Caffe model was trained with the SSD framework (https://github.com/weiliu89/caffe/tree/ssd/). Thus, its topology is similar to the MobileNet-SSD model that we used in the previous section...

Summary

This chapter scratched the surface of the vast and fascinating world of ANNs. We learned about the structure of ANNs, and how to design a network topology based on application requirements. Then, we focused on OpenCV's implementation of MLP ANNs, as well as on OpenCV's support for diverse DNNs that have been trained in other frameworks.

We applied neural networks to real-world problems: notably, handwritten digit recognition; object detection and classification; and a combination of face detection, age classification, and gender classification in real time. We saw that even in these introductory demos, neural networks show a lot of promise in terms of versatility, accuracy, and speed. Hopefully, this encourages you to try out pre-trained models from various authors, and to learn to train advanced models of your own in various frameworks.

With this thought,...