Object Detection in Real Time

In the preceding chapter, we learned about Optical Character Recognition (OCR) technology. We recognized text in scanned documents and photos with the help of the Tesseract library and a pretrained deep learning model (the EAST model), which is loaded with OpenCV. In this chapter, we will move on to the topic of object detection. We will discuss several approaches to object detection provided by OpenCV and other libraries and frameworks.

The following topics will be covered in this chapter:

• Training and using cascade classifiers to detect objects
• Object detection using deep learning models

Technical requirements

As with the previous chapters, readers are required to have Qt of at least version 5 and OpenCV 4.0.0 installed. Having some basic knowledge about C++ and Qt programming is also a basic requirement.

Though we are focusing on OpenCV 4.0.0, OpenCV 3.4.x is also required in this chapter. You should have multiple versions (4.0.0 and 3.4.5) of OpenCV installed to follow along with this chapter. I will explain why later.

Since we will use deep learning models to detect objects, having knowledge of deep learning will also be a big help in understanding the contents of this chapter.

All the code for this chapter can be found in our code repository at https://github.com/PacktPublishing/Qt-5-and-OpenCV-4-Computer-Vision-Projects/tree/master/Chapter-06.

Check out the following video to see the code in action: http://bit.ly/2Fjx5SS

...

Detecting objects using OpenCV

There are many approaches to object detection in OpenCV. These approaches can be categorized as follows:

• Color-based algorithms such as meanshift and Continuously Adaptive Meanshift (CAMshift)
• Template matching
• Feature extracting and matching
• Artificial Neural Networks (ANNs)
• Cascade classifier
• Pretrained deep learning models

The first three are the traditional approaches to object detection, while the last three are approaches of machine learning.

The color-based algorithms, such as meanshift and CAMshift, use histograms and back-projection images to locate an object in an image with incredible speed. The template matching approach uses the object of interest as a template and tries to find the object by scanning the image of a given scene. Feature extracting and matching approaches first extract all features, usually edge features and corner...

Detecting objects using a cascade classifier

First, let's see how we can use a cascade classifier to detect objects. Actually, we have already used cascade classifiers in this book. In Chapter 4, Fun with Faces, we used a pretrained cascade classifier to detect faces in real time. The pretrained cascade classifier we used was one of the OpenCV built-in cascade classifiers and can be found in the data directory of the OpenCV installation:

$ ls ~/programs/opencv/share/opencv4/haarcascades/
haarcascade_eye_tree_eyeglasses.xml haarcascade_lefteye_2splits.xml
haarcascade_eye.xml haarcascade_licence_plate_rus_16stages.xml
haarcascade_frontalcatface_extended.xml haarcascade_lowerbody.xml
haarcascade_frontalcatface.xml haarcascade_profileface.xml
haarcascade_frontalface_alt2.xml haarcascade_righteye_2splits.xml
haarcascade_frontalface_alt_tree.xml haarcascade_russian_plate_number.xml
haarcascade_frontalface_alt...

Training a cascade classifier

OpenCV provided several tools to train cascade classifiers, but they were removed from version 4.0.0. That removal was mainly because of the rise of the deep learning approach, as we mentioned. The deep learning approach became the modern one, while the others, including cascade classifiers, became legacy. But many cascade classifiers are still in use in the world, and in many situations, they are still a good choice. These tools may be added back in some day. You can find and participate in the discussion about this topic at https://github.com/opencv/opencv/issues/13231.

Fortunately, we can use OpenCV v3.4.x, which provides these tools to train the cascade classifiers. The resulting cascade classifier files trained by v3.4 are compatible with v4.0.x. In other words, we can train cascade classifiers with OpenCV v3.4.x and use them with OpenCV v4.0...

Detecting objects using deep learning models

In the preceding section, we learned how to train and use cascade classifiers to detect objects. But that approach, compared to the increasingly expanding deep learning approach, provides worse performance, both in terms of the recall rate and accuracy. The OpenCV library has started to move on to the deep learning approach already. In version 3.x, it introduced the Deep Neural Network (DNN) module, and now in the latest version, v4.x, we can load many formats of neural network architecture, along with the pretrained weights for them. Also, as we mentioned, the tools for training cascade classifiers are deprecated in the latest version.

In this section, we will move on to the deep learning approach to see how to use OpenCV to detect objects the deep learning way. We used this approach once already. In Chapter 5, Optical Character Recognition...

About real time

When we handle videos, either video files or real-time video feeds from cameras, we know that the frame rate of the videos is about 24-30 FPS in general. That means we have 33-40 milliseconds to process each frame. If we take more time than that, we will lose some frames from a real-time video feed, or get a slower playing speed from a video file.

Now, let's add some code to our application to measure how much time we spend on each frame while detecting objects. First, in the Detective.pro project file, we add a new macro definition:

DEFINES += TIME_MEASURE=1

We will use this macro to switch the time-measuring code on or off. If you want to turn off the time measuring, just comment this line out, then rebuild the application by running the make clean && make command.

Then, in the CaptureThread::run method, in the capture_thread.cpp file, we add some...

Summary

In this chapter, we created a new application named Detective to detect objects using different approaches. First, we used an OpenCV built-in cascade classifier to detect the faces of cats. Then we learned how to train cascade classifiers by ourselves. We trained a cascade classifier for a rigid object (a no-entry traffic sign) and a cascade classifier for a less rigid object (the faces of Boston Bulls), then tested this with our application.

We moved on to the deep learning approach. We talked about the increasingly expanding deep learning technology, introduced many frameworks, and learned about the different ways in which a DNN model may detect objects using two-stage detectors and one-stage detectors. We combined the DNN module of the OpenCV library and the pretrained YOLOv3 model to detect objects in our application.

At the end, we talked about real time and the performance...

Questions

Try these questions to test your knowledge from this chapter:

1. When we trained the cascade classifier for the faces of the Boston Bulls, we annotated the dog faces on each image by ourselves. The annotation process cost us much time. There is a tarball of annotation data for that dataset at this website: http://vision.stanford.edu/aditya86/ImageNetDogs/annotation.tar. Could we generate the info.txt file from this annotation data via a piece of code? How would we do that?
2. Try to find a pretrained (fast/faster) R-CNN model and a pretrained SSD model. Run them and compare their performance to YOLOv3.
3. Could we use YOLOv3 to detect a certain kind of object, but not all the 80 classes of objects?