OpenCV By Example: Enhance your understanding of Computer Vision and image processing by developing real-world projects in OpenCV 3

We now understand how visual data enters the human visual system and how our system processes it. The issue is that we still don't completely understand how our brain recognizes and organizes this visual data. We just extract some features from images and ask the computer to learn from them using machine learning algorithms. We still have those variations such as shape, size, perspective, angle, illumination, occlusion, and so on. For example, the same chair looks very different to a machine when you look at it from the side view. Humans can easily recognize that it's a chair regardless of how it's presented to us. So, how do we explain this to our machines?

One way to do this would be to store all the different variations of an object, including sizes, angles, perspectives, and so on. But this process is cumbersome and time-consuming! Also, it's actually not possible to gather data that can encompass every single variation. The machines will consume a huge amount of memory and a lot of time to build a model that can recognize these objects. Even with all this, if an object is partially occluded, computers still won't be able to recognize it. This is because they think that this is a new object. So, when we build a Computer Vision library, we need to build the underlying functional blocks that can be combined in many different ways to formulate complex algorithms. OpenCV provides a lot of these functions and they are highly optimized. So, once we understand what OpenCV provides out of the box, we can use it effectively to build interesting applications. Let's go ahead and explore this in the next section.

One of the best things about OpenCV is that it provides a lot of in-built primitives to handle operations related to image processing and Computer Vision. If you have to write something from scratch, you will have to define things, such as an image, point, rectangle, and so on. These are fundamental to almost any Computer Vision algorithm. OpenCV comes with all these basic structures out of the box, and they are contained in the core module. Another advantage is that these structures have already been optimized for speed and memory, so you don't have to worry about the implementation details.

The imgcodecs module handles reading and writing image files. When you operate on an input image and create an output image, you can save it as a jpg or a png file with a simple command. You will be dealing with a lot of video files when you are working with cameras. The videoio module handles everything related to the input/output of video files. You can easily capture a video from a webcam or read a video file in many different formats. You can even save a bunch of frames as a video file by setting properties such as frames per second, frame size, and so on.

When you write a Computer Vision algorithm, there are a lot of basic image processing operations that you will use over and over again. Most of these functions are present in the imgproc module. You can do things such as image filtering, morphological operations, geometric transformations, color conversions, drawing on images, histograms, shape analysis, motion analysis, feature detection, and so on. Let's consider the following figure:

The right-hand side image is a rotated version of the left-hand side image. We can do this transformation with a single line in OpenCV. There is another module called ximgproc that contains advanced image processing algorithms such as structured forests for edge detection, domain transform filters, adaptive manifold filters, and so on.

OpenCV provides a module called highgui that handles all the high-level user interface operations. Let's say that you are working on a problem and you want to check what the image looks like before you proceed to the next step. This module has functions that can be used to create windows to display images and/or video. There is also a waiting function that will wait until you hit a key on your keyboard before it goes to the next step. There is a function that can detect mouse events as well. This is very useful to develop interactive applications. Using this functionality, you can draw rectangles on these input windows and then proceed based on the selected region.

Consider the following image:

As you can see, we have drawn a green rectangle on the image and applied a negative film effect to that region. Once we have the coordinates of this rectangle, we can operate only on that region.

Video analysis includes tasks such as analyzing the motion between successive frames in a video, tracking different objects in a video, creating models for video surveillance, and so on. OpenCV provides a module called video that can handle all of this. There is a module called videostab that deals with video stabilization. Video stabilization is an important part of video cameras. When you capture videos by holding the camera in your hands, it's hard to keep your hands perfectly steady. If you look at that video as it is, it will look bad and jittery. All modern devices use video stabilization techniques to process the videos before they are presented to the end user.

3D reconstruction is an important topic in Computer Vision. Given a set of 2D images, we can reconstruct the 3D scene using the relevant algorithms. OpenCV provides algorithms that can find the relationship between various objects in these 2D images to compute their 3D positions. We have a module called calib3d that can handle all this. This module can also handle camera calibration, which is essential to estimate the parameters of the camera. These parameters are basically the internal parameters of any given camera that uses them to transform the captured scene into an image. We need to know these parameters to design algorithms, or else we might get unexpected results. Let's consider the following figure:

As shown in the preceding image, the same object is captured from multiple poses. Our job is to reconstruct the original object using these 2D images.

As discussed earlier, the human visual system tends to extract the salient features from a given scene so that it can be retrieved later. To mimic this, people started designing various feature extractors that can extract these salient points from a given image. Some of the popular algorithms include SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), FAST (Features from Accelerated Segment Test), and so on. There is a module called features2d that provides functions to detect and extract all these features. There is another module called xfeatures2d that provides a few more feature extractors, some of which are still in the experimental phase. You can play around with these if you get a chance. There is also a module called bioinspired that provides algorithms for biologically inspired Computer Vision models.

Object detection refers to detecting the location of an object in a given image. This process is not concerned with the type of object. If you design a chair detector, it will just tell you the location of the chair in a given image. It will not tell you whether it's a red chair with a high back or a blue chair with a low back. Detecting the location of objects is a very critical step in many Computer Vision systems. Consider the following image:

If you run a chair detector on this image, it will put a green box around all the chairs. It won't tell you what kind of chair it is! Object detection used to be a computationally intensive task because of the number of calculations required to perform the detection at various scales. To solve this, Paul Viola and Michael Jones came up with a great algorithm in their seminal paper in 2001. You can read it at https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf. They provided a fast way to design an object detector for any object. OpenCV has modules called objdetect and xobjdetect that provide the framework to design an object detector. You can use it to develop detectors for random items such as sunglasses, boots, and so on.

Computer Vision uses various machine learning algorithms to achieve different things. OpenCV provides a module called ml that has many machine learning algorithms bundled into it. Some of the algorithms include Bayes Classifier, K-Nearest Neighbors, Support Vector Machines, Decision Trees, Neural Networks, and so on. It also has a module called flann that contains algorithms for fast-nearest-neighbor searches in large datasets. Machine learning algorithms are used extensively to build systems for object recognition, image classification, face detection, visual searches, and so on.

Computational photography refers to using advanced image processing techniques to improve the images captured by cameras. Instead of focusing on optical processes and image capture methods, computational photography uses software to manipulate visual data. Some applications include high dynamic range imaging, panoramic images, image relighting, light field cameras, and so on.

Let's take a look at the following image:

Look at those vivid colors! This is an example of a high dynamic range image and it wouldn't be possible to get this using conventional image capture techniques. To do this, we have to capture the same scene at multiple exposures, register those images with each other, and then blend them nicely to create this image. The photo and xphoto modules contain various algorithms that provide algorithms pertaining to computational photography. There is a module called stitching that provides algorithms to create panoramic images.

The notion of shape is crucial in Computer Vision. We analyze the visual data by recognizing various different shapes in the image. This is actually an important step in many algorithms. Let's say you are trying to identify a particular logo in an image. Now, you know that it can appear in various shapes, orientations, sizes, and so on. One good way to get started is to quantify the characteristics of the shape of the object. The module shape provides all the algorithms required to extract different shapes, measure similarities between them, transform shapes of objects, and so on.

Optical flow algorithms are used in videos to track features across successive frames. Let's say you want to track a particular object in a video. Running a feature extractor on each frame would be computationally expensive; hence, the process would be slow. So, you just need to extract the features from the current frame and then track these features in successive frames. Optical flow algorithms are heavily used in video-based applications in Computer Vision. The optflow module contains a number of algorithms required to perform optical flow. There is also a module called tracking that contains more algorithms that can be used to track features.

Face recognition refers to identifying the person in a given image. This is not the same as face detection where you identify the location of a face in the given image. So, if you want to build a practical biometric system that can recognize the person in front of the camera, you first need to run the face detector that can identify the location of the face, and then, run a face recognizer that can recognize who that person is. There is a module called face that deals with face recognition.

As discussed earlier, Computer Vision tries to model algorithms based on how humans perceive the visual data. So, it would be helpful to find salient regions and objects in the images that can help different applications, such as object recognition, object detection and tracking, and so on. There is a module called saliency that's designed for this purpose. It provides algorithms that can detect salient regions in static images and videos.

We are increasingly interacting with devices that can capture the 3D structure of the objects around us. These devices basically capture the depth information along with the regular 2D color images. So, it's important for us to build algorithms that can understand and process 3D objects. Kinect is a good example of a device that captures the depth information along with the visual data. The task at hand is to recognize the input 3D object by matching it with one of the models in our database. If we have a system that can recognize and locate objects, then it can be used for many different applications. There is a module called surface_matching that contains algorithms for 3D object recognition and a pose estimation algorithm using 3D features.

Identifying text in a given scene and recognizing the content is becoming increasingly important. Some applications include nameplate recognition, recognizing road signs for self-driving cars, book scanning to digitize the contents, and so on. There is a module called text that contains various algorithms to handle text detection and recognition.

To keep things easy, let's install OpenCV using prebuilt libraries. Let's go to http://opencv.org and download the latest version for Windows. The current version is 3.0.0, and you can go to the OpenCV homepage to get the latest link to download the package.

You need to make sure you have admin rights before you proceed. The downloaded file will be an executable file, so just double-click on it to start the installation process. The installer expands the content into a folder. You will be able to choose the installation path and check the installation by inspecting the files.

Once you are done with the previous step, we need to set the OpenCV environment variables and add it to the system path to complete the installation. We will set up an environment variable that will hold the build directory of the OpenCV library. We will be using this in our projects. Open the terminal and type the following command:

C:\> setx -m OPENCV_DIR D:\OpenCV\Build\x64\vc11

Let's go ahead and add the path to the bin folder of our system's path. The reason we need to do this is because we will be using the OpenCV library in the form of Dynamic Link Libraries (DLLs). Basically, all the OpenCV algorithms are stored here, and our operating system will only load them during runtime. In order to do this, our operating system needs to know where they are located. The system's PATH variable will contain a list of all the folders where it can find the DLLs. So, naturally, we need to add the path to the OpenCV library to this list. Now, why do we need to do all this? Well, the other option is to copy the required DLLs to the same folder as the application's executable file (the .exe file). This is an unnecessary overhead, especially when we are working with many different projects.

We need to edit the PATH variable in order to add it to this folder. You can use software such as Path Editor to do this. You can download it from https://patheditor2.codeplex.com. Once you install it, start it and add the following new entry (you can right-click on the path to insert a new item):

%OPENCV_DIR%\bin

Go ahead and save it to the registry. We are done!

In this section, we will see how to install OpenCV on Mac OS X. Precompiled binaries are not available for Mac OS X, so we need to compile OpenCV from scratch. Before we proceed, we need to install CMake. If you don't have CMake already installed, you can download it from https://cmake.org/files/v3.3/cmake-3.3.2-Darwin-x86_64.dmg. It's a dmg file! So, once you download it, just run the installer.

Download the latest version of OpenCV from opencv.org. The current version is 3.0.0, and you can download it from https://github.com/Itseez/opencv/archive/3.0.0.zip.

Unzip the contents into a folder of your choice. OpenCV 3.0.0 also has a new package called opencv_contrib that contains user contributions that are not yet considered stable. One thing to keep in mind is that some of the algorithms in the opencv_contrib package are not freely available for commercial use. Also, installing this package is optional. OpenCV will work just fine if you don't install opencv_contrib. Since we are installing OpenCV anyway, it's good to install this package so that you can experiment with it later on (as opposed to going through the whole installation process again). This package is a great way to learn and play around with new algorithms. You can download it from https://github.com/Itseez/opencv_contrib/archive/3.0.0.zip.

Unzip the contents of the ZIP file into a folder of your choice. For convenience, unzip it into the same folder, as mentioned earlier, so that the opencv-3.0.0 and opencv_contrib-3.0.0 folders are in the same main folder.

We are now ready to build OpenCV. Open your terminal and navigate to the folder where you unzipped the contents of OpenCV 3.0.0. Run the following commands after substituting the right paths in the commands:

$ cd /full/path/to/opencv-3.0.0/
$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/full/path/to/opencv-3.0.0/build -D INSTALL_C_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=/full/path/to/opencv_contrib-3.0.0/modules ../

It's time to install OpenCV 3.0.0. Go inside the /full/path/to/opencv-3.0.0/build directory and run the following commands on your terminal:

$ make -j4
$ make install

In the preceding command, the -j4 flag indicates that it is using four cores to install it. It's faster this way! Now, let's set the library path. Open your ~/.profile file in your terminal using the vi ~/.profile command, and add the following line:

export DYLD_LIBRARY_PATH=/full/path/to/opencv-3.0.0/build/lib:$DYLD_LIBRARY_PATH

We need to copy the pkg-config file opencv.pc to /usr/local/lib/pkgconfig and name it opencv3.pc. This way, if you already have an existing OpenCV 2.4.x installation, there will be no conflicts. Let's go ahead and do this:

$ cp /full/path/to/opencv-3.0.0/build/lib/pkgconfig/opencv.pc /usr/local/lib/pkgconfig/opencv3.pc

We need to update our PKG_CONFIG_PATH variable as well. Open your ~/.profile file, and add the following line:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/:$PKG_CONFIG_PATH

Reload your ~/.profile file using the following command:

$ source ~/.profile

We are done! Let's see if it's working:

$ cd /full/path/to/opencv-3.0.0/samples/cpp
$ g++ -ggdb 'pkg-config --cflags --libs opencv3' opencv_version.cpp -o /tmp/opencv_version && /tmp/opencv_version

If you see Welcome to OpenCV 3.0.0 printed on your terminal, you are good to go. We will be using CMake to build our OpenCV projects throughout this book. We will cover this in more detail in the next chapter.

Let's see how to install OpenCV on Ubuntu. We need to install some dependencies before we begin. Let's install them using the package manager by running the following command on your terminal:

$ sudo apt-get -y install libopencv-dev build-essential cmake libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff4-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev libtbb-dev libqt4-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils

Now that you have installed the dependencies, let's download, build, and install OpenCV:

$ wget "https://github.com/Itseez/opencv/archive/3.0.0.zip" -O opencv.zip
$ wget "https://github.com/Itseez/opencv_contrib/archive/3.0.0.zip" -O opencv_contrib.zip
$ unzip opencv.zip –d .
$ unzip opencv_contrib.zip –d .
$ cd opencv-3.0.0
$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/full/path/to/opencv-3.0.0/build -D INSTALL_C_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=/full/path/to/opencv_contrib-3.0.0/modules ../
$ make –j4
$ sudo make install

Let's copy the pkg-config file's opencv.pc to /usr/local/lib/pkgconfig and name it opencv3.pc:

$ cp /full/path/to/opencv-3.0.0/build/lib/pkgconfig/opencv.pc /usr/local/lib/pkgconfig/opencv3.pc

We are done! We will now be able to use it to compile our OpenCV programs from the command line. Also, if you already have an existing OpenCV 2.4.x installation, there will be no conflicts. Let's check whether the installation is working properly:

$ cd /full/path/to/opencv-3.0.0/samples/cpp
$ g++ -ggdb 'pkg-config --cflags --libs opencv3' opencv_version.cpp -o /tmp/opencv_version && /tmp/opencv_version

If you see Welcome to OpenCV 3.0.0 printed on your terminal, you are good to go. In the following chapters, you will learn how to use CMake to build your OpenCV projects.