OpenCV Computer Vision Application Programming Cookbook Second Edition

When you visit the OpenCV official website at http://opencv.org/, you will find the latest release of the library, the online documentation, and many other useful resources on OpenCV.

From the OpenCV website, go to the DOWNLOADS page that corresponds to the platform of your choice (Unix/Windows or Android). From there, you will be able to download the OpenCV package. You will then need to uncompress it, normally under a directory with a name that corresponds to the library version (for example, in Windows, you can save the uncompressed directory under C:\OpenCV2.4.9). Once this is done, you will find a collection of files and directories that constitute the library at the chosen location. Notably, you will find the sources directory here, which contains all the source files. (Yes, it is open source!) However, in order to complete the installation of the library and have it ready for use, you need to undertake an additional step: generating the binary files of the library for the environment of your choice. This is indeed the point where you have to make a decision on the target platform that you will use to create your OpenCV applications. Which operating system should you use? Windows or Linux? Which compiler should you use? Microsoft VS2013 or MinGW? 32-bit or 64-bit? The Integrated Development Environment (IDE) that you will use in your project development will also guide you to make these choices.

Note that if you are working under Windows with Visual Studio, the executable installation package will, most probably, not only install the library sources, but also install all of the precompiled binaries needed to build your applications. Check for the build directory; it should contain the x64 and x86 subdirectories (corresponding to the 64-bit and 32-bit versions). Within these subdirectories, you should find directories such as vc10, vc11, and vc12; these contain the binaries for the different versions of MS Visual Studio. In that case, you are ready to start using OpenCV. Therefore, you can skip the compilation step described in this recipe, unless you want a customized build with specific options.

To complete the installation process and build the OpenCV binaries, you need to use the CMake tool, available at http://cmake.org. CMake is another open source software tool designed to control the compilation process of a software system using platform-independent configuration files. It generates the required makefiles or workspaces needed for compiling a software library in your environment. Therefore, you need to download and install CMake. You can then run it using the command line, but it is easier to use CMake with its GUI (cmake-gui). In the latter case, all you need to do is specify the folder containing the OpenCV library source and the one that will contain the binaries. You need to click on Configure in order to select the compiler of your choice and then click on Configure again.

You are now ready to generate your project files by clicking on the Generate button. These files will allow you to compile the library. This is the last step of the installation process, which will make the library ready to be used under your development environment. For example, if you have selected Visual Studio, then all you need to do is to open the top-level solution file that CMake has created for you (most probably, the OpenCV.sln file). You then issue the Build Solution command in Visual Studio. To get both a Release and a Debug build, you will have to repeat the compilation process twice, one for each configuration. The bin directory that is created contains the dynamic library files that your executable will call at runtime. Make sure to set your system PATH environment variable from the control panel such that your operating system can find the dll files when you run your applications.

In Linux environments, you will use the generated makefiles by running your make utility command. To complete the installation of all the directories, you also have to run a Build INSTALL or sudo make INSTALL command.

However, before you build the libraries, make sure to check what the OpenCV installer has installed for you; the built library that you are looking for might already be there, which will save you the compilation step. If you wish to use Qt as your IDE, the There's more... section of this recipe describes an alternative way to compile the OpenCV project.

Since Version 2.2, the OpenCV library is divided into several modules. These modules are built-in library files located in the lib directory. Some of the commonly-used modules are as follows:

The library also includes other utility modules that contain machine learning functions (opencv_ml), computational geometry algorithms (opencv_flann), contributed code (opencv_contrib), obsolete code (opencv_legacy), and gpu-accelerated code (opencv_gpu). You will also find other specialized libraries that implement higher-level functions, such as opencv_photo for computational photography and opencv_stitching for image-stitching algorithms. There is also a library module, called opencv_nonfree, which contains functions that have a potential limitation in use. When you compile your application, you will have to link your program with the libraries that contain the OpenCV functions you are using. Most likely, these will be the first three functions of the list given previously plus some of the others depending on the scope of your application.

All these modules have a header file associated with them (located in the include directory). A typical OpenCV C++ code will, therefore, start by including the required modules. For example (and this is the suggested declaration style):

#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>

You might see an OpenCV code starting with the following command:

#include "cv.h"

This is because it uses the old style, before the library was restructured into modules. Finally, note that OpenCV will be restructured in the future; so, if you download a more recent version than 2.4, you will probably not see the same module subdivision.

The OpenCV website at http://opencv.org/ contains detailed instructions on how to install the library. It also contains a complete online documentation that includes several tutorials on the different components of the library.

Using Qt for OpenCV developments

Qt is a cross-platform IDE for C++ applications developed as an open source project. It is offered under the LPGL open source license as well as under a commercial (and paid) license for the development of proprietary projects. It is composed of two separate elements: a cross-platform IDE called Qt creator and a set of Qt class libraries and development tools. Using Qt to develop C++ applications has the following benefits:

• It is an open source initiative developed by the Qt community, which gives you access to the source code of the different Qt components

• It is a cross-platform IDE, meaning that you can develop applications that can run on different operating systems, such as Windows, Linux, Mac OS X, and so on

• It includes a complete and cross-platform GUI library that follows an effective object-oriented and event-driven model

• Qt also includes several cross-platform libraries that help you to develop multimedia, graphics, databases, multithreading, web applications, and many other interesting building blocks useful for designing advanced applications

You can download Qt from http://qt-project.org/. When you install it, you will be offered the choice of different compilers. Under Windows, MinGW is an excellent alternative to the Visual Studio compilers.

Compiling the OpenCV library with Qt is particularly easy because it can read CMake files. Once OpenCV and CMake have been installed, simply select Open File or Project... from the Qt menu and open the CMakeLists.txt file that you will find under the sources directory of OpenCV. This will create an OpenCV project that you build using the Build Project Qt command.

You might get a few warnings, but these are without consequences.

Using your favorite IDE (for example, MS Visual Studio or Qt), create a new console application with a main function that is ready to be filled.

The first thing to do is to include the header files, declaring the classes and functions you will use. Here, we simply want to display an image, so we need the core library that declares the image data structure and the highgui header file that contains all the graphical interface functions:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

Our main function starts by declaring a variable that will hold the image. Under OpenCV 2, define an object of the cv::Mat class:

cv::Mat image; // create an empty image

This definition creates an image of the size 0 x 0. This can be confirmed by accessing the cv::Mat size attributes:

std::cout << "This image is " << image.rows << " x " 
          << image.cols << std::endl;

Next, a simple call to the reading function will read an image from the file, decode it, and allocate the memory:

image=  cv::imread("puppy.bmp"); // read an input image

You are now ready to use this image. However, you should first check whether the image has been correctly read (an error will occur if the file is not found, if the file is corrupted, or if it is not in a recognizable format). The validity of the image is tested using the following code:

if (image.empty()) {  // error handling
   // no image has been created…
   // possibly display an error message
   // and quit the application 
   …
}

The empty method returns true if no image data has been allocated.

The first thing you might want to do with this image is to display it. You can do this using the functions of the highgui module. Start by declaring the window on which you want to display the images, and then specify the image to be shown on this special window:

// define the window (optional)
cv::namedWindow("Original Image");
// show the image 
cv::imshow("Original Image", image);

As you can see, the window is identified by a name. You can reuse this window to display another image later, or you can create multiple windows with different names. When you run this application, you will see an image window as follows:

Now, you would normally apply some processing to the image. OpenCV offers a wide selection of processing functions, and several of them are explored in this book. Let's start with a very simple one that flips an image horizontally. Several image transformations in OpenCV can be performed in-place, meaning that the transformation is applied directly on the input image (no new image is created). This is the case of the flipping method. However, we can always create another matrix to hold the output result, and that is what we will do:

cv::Mat result; // we create another empty image
cv::flip(image,result,1); // positive for horizontal
                          // 0 for vertical,                     
                          // negative for both

The result is displayed on another window:

cv::namedWindow("Output Image"); // the output window
cv::imshow("Output Image", result);

Since it is a console window that will terminate when it reaches the end of the main function, we add an extra highgui function to wait for a user key before ending the program:

cv::waitKey(0); // 0 to indefinitely wait for a key pressed
                // specifying a positive value will wait for
                // the given amount of msec

You can then see that the output image is displayed on a distinct window, as shown in the following screenshot:

Finally, you will probably want to save the processed image on your disk. This is done using the following highgui function:

cv::imwrite("output.bmp", result); // save result

The file extension determines which codec will be used to save the image. Other popular supported image formats are JPG, TIFF, and PNG.

All classes and functions in the C++ API of OpenCV are defined within the cv namespace. You have two ways to access them. First, precede the main function's definition with the following declaration:

using namespace cv;

Alternatively, prefix all OpenCV class and function names with the namespace specification, that is, cv::, as we will do so in this book. The use of this prefix makes the OpenCV classes and functions easier to identify.

The highgui module contains a set of functions that allow you to easily visualize and interact with your images. When you load an image with the imread function, you also have the option to read it as a gray-level image. This is very advantageous since several computer vision algorithms require gray-level images. Converting an input color image on the fly as you read it will save you time and minimize your memory usage. This can be done as follows:

// read the input image as a gray-scale image
image=  cv::imread("puppy.bmp", CV_LOAD_IMAGE_GRAYSCALE);

This will produce an image made of unsigned bytes (unsigned char in C++) that OpenCV designates with the CV_8U defined constant. Alternatively, it is sometimes necessary to read an image as a 3-channel color image even if it has been saved as a gray-level image. This can be achieved by calling the imread function with a positive second argument:

// read the input image as a 3-channel color image
image=  cv::imread("puppy.bmp", CV_LOAD_IMAGE_COLOR);

This time, an image made of 3 bytes per pixel will be created, designated as CV_8UC3 in OpenCV. Of course, if your input image has been saved as a gray-level image, all three channels will contain the same value. Finally, if you wish to read the image in the format in which it has been saved, then simply input a negative value as the second argument. The number of channels in an image can be checked by using the channels method:

std::cout << "This image has " 
          << image.channels() << " channel(s)";

Pay attention when you open an image with imread without specifying a full path (as we did here). In that case, the default directory will be used. When you run your application from the console, this directory is obviously the one of your executable file. However, if you run the application directly from your IDE, the default directory will most often be the one that contains your project file. Consequently, make sure that your input image file is located in the right directory.

When you use imshow to display an image made up of integers (designated as CV_16U for 16-bit unsigned integers, or as CV_32S for 32-bit signed integers), the pixel values of this image will be divided by 256 first, in an attempt to make it displayable with 256 gray shades. Similarly, an image made of floating points will be displayed by assuming a range of possible values between 0.0 (displayed as black) and 1.0 (displayed as white). Values outside this defined range are displayed in white (for values above 1.0) or black (for values below 1.0).

The highgui module is very useful to build quick prototypal applications. When you are ready to produce a finalized version of your application, you will probably want to use the GUI module offered by your IDE in order to build an application with a more professional look.

Here, our application uses both input and output images. As an exercise, you should rewrite this simple program such that it takes advantage of the function's in-place processing, that is, by not declaring the output image and writing it instead:

cv::flip(image,image,1); // in-place processing

The highgui module contains a rich set of functions that help you to interact with your images. Using these, your applications can react to mouse or key events. You can also draw shapes and write text on images.

void onMouse( int event, int x, int y, int flags, void* param);

cv::setMouseCallback("Original Image", onMouse, 
                     reinterpret_cast<void*>(&image));

void onMouse( int event, int x, int y, int flags, void* param)  {

  cv::Mat *im= reinterpret_cast<cv::Mat*>(param);

  switch (event) {  // dispatch the event

    case CV_EVENT_LBUTTONDOWN: // left mouse button down event

      // display pixel value at (x,y)
      std::cout << "at (" << x << "," << y << ") value is: " 
        << static_cast<int>(
                  im->at<uchar>(cv::Point(x,y))) << std::endl;
      break;
  }
}

Drawing on images

OpenCV also offers a few functions to draw shapes and write text on images. The examples of basic shape-drawing functions are circle, ellipse, line, and rectangle. The following is an example of how to use the circle function:

cv::circle(image,             // destination image 
        cv::Point(155,110),   // center coordinate
        65,                   // radius  
        0,                    // color (here black)
        3);                   // thickness

The cv::Point structure is often used in OpenCV methods and functions to specify a pixel coordinate. Note that here we assume that the drawing is done on a gray-level image; this is why the color is specified with a single integer. In the next recipe, you will learn how to specify a color value in the case of color images that use the cv::Scalar structure. It is also possible to write text on an image. This can be done as follows:

cv::putText(image,                  // destination image
        "This is a dog.",           // text
        cv::Point(40,200),          // text position
        cv::FONT_HERSHEY_PLAIN,     // font type
        2.0,                        // font scale
        255,                        // text color (here white)
        2);                         // text thickness

Calling these two functions on our test image will then result in the following screenshot:

QT       += core
QT       -= gui

TARGET = loadDisplaySave
CONFIG   += console
CONFIG   -= app_bundle

TEMPLATE = app

SOURCES += loadDisplaySave.cpp
INCLUDEPATH += C:\OpenCV2.4.9\build\include
LIBS += -LC:\OpenCV2.4.9\build\x86\MinGWqt32\lib \
-lopencv_core249 \
-lopencv_imgproc249 \
-lopencv_highgui249

Let's write the following test program that will allow us to test the different properties of the cv::Mat data structure:

#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

// test function that creates an image
cv::Mat function() {
   // create image
   cv::Mat ima(500,500,CV_8U,50);
   // return it
   return ima;
}

int main() {
  // define image windows
  cv::namedWindow("Image 1"); 
  cv::namedWindow("Image 2"); 
  cv::namedWindow("Image 3"); 
  cv::namedWindow("Image 4"); 
  cv::namedWindow("Image 5"); 
  cv::namedWindow("Image"); 

  // create a new image made of 240 rows and 320 columns
  cv::Mat image1(240,320,CV_8U,100);

  cv::imshow("Image", image1); // show the image
  cv::waitKey(0); // wait for a key pressed

  // re-allocate a new image
  image1.create(200,200,CV_8U);
  image1= 200;

  cv::imshow("Image", image1); // show the image
  cv::waitKey(0); // wait for a key pressed

  // create a red color image
  // channel order is BGR
  cv::Mat image2(240,320,CV_8UC3,cv::Scalar(0,0,255));

  // or:
  // cv::Mat image2(cv::Size(320,240),CV_8UC3);
  // image2= cv::Scalar(0,0,255);

  cv::imshow("Image", image2); // show the image
  cv::waitKey(0); // wait for a key pressed

  // read an image
  cv::Mat image3=  cv::imread("puppy.bmp"); 

  // all these images point to the same data block
  cv::Mat image4(image3);
  image1= image3;

  // these images are new copies of the source image
  image3.copyTo(image2);
  cv::Mat image5= image3.clone();

  // transform the image for testing
  cv::flip(image3,image3,1); 

  // check which images have been affected by the processing
  cv::imshow("Image 3", image3); 
  cv::imshow("Image 1", image1); 
  cv::imshow("Image 2", image2); 
  cv::imshow("Image 4", image4); 
  cv::imshow("Image 5", image5); 
  cv::waitKey(0); // wait for a key pressed


  // get a gray-level image from a function
  cv::Mat gray= function();

  cv::imshow("Image", gray); // show the image
  cv::waitKey(0); // wait for a key pressed

  // read the image in gray scale
  image1= cv::imread("puppy.bmp", CV_LOAD_IMAGE_GRAYSCALE); 
  image1.convertTo(image2,CV_32F,1/255.0,0.0);

  cv::imshow("Image", image2); // show the image
  cv::waitKey(0); // wait for a key pressed

  return 0;
}

Run this program and take a look at the following images produced:

The cv::Mat data structure is essentially made up of two parts: a header and a data block. The header contains all the information associated with the matrix (size, number of channels, data type, and so on). The previous recipe showed you how to access some of the attributes of this structure contained in its header (for example, by using cols, rows, or channels). The data block holds all the pixel values of an image. The header contains a pointer variable that points to this data block; it is the data attribute. An important property of the cv::Mat data structure is the fact that the memory block is only copied when explicitly requested for. Indeed, most operations will simply copy the cv::Mat header such that multiple objects will point to the same data block at the same time. This memory management model makes your applications more efficient while avoiding memory leaks, but its consequences have to be understood. The examples of this recipe illustrate this fact.

By default, the cv::Mat objects have a zero size when they are created, but you can also specify an initial size as follows:

// create a new image made of 240 rows and 320 columns
cv::Mat image1(240,320,CV_8U,100);

In this case, you also need to specify the type of each matrix element; CV_8U here, which corresponds to 1-byte pixel images. The letter U means it is unsigned. You can also declare signed numbers by using the letter S. For a color image, you would specify three channels (CV_8UC3). You can also declare integers (signed or unsigned) of size 16 and 32 (for example, CV_16SC3). You also have access to 32-bit and 64-bit floating-point numbers (for example, CV_32F).

Each element of an image (or a matrix) can be composed of more than one value (for example, the three channels of a color image); therefore, OpenCV has introduced a simple data structure that is used when pixel values are passed to functions. It is the cv::Scalar structure, which is generally used to hold one value or three values. For example, to create a color image initialized with red pixels, you will write the following code:

// create a red color image
// channel order is BGR
cv::Mat image2(240,320,CV_8UC3,cv::Scalar(0,0,255));

Similarly, the initialization of the gray-level image could also have been done using this structure by writing cv::Scalar(100).

The image size also often needs to be passed to functions. We have already mentioned that the cols and rows attributes can be used to get the dimensions of a cv::Mat instance. The size information can also be provided through the cv::Size structure that simply contains the height and width of the matrix. The size() method allows you to obtain the current matrix size. This is the format that is used in many methods where a matrix size must be specified.

For example, an image could be created as follows:

// create a non-initialized color image 
cv::Mat image2(cv::Size(320,240),CV_8UC3);

The data block of an image can always be allocated or re-allocated using the create method. When an image has been previously allocated, its old content is de-allocated first. For reasons of efficiency, if the new proposed size and type matches the already existing size and type, then no new memory allocation is performed:

// re-allocate a new image
// (only if size or type are different)
image1.create(200,200,CV_8U);

When no more references point to a given cv::Mat object, the allocated memory is automatically released. This is very convenient because it avoids the common memory leak problems often associated with dynamic memory allocation in C++. This is a key mechanism in OpenCV 2 that is accomplished by having the cv::Mat class implement reference counting and shallow copy. Therefore, when an image is assigned to another one, the image data (that is, the pixels) is not copied; both the images will point to the same memory block. This also applies to images passed by value or returned by value. A reference count is kept such that the memory will be released only when all the references to the image will be destructed or assigned to another image:

// all these images point to the same data block
cv::Mat image4(image3);
image1= image3;

Any transformation applied to one of the preceding images will also affect the other images. If you wish to create a deep copy of the content of an image, use the copyTo method. In that case, the create method is called on the destination image. Another method that produces a copy of an image is the clone method, which creates a new identical image as follows:

// these images are new copies of the source image
image3.copyTo(image2);
cv::Mat image5= image3.clone();

If you need to copy an image into another image that does not necessarily have the same data type, you have to use the convertTo method:

// convert the image into a floating point image [0,1]
image1.convertTo(image2,CV_32F,1/255.0,0.0);

In this example, the source image is copied into a floating-point image. The method includes two optional parameters: a scaling factor and an offset. Note that both the images must, however, have the same number of channels.

The allocation model for the cv::Mat objects also allows you to safely write functions (or class methods) that return an image:

cv::Mat function() {

   // create image
   cv::Mat ima(240,320,CV_8U,cv::Scalar(100));
   // return it
   return ima;
}

We can also call this function from our main function as follows:

   // get a gray-level image
   cv::Mat gray= function();

If we do this, then the gray variable will now hold the image created by the function without extra memory allocation. Indeed, as we explained, only a shallow copy of the image will be transferred from the returned cv::Mat instance to the gray image. When the ima local variable goes out of scope, this variable is de-allocated, but since the associated reference counter indicates that its internal image data is being referred to by another instance (that is, the gray variable), its memory block is not released.

It's worth noting that in the case of classes, you should be careful and not return image class attributes. Here is an example of an error-prone implementation:

class Test {
   // image attribute
   cv::Mat ima;
  public:
     // constructor creating a gray-level image
     Test() : ima(240,320,CV_8U,cv::Scalar(100)) {}

     // method return a class attribute, not a good idea...
     cv::Mat method() { return ima; }
};

Here, if a function calls the method of this class, it obtains a shallow copy of the image attributes. If later this copy is modified, the class attribute will also be surreptitiously modified, which can affect the subsequent behavior of the class (and vice versa). To avoid these kinds of errors, you should instead return a clone of the attribute.

When you are manipulating the cv::Mat class, you will discover that OpenCV also includes several other related classes. It will be important for you to become familiar with them.

IplImage* iplImage = cvLoadImage("puppy.bmp");
cv::Mat image(iplImage,false);

cv::Ptr<IplImage> iplImage = cvLoadImage("puppy.bmp");

cvReleaseImage(&iplImage);

Suppose we want to copy a small image onto a larger one. For example, let's say we want to insert the following small logo in our test image:

To do this, a Region Of Interest (ROI) can be defined over which the copy operation can be applied. As we will see, the position of the ROI will determine where the logo will be inserted in the image.

The first step consists of defining the ROI. Once defined, the ROI can be manipulated as a regular cv::Mat instance. The key is that the ROI is indeed a cv::Mat object that points to the same data buffer as its parent image and has a header that specifies the coordinates of the ROI. Inserting the logo would then be accomplished as follows:

  // define image ROI at image bottom-right
  cv::Mat imageROI(image, 
              cv::Rect(image.cols-logo.cols, //ROI coordinates
                       image.rows-logo.rows,
                       logo.cols,logo.rows));// ROI size

  // insert logo
  logo.copyTo(imageROI);

Here, image is the destination image, and logo is the logo image (of a smaller size). The following image is then obtained by executing the previous code:

One way to define an ROI is to use a cv::Rect instance. As the name indicates, it describes a rectangular region by specifying the position of the upper-left corner (the first two parameters of the constructor) and the size of the rectangle (the width and height are given in the last two parameters). In our example, we used the size of the image and the size of the logo in order to determine the position where the logo would cover the bottom-right corner of the image. Obviously, the ROI should always be completely inside the parent image.

The ROI can also be described using row and column ranges. A range is a continuous sequence from a start index to an end index (excluding both). The cv::Range structure is used to represent this concept. Therefore, an ROI can be defined from two ranges; in our example, the ROI could have been equivalently defined as follows:

imageROI= image(cv::Range(image.rows-logo.rows,image.rows), 
                cv::Range(image.cols-logo.cols,image.cols));

In this case, the operator() function of cv ::Mat returns another cv::Mat instance that can then be used in subsequent calls. Any transformation of the ROI will affect the original image in the corresponding area because the image and the ROI share the same image data. Since the definition of an ROI does not include the copying of data, it is executed in a constant amount of time, no matter the size of the ROI.

If you want to define an ROI made of some lines of an image, the following call can be used:

cv::Mat imageROI= image.rowRange(start,end);

Similarly, for an ROI made of some image columns, the following can be used:

cv::Mat imageROI= image.colRange(start,end);

The OpenCV methods and functions include many optional parameters that are not discussed in the recipes of this book. When you wish to use a function for the first time, you should always take the time to look at the documentation to learn more about the possible options that this function offers. One very common option is the possibility to define image masks.

Using image masks

Some OpenCV operations allow you to define a mask that will limit the applicability of a given function or method, which is normally supposed to operate on all the image pixels. A mask is an 8-bit image that should be nonzero at all locations where you want an operation to be applied. At the pixel locations that correspond to the zero values of the mask, the image is untouched. For example, the copyTo method can be called with a mask. We can use it here to copy only the white portion of the logo shown previously, as follows:

// define image ROI at image bottom-right
imageROI= image(cv::Rect(image.cols-logo.cols,
                         image.rows-logo.rows,
                       logo.cols,logo.rows));
// use the logo as a mask (must be gray-level)
cv::Mat mask(logo);

// insert by copying only at locations of non-zero mask
logo.copyTo(imageROI,mask);

The following image is obtained by executing the previous code:

The background of our logo was black (therefore, it had the value 0); therefore, it was easy to use it as both the copied image and the mask. Of course, you can define the mask of your choice in your application; most OpenCV pixel-based operations give you the opportunity to use masks.