Home Data OpenCV Computer Vision Application Programming Cookbook

OpenCV Computer Vision Application Programming Cookbook

By Robert Laganiere
books-svg-icon Book
Subscription FREE
eBook + Subscription $29.99
eBook $28.99
Print + eBook $48.99
READ FOR FREE Free Trial for 7 days. $15.99 p/m after trial. Cancel Anytime! BUY NOW
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
READ FOR FREE Free Trial for 7 days. $15.99 p/m after trial. Cancel Anytime! BUY NOW
Subscription FREE
eBook + Subscription $29.99
eBook $28.99
Print + eBook $48.99
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
  1. Free Chapter
    Playing with Images
About this book
Publication date:
August 2014
Publisher
Packt
Pages
374
ISBN
9781782161486

 

Chapter 1. Playing with Images

In this chapter, we will get you started with the OpenCV library. You will learn how to perform the following tasks:

  • Installing the OpenCV library

  • Loading, displaying, and saving images

  • Exploring the cv::Mat data structure

  • Defining regions of interest

 

Introduction


This chapter will teach you the basic elements of OpenCV and will show you how to accomplish the most fundamental image processing tasks: reading, displaying, and saving images. However, before you can start with OpenCV, you need to install the library. This is a simple process that is explained in the first recipe of this chapter.

All your computer vision applications will involve the processing of images. This is why the most fundamental tool that OpenCV offers you is a data structure to handle images and matrices. It is a powerful data structure, with many useful attributes and methods. It also incorporates an advanced memory management model that greatly facilitates the development of applications. The last two recipes of this chapter will teach you how to use this important data structure of OpenCV.

 

Installing the OpenCV library


OpenCV is an open source library for developing computer vision applications that run on Windows, Linux, Android, and Mac OS. It can be used in both academic and commercial applications under a BSD license that allows you to freely use, distribute, and adapt it. This recipe will show you how to install the library on your machine.

Getting ready

When you visit the OpenCV official website at http://opencv.org/, you will find the latest release of the library, the online documentation, and many other useful resources on OpenCV.

How to do it...

From the OpenCV website, go to the DOWNLOADS page that corresponds to the platform of your choice (Unix/Windows or Android). From there, you will be able to download the OpenCV package. You will then need to uncompress it, normally under a directory with a name that corresponds to the library version (for example, in Windows, you can save the uncompressed directory under C:\OpenCV2.4.9). Once this is done, you will find a collection of files and directories that constitute the library at the chosen location. Notably, you will find the sources directory here, which contains all the source files. (Yes, it is open source!) However, in order to complete the installation of the library and have it ready for use, you need to undertake an additional step: generating the binary files of the library for the environment of your choice. This is indeed the point where you have to make a decision on the target platform that you will use to create your OpenCV applications. Which operating system should you use? Windows or Linux? Which compiler should you use? Microsoft VS2013 or MinGW? 32-bit or 64-bit? The Integrated Development Environment (IDE) that you will use in your project development will also guide you to make these choices.

Note that if you are working under Windows with Visual Studio, the executable installation package will, most probably, not only install the library sources, but also install all of the precompiled binaries needed to build your applications. Check for the build directory; it should contain the x64 and x86 subdirectories (corresponding to the 64-bit and 32-bit versions). Within these subdirectories, you should find directories such as vc10, vc11, and vc12; these contain the binaries for the different versions of MS Visual Studio. In that case, you are ready to start using OpenCV. Therefore, you can skip the compilation step described in this recipe, unless you want a customized build with specific options.

To complete the installation process and build the OpenCV binaries, you need to use the CMake tool, available at http://cmake.org. CMake is another open source software tool designed to control the compilation process of a software system using platform-independent configuration files. It generates the required makefiles or workspaces needed for compiling a software library in your environment. Therefore, you need to download and install CMake. You can then run it using the command line, but it is easier to use CMake with its GUI (cmake-gui). In the latter case, all you need to do is specify the folder containing the OpenCV library source and the one that will contain the binaries. You need to click on Configure in order to select the compiler of your choice and then click on Configure again.

You are now ready to generate your project files by clicking on the Generate button. These files will allow you to compile the library. This is the last step of the installation process, which will make the library ready to be used under your development environment. For example, if you have selected Visual Studio, then all you need to do is to open the top-level solution file that CMake has created for you (most probably, the OpenCV.sln file). You then issue the Build Solution command in Visual Studio. To get both a Release and a Debug build, you will have to repeat the compilation process twice, one for each configuration. The bin directory that is created contains the dynamic library files that your executable will call at runtime. Make sure to set your system PATH environment variable from the control panel such that your operating system can find the dll files when you run your applications.

In Linux environments, you will use the generated makefiles by running your make utility command. To complete the installation of all the directories, you also have to run a Build INSTALL or sudo make INSTALL command.

However, before you build the libraries, make sure to check what the OpenCV installer has installed for you; the built library that you are looking for might already be there, which will save you the compilation step. If you wish to use Qt as your IDE, the There's more... section of this recipe describes an alternative way to compile the OpenCV project.

How it works...

Since Version 2.2, the OpenCV library is divided into several modules. These modules are built-in library files located in the lib directory. Some of the commonly-used modules are as follows:

  • The opencv_core module that contains the core functionalities of the library, in particular, basic data structures and arithmetic functions

  • The opencv_imgproc module that contains the main image processing functions

  • The opencv_highgui module that contains the image and video reading and writing functions along with some user interface functions

  • The opencv_features2d module that contains the feature point detectors and descriptors and the feature point matching framework

  • The opencv_calib3d module that contains the camera calibration, two-view geometry estimation, and stereo functions

  • The opencv_video module that contains the motion estimation, feature tracking, and foreground extraction functions and classes

  • The opencv_objdetect module that contains the object detection functions such as the face and people detectors

The library also includes other utility modules that contain machine learning functions (opencv_ml), computational geometry algorithms (opencv_flann), contributed code (opencv_contrib), obsolete code (opencv_legacy), and gpu-accelerated code (opencv_gpu). You will also find other specialized libraries that implement higher-level functions, such as opencv_photo for computational photography and opencv_stitching for image-stitching algorithms. There is also a library module, called opencv_nonfree, which contains functions that have a potential limitation in use. When you compile your application, you will have to link your program with the libraries that contain the OpenCV functions you are using. Most likely, these will be the first three functions of the list given previously plus some of the others depending on the scope of your application.

All these modules have a header file associated with them (located in the include directory). A typical OpenCV C++ code will, therefore, start by including the required modules. For example (and this is the suggested declaration style):

#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

You might see an OpenCV code starting with the following command:

#include "cv.h"

This is because it uses the old style, before the library was restructured into modules. Finally, note that OpenCV will be restructured in the future; so, if you download a more recent version than 2.4, you will probably not see the same module subdivision.

There's more...

The OpenCV website at http://opencv.org/ contains detailed instructions on how to install the library. It also contains a complete online documentation that includes several tutorials on the different components of the library.

Using Qt for OpenCV developments

Qt is a cross-platform IDE for C++ applications developed as an open source project. It is offered under the LPGL open source license as well as under a commercial (and paid) license for the development of proprietary projects. It is composed of two separate elements: a cross-platform IDE called Qt creator and a set of Qt class libraries and development tools. Using Qt to develop C++ applications has the following benefits:

  • It is an open source initiative developed by the Qt community, which gives you access to the source code of the different Qt components

  • It is a cross-platform IDE, meaning that you can develop applications that can run on different operating systems, such as Windows, Linux, Mac OS X, and so on

  • It includes a complete and cross-platform GUI library that follows an effective object-oriented and event-driven model

  • Qt also includes several cross-platform libraries that help you to develop multimedia, graphics, databases, multithreading, web applications, and many other interesting building blocks useful for designing advanced applications

You can download Qt from http://qt-project.org/. When you install it, you will be offered the choice of different compilers. Under Windows, MinGW is an excellent alternative to the Visual Studio compilers.

Compiling the OpenCV library with Qt is particularly easy because it can read CMake files. Once OpenCV and CMake have been installed, simply select Open File or Project... from the Qt menu and open the CMakeLists.txt file that you will find under the sources directory of OpenCV. This will create an OpenCV project that you build using the Build Project Qt command.

You might get a few warnings, but these are without consequences.

The OpenCV developer site

OpenCV is an open source project that welcomes user contributions. You can access the developer site at http://code.opencv.org. Among other things, you can access the currently developed version of OpenCV. The community uses Git as their version control system. You then have to use it to check out the latest version of OpenCV. Git is also a free and open source software system; it is probably the best tool you can use to manage your own source code. You can download it from http://git-scm.com/.

See also

  • My website (www.laganiere.name) also presents step-by-step instructions on how to install the latest versions of the OpenCV library

  • The There's more... section of the next recipe explains how to create an OpenCV project with Qt

 

Loading, displaying, and saving images


It is now time to run your first OpenCV application. Since OpenCV is about processing images, this task will show you how to perform the most fundamental operations needed in the development of imaging applications. These are loading an input image from a file, displaying an image on a window, applying a processing function, and storing an output image on a disk.

Getting ready

Using your favorite IDE (for example, MS Visual Studio or Qt), create a new console application with a main function that is ready to be filled.

How to do it...

The first thing to do is to include the header files, declaring the classes and functions you will use. Here, we simply want to display an image, so we need the core library that declares the image data structure and the highgui header file that contains all the graphical interface functions:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

Our main function starts by declaring a variable that will hold the image. Under OpenCV 2, define an object of the cv::Mat class:

cv::Mat image; // create an empty image

This definition creates an image of the size 0 x 0. This can be confirmed by accessing the cv::Mat size attributes:

std::cout << "This image is " << image.rows << " x " 
          << image.cols << std::endl;

Next, a simple call to the reading function will read an image from the file, decode it, and allocate the memory:

image=  cv::imread("puppy.bmp"); // read an input image

You are now ready to use this image. However, you should first check whether the image has been correctly read (an error will occur if the file is not found, if the file is corrupted, or if it is not in a recognizable format). The validity of the image is tested using the following code:

if (image.empty()) {  // error handling
   // no image has been created…
   // possibly display an error message
   // and quit the application 
   …
}

The empty method returns true if no image data has been allocated.

The first thing you might want to do with this image is to display it. You can do this using the functions of the highgui module. Start by declaring the window on which you want to display the images, and then specify the image to be shown on this special window:

// define the window (optional)
cv::namedWindow("Original Image");
// show the image 
cv::imshow("Original Image", image);

As you can see, the window is identified by a name. You can reuse this window to display another image later, or you can create multiple windows with different names. When you run this application, you will see an image window as follows:

Now, you would normally apply some processing to the image. OpenCV offers a wide selection of processing functions, and several of them are explored in this book. Let's start with a very simple one that flips an image horizontally. Several image transformations in OpenCV can be performed in-place, meaning that the transformation is applied directly on the input image (no new image is created). This is the case of the flipping method. However, we can always create another matrix to hold the output result, and that is what we will do:

cv::Mat result; // we create another empty image
cv::flip(image,result,1); // positive for horizontal
                          // 0 for vertical,                     
                          // negative for both

The result is displayed on another window:

cv::namedWindow("Output Image"); // the output window
cv::imshow("Output Image", result);

Since it is a console window that will terminate when it reaches the end of the main function, we add an extra highgui function to wait for a user key before ending the program:

cv::waitKey(0); // 0 to indefinitely wait for a key pressed
                // specifying a positive value will wait for
                // the given amount of msec

You can then see that the output image is displayed on a distinct window, as shown in the following screenshot:

Finally, you will probably want to save the processed image on your disk. This is done using the following highgui function:

cv::imwrite("output.bmp", result); // save result

The file extension determines which codec will be used to save the image. Other popular supported image formats are JPG, TIFF, and PNG.

How it works...

All classes and functions in the C++ API of OpenCV are defined within the cv namespace. You have two ways to access them. First, precede the main function's definition with the following declaration:

using namespace cv;

Alternatively, prefix all OpenCV class and function names with the namespace specification, that is, cv::, as we will do so in this book. The use of this prefix makes the OpenCV classes and functions easier to identify.

The highgui module contains a set of functions that allow you to easily visualize and interact with your images. When you load an image with the imread function, you also have the option to read it as a gray-level image. This is very advantageous since several computer vision algorithms require gray-level images. Converting an input color image on the fly as you read it will save you time and minimize your memory usage. This can be done as follows:

// read the input image as a gray-scale image
image=  cv::imread("puppy.bmp", CV_LOAD_IMAGE_GRAYSCALE);

This will produce an image made of unsigned bytes (unsigned char in C++) that OpenCV designates with the CV_8U defined constant. Alternatively, it is sometimes necessary to read an image as a 3-channel color image even if it has been saved as a gray-level image. This can be achieved by calling the imread function with a positive second argument:

// read the input image as a 3-channel color image
image=  cv::imread("puppy.bmp", CV_LOAD_IMAGE_COLOR);

This time, an image made of 3 bytes per pixel will be created, designated as CV_8UC3 in OpenCV. Of course, if your input image has been saved as a gray-level image, all three channels will contain the same value. Finally, if you wish to read the image in the format in which it has been saved, then simply input a negative value as the second argument. The number of channels in an image can be checked by using the channels method:

std::cout << "This image has " 
          << image.channels() << " channel(s)";

Pay attention when you open an image with imread without specifying a full path (as we did here). In that case, the default directory will be used. When you run your application from the console, this directory is obviously the one of your executable file. However, if you run the application directly from your IDE, the default directory will most often be the one that contains your project file. Consequently, make sure that your input image file is located in the right directory.

When you use imshow to display an image made up of integers (designated as CV_16U for 16-bit unsigned integers, or as CV_32S for 32-bit signed integers), the pixel values of this image will be divided by 256 first, in an attempt to make it displayable with 256 gray shades. Similarly, an image made of floating points will be displayed by assuming a range of possible values between 0.0 (displayed as black) and 1.0 (displayed as white). Values outside this defined range are displayed in white (for values above 1.0) or black (for values below 1.0).

The highgui module is very useful to build quick prototypal applications. When you are ready to produce a finalized version of your application, you will probably want to use the GUI module offered by your IDE in order to build an application with a more professional look.

Here, our application uses both input and output images. As an exercise, you should rewrite this simple program such that it takes advantage of the function's in-place processing, that is, by not declaring the output image and writing it instead:

cv::flip(image,image,1); // in-place processing

There's more...

The highgui module contains a rich set of functions that help you to interact with your images. Using these, your applications can react to mouse or key events. You can also draw shapes and write text on images.

Clicking on images

You can program your mouse to perform specific operations when it is over one of the image windows you created. This is done by defining an appropriate callback function. A callback function is a function that you do not explicitly call but which is called by your application in response to specific events (here, the events that concern the mouse interacting with an image window). To be recognized by applications, callback functions need to have a specific signature and must be registered. In the case of the mouse event handler, the callback function must have the following signature:

void onMouse( int event, int x, int y, int flags, void* param);

The first parameter is an integer that is used to specify which type of mouse event has triggered the call to the callback function. The other two parameters are simply the pixel coordinates of the mouse location when the event occurred. The flags are used to determine which button was pressed when the mouse event was triggered. Finally, the last parameter is used to send an extra parameter to the function in the form of a pointer to any object. This callback function can be registered in the application through the following call:

cv::setMouseCallback("Original Image", onMouse, 
                     reinterpret_cast<void*>(&image));

In this example, the onMouse function is associated with the image window called Original Image, and the address of the displayed image is passed as an extra parameter to the function. Now, if we define the onMouse callback function as shown in the following code, then each time the mouse is clicked, the value of the corresponding pixel will be displayed on the console (here, we assume that it is a gray-level image):

void onMouse( int event, int x, int y, int flags, void* param)  {

  cv::Mat *im= reinterpret_cast<cv::Mat*>(param);

  switch (event) {  // dispatch the event

    case CV_EVENT_LBUTTONDOWN: // left mouse button down event

      // display pixel value at (x,y)
      std::cout << "at (" << x << "," << y << ") value is: " 
        << static_cast<int>(
                  im->at<uchar>(cv::Point(x,y))) << std::endl;
      break;
  }
}

Note that in order to obtain the pixel value at (x,y), we used the at method of the cv::Mat object here; this has been discussed in Chapter 2, Manipulating Pixels. Other possible events that can be received by the mouse event callback function include CV_EVENT_MOUSEMOVE, CV_EVENT_LBUTTONUP, CV_EVENT_RBUTTONDOWN, and CV_EVENT_RBUTTONUP.

Drawing on images

OpenCV also offers a few functions to draw shapes and write text on images. The examples of basic shape-drawing functions are circle, ellipse, line, and rectangle. The following is an example of how to use the circle function:

cv::circle(image,             // destination image 
        cv::Point(155,110),   // center coordinate
        65,                   // radius  
        0,                    // color (here black)
        3);                   // thickness

The cv::Point structure is often used in OpenCV methods and functions to specify a pixel coordinate. Note that here we assume that the drawing is done on a gray-level image; this is why the color is specified with a single integer. In the next recipe, you will learn how to specify a color value in the case of color images that use the cv::Scalar structure. It is also possible to write text on an image. This can be done as follows:

cv::putText(image,                  // destination image
        "This is a dog.",           // text
        cv::Point(40,200),          // text position
        cv::FONT_HERSHEY_PLAIN,     // font type
        2.0,                        // font scale
        255,                        // text color (here white)
        2);                         // text thickness

Calling these two functions on our test image will then result in the following screenshot:

Running the example with Qt

If you wish to use Qt to run your OpenCV applications, you will need to create project files. For the example of this recipe, here is how the project file (loadDisplaySave.pro) will look:

QT       += core
QT       -= gui

TARGET = loadDisplaySave
CONFIG   += console
CONFIG   -= app_bundle

TEMPLATE = app

SOURCES += loadDisplaySave.cpp
INCLUDEPATH += C:\OpenCV2.4.9\build\include
LIBS += -LC:\OpenCV2.4.9\build\x86\MinGWqt32\lib \
-lopencv_core249 \
-lopencv_imgproc249 \
-lopencv_highgui249

This file shows you where to find the include and library files. It also lists the library modules that are used by the example. Make sure to use the library binaries compatible with the compiler that Qt is using. Note that if you download the source code of the examples of this book, you will find the CMakeLists files that you can open with Qt (or CMake) in order to create the associated projects.

See also

  • The cv::Mat class is the data structure that is used to hold your images (and obviously, other matrix data). This data structure is at the core of all OpenCV classes and functions; the next recipe offers a detailed explanation of this data structure.

  • You can download the source code of the examples of this book from https://github.com/laganiere/.

 

Exploring the cv::Mat data structure


In the previous recipe, you were introduced to the cv::Mat data structure. As mentioned, this is a key element of the library. It is used to manipulate images and matrices (in fact, an image is a matrix from a computational and mathematical point of view). Since you will be using this data structure extensively in your application developments, it is imperative that you become familiar with it. Notably, you will learn in this recipe that this data structure incorporates an elegant memory management mechanism, allowing efficient usage.

How to do it...

Let's write the following test program that will allow us to test the different properties of the cv::Mat data structure:

#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

// test function that creates an image
cv::Mat function() {
   // create image
   cv::Mat ima(500,500,CV_8U,50);
   // return it
   return ima;
}

int main() {
  // define image windows
  cv::namedWindow("Image 1"); 
  cv::namedWindow("Image 2"); 
  cv::namedWindow("Image 3"); 
  cv::namedWindow("Image 4"); 
  cv::namedWindow("Image 5"); 
  cv::namedWindow("Image"); 

  // create a new image made of 240 rows and 320 columns
  cv::Mat image1(240,320,CV_8U,100);

  cv::imshow("Image", image1); // show the image
  cv::waitKey(0); // wait for a key pressed

  // re-allocate a new image
  image1.create(200,200,CV_8U);
  image1= 200;

  cv::imshow("Image", image1); // show the image
  cv::waitKey(0); // wait for a key pressed

  // create a red color image
  // channel order is BGR
  cv::Mat image2(240,320,CV_8UC3,cv::Scalar(0,0,255));

  // or:
  // cv::Mat image2(cv::Size(320,240),CV_8UC3);
  // image2= cv::Scalar(0,0,255);

  cv::imshow("Image", image2); // show the image
  cv::waitKey(0); // wait for a key pressed

  // read an image
  cv::Mat image3=  cv::imread("puppy.bmp"); 

  // all these images point to the same data block
  cv::Mat image4(image3);
  image1= image3;

  // these images are new copies of the source image
  image3.copyTo(image2);
  cv::Mat image5= image3.clone();

  // transform the image for testing
  cv::flip(image3,image3,1); 

  // check which images have been affected by the processing
  cv::imshow("Image 3", image3); 
  cv::imshow("Image 1", image1); 
  cv::imshow("Image 2", image2); 
  cv::imshow("Image 4", image4); 
  cv::imshow("Image 5", image5); 
  cv::waitKey(0); // wait for a key pressed


  // get a gray-level image from a function
  cv::Mat gray= function();

  cv::imshow("Image", gray); // show the image
  cv::waitKey(0); // wait for a key pressed

  // read the image in gray scale
  image1= cv::imread("puppy.bmp", CV_LOAD_IMAGE_GRAYSCALE); 
  image1.convertTo(image2,CV_32F,1/255.0,0.0);

  cv::imshow("Image", image2); // show the image
  cv::waitKey(0); // wait for a key pressed

  return 0;
}

Run this program and take a look at the following images produced:

How it works...

The cv::Mat data structure is essentially made up of two parts: a header and a data block. The header contains all the information associated with the matrix (size, number of channels, data type, and so on). The previous recipe showed you how to access some of the attributes of this structure contained in its header (for example, by using cols, rows, or channels). The data block holds all the pixel values of an image. The header contains a pointer variable that points to this data block; it is the data attribute. An important property of the cv::Mat data structure is the fact that the memory block is only copied when explicitly requested for. Indeed, most operations will simply copy the cv::Mat header such that multiple objects will point to the same data block at the same time. This memory management model makes your applications more efficient while avoiding memory leaks, but its consequences have to be understood. The examples of this recipe illustrate this fact.

By default, the cv::Mat objects have a zero size when they are created, but you can also specify an initial size as follows:

// create a new image made of 240 rows and 320 columns
cv::Mat image1(240,320,CV_8U,100);

In this case, you also need to specify the type of each matrix element; CV_8U here, which corresponds to 1-byte pixel images. The letter U means it is unsigned. You can also declare signed numbers by using the letter S. For a color image, you would specify three channels (CV_8UC3). You can also declare integers (signed or unsigned) of size 16 and 32 (for example, CV_16SC3). You also have access to 32-bit and 64-bit floating-point numbers (for example, CV_32F).

Each element of an image (or a matrix) can be composed of more than one value (for example, the three channels of a color image); therefore, OpenCV has introduced a simple data structure that is used when pixel values are passed to functions. It is the cv::Scalar structure, which is generally used to hold one value or three values. For example, to create a color image initialized with red pixels, you will write the following code:

// create a red color image
// channel order is BGR
cv::Mat image2(240,320,CV_8UC3,cv::Scalar(0,0,255));

Similarly, the initialization of the gray-level image could also have been done using this structure by writing cv::Scalar(100).

The image size also often needs to be passed to functions. We have already mentioned that the cols and rows attributes can be used to get the dimensions of a cv::Mat instance. The size information can also be provided through the cv::Size structure that simply contains the height and width of the matrix. The size() method allows you to obtain the current matrix size. This is the format that is used in many methods where a matrix size must be specified.

For example, an image could be created as follows:

// create a non-initialized color image 
cv::Mat image2(cv::Size(320,240),CV_8UC3);

The data block of an image can always be allocated or re-allocated using the create method. When an image has been previously allocated, its old content is de-allocated first. For reasons of efficiency, if the new proposed size and type matches the already existing size and type, then no new memory allocation is performed:

// re-allocate a new image
// (only if size or type are different)
image1.create(200,200,CV_8U);

When no more references point to a given cv::Mat object, the allocated memory is automatically released. This is very convenient because it avoids the common memory leak problems often associated with dynamic memory allocation in C++. This is a key mechanism in OpenCV 2 that is accomplished by having the cv::Mat class implement reference counting and shallow copy. Therefore, when an image is assigned to another one, the image data (that is, the pixels) is not copied; both the images will point to the same memory block. This also applies to images passed by value or returned by value. A reference count is kept such that the memory will be released only when all the references to the image will be destructed or assigned to another image:

// all these images point to the same data block
cv::Mat image4(image3);
image1= image3;

Any transformation applied to one of the preceding images will also affect the other images. If you wish to create a deep copy of the content of an image, use the copyTo method. In that case, the create method is called on the destination image. Another method that produces a copy of an image is the clone method, which creates a new identical image as follows:

// these images are new copies of the source image
image3.copyTo(image2);
cv::Mat image5= image3.clone();

If you need to copy an image into another image that does not necessarily have the same data type, you have to use the convertTo method:

// convert the image into a floating point image [0,1]
image1.convertTo(image2,CV_32F,1/255.0,0.0);

In this example, the source image is copied into a floating-point image. The method includes two optional parameters: a scaling factor and an offset. Note that both the images must, however, have the same number of channels.

The allocation model for the cv::Mat objects also allows you to safely write functions (or class methods) that return an image:

cv::Mat function() {

   // create image
   cv::Mat ima(240,320,CV_8U,cv::Scalar(100));
   // return it
   return ima;
}

We can also call this function from our main function as follows:

   // get a gray-level image
   cv::Mat gray= function();

If we do this, then the gray variable will now hold the image created by the function without extra memory allocation. Indeed, as we explained, only a shallow copy of the image will be transferred from the returned cv::Mat instance to the gray image. When the ima local variable goes out of scope, this variable is de-allocated, but since the associated reference counter indicates that its internal image data is being referred to by another instance (that is, the gray variable), its memory block is not released.

It's worth noting that in the case of classes, you should be careful and not return image class attributes. Here is an example of an error-prone implementation:

class Test {
   // image attribute
   cv::Mat ima;
  public:
     // constructor creating a gray-level image
     Test() : ima(240,320,CV_8U,cv::Scalar(100)) {}

     // method return a class attribute, not a good idea...
     cv::Mat method() { return ima; }
};

Here, if a function calls the method of this class, it obtains a shallow copy of the image attributes. If later this copy is modified, the class attribute will also be surreptitiously modified, which can affect the subsequent behavior of the class (and vice versa). To avoid these kinds of errors, you should instead return a clone of the attribute.

There's more...

When you are manipulating the cv::Mat class, you will discover that OpenCV also includes several other related classes. It will be important for you to become familiar with them.

The input and output arrays

If you look at the OpenCV documentation, you will see that many methods and functions accept parameters of the cv::InputArray type as the input. This type is a simple proxy class introduced to generalize the concept of arrays in OpenCV, and thus avoid the duplication of several versions of the same method or function with different input parameter types. It basically means that you can supply a cv::Mat object or other compatible types as an argument. This class is just an interface, so you should never declare it explicitly in your code. It is interesting to know that cv::InputArray can also be constructed from the popular std::vector class. This means that such objects can be used as the input to OpenCV methods and functions (as long as it makes sense to do so). Other compatible types are the cv::Scalar and the cv::Vec; this later structure will be presented in the next chapter. There is also a cv::OutputArray proxy class that is used to designate the arrays returned by some methods or functions.

The old IplImage structure

With Version 2 of OpenCV, a new C++ interface has been introduced. Previously, C-like functions and structures were used (and can still be used). In particular, images were manipulated using the IplImage structure. This structure was inherited from the IPL library (that is, the Intel Image Processing library), now integrated with the IPP library (the Intel Integrated Performance Primitive library). If you use the code and libraries that have been created with the old C interface, you might need to manipulate those IplImage structures. Fortunately, there is a convenient way to convert an IplImage structure into a cv::Mat object, which is shown in the following code:

IplImage* iplImage = cvLoadImage("puppy.bmp");
cv::Mat image(iplImage,false);

The cvLoadImage function is the C-interface function to load images. The second parameter in the constructor of the cv::Mat object indicates that the data will not be copied (set this to true if you want a new copy; false is the default value, so it could have been omitted), that is, both IplImage and image will share the same image data. Here, you need to be careful to not create dangling pointers. For this reason, it is safer to encapsulate the IplImage pointer in the reference-counting pointer class provided by OpenCV 2:

cv::Ptr<IplImage> iplImage = cvLoadImage("puppy.bmp");

Otherwise, if you need to de-allocate the memory pointed out by your IplImage structure, you need to do it explicitly:

cvReleaseImage(&iplImage);

Remember that you should avoid using this deprecated data structure. Instead, always use the cv::Mat data structure.

See also

  • The complete OpenCV documentation can be found at http://docs.opencv.org/

  • Chapter 2, Manipulating Pixels, will show you how to efficiently access and modify the pixel values of an image represented by the cv::Mat class

  • The next recipe, which will explain how to define a region of interest inside an image

 

Defining regions of interest


Sometimes, a processing function needs to be applied only to a portion of an image. OpenCV incorporates an elegant and simple mechanism to define a subregion in an image and manipulate it as a regular image. This recipe will teach you how to define a region of interest inside an image.

Getting ready

Suppose we want to copy a small image onto a larger one. For example, let's say we want to insert the following small logo in our test image:

To do this, a Region Of Interest (ROI) can be defined over which the copy operation can be applied. As we will see, the position of the ROI will determine where the logo will be inserted in the image.

How to do it...

The first step consists of defining the ROI. Once defined, the ROI can be manipulated as a regular cv::Mat instance. The key is that the ROI is indeed a cv::Mat object that points to the same data buffer as its parent image and has a header that specifies the coordinates of the ROI. Inserting the logo would then be accomplished as follows:

  // define image ROI at image bottom-right
  cv::Mat imageROI(image, 
              cv::Rect(image.cols-logo.cols, //ROI coordinates
                       image.rows-logo.rows,
                       logo.cols,logo.rows));// ROI size

  // insert logo
  logo.copyTo(imageROI);

Here, image is the destination image, and logo is the logo image (of a smaller size). The following image is then obtained by executing the previous code:

How it works...

One way to define an ROI is to use a cv::Rect instance. As the name indicates, it describes a rectangular region by specifying the position of the upper-left corner (the first two parameters of the constructor) and the size of the rectangle (the width and height are given in the last two parameters). In our example, we used the size of the image and the size of the logo in order to determine the position where the logo would cover the bottom-right corner of the image. Obviously, the ROI should always be completely inside the parent image.

The ROI can also be described using row and column ranges. A range is a continuous sequence from a start index to an end index (excluding both). The cv::Range structure is used to represent this concept. Therefore, an ROI can be defined from two ranges; in our example, the ROI could have been equivalently defined as follows:

imageROI= image(cv::Range(image.rows-logo.rows,image.rows), 
                cv::Range(image.cols-logo.cols,image.cols));

In this case, the operator() function of cv ::Mat returns another cv::Mat instance that can then be used in subsequent calls. Any transformation of the ROI will affect the original image in the corresponding area because the image and the ROI share the same image data. Since the definition of an ROI does not include the copying of data, it is executed in a constant amount of time, no matter the size of the ROI.

If you want to define an ROI made of some lines of an image, the following call can be used:

cv::Mat imageROI= image.rowRange(start,end);

Similarly, for an ROI made of some image columns, the following can be used:

cv::Mat imageROI= image.colRange(start,end);

There's more...

The OpenCV methods and functions include many optional parameters that are not discussed in the recipes of this book. When you wish to use a function for the first time, you should always take the time to look at the documentation to learn more about the possible options that this function offers. One very common option is the possibility to define image masks.

Using image masks

Some OpenCV operations allow you to define a mask that will limit the applicability of a given function or method, which is normally supposed to operate on all the image pixels. A mask is an 8-bit image that should be nonzero at all locations where you want an operation to be applied. At the pixel locations that correspond to the zero values of the mask, the image is untouched. For example, the copyTo method can be called with a mask. We can use it here to copy only the white portion of the logo shown previously, as follows:

// define image ROI at image bottom-right
imageROI= image(cv::Rect(image.cols-logo.cols,
                         image.rows-logo.rows,
                       logo.cols,logo.rows));
// use the logo as a mask (must be gray-level)
cv::Mat mask(logo);

// insert by copying only at locations of non-zero mask
logo.copyTo(imageROI,mask);

The following image is obtained by executing the previous code:

The background of our logo was black (therefore, it had the value 0); therefore, it was easy to use it as both the copied image and the mask. Of course, you can define the mask of your choice in your application; most OpenCV pixel-based operations give you the opportunity to use masks.

See also

  • The row and col methods that will be used in the Scanning an image with neighbor access recipe of Chapter 2, Manipulating Pixels. These are a special case of the rowRange and colRange methods in which the start and end indexes are equal in order to define a single-line or single-column ROI.

About the Author
  • Robert Laganiere

    Robert Laganiere is a professor at the School of Electrical Engineering and Computer Science of the University of Ottawa, Canada. He is also a faculty member of the VIVA research lab and is the co-author of several scientific publications and patents in content based video analysis, visual surveillance, driver-assistance, object detection, and tracking. Robert authored the OpenCV2 Computer Vision Application Programming Cookbook in 2011 and co-authored Object Oriented Software Development published by McGraw Hill in 2001. He co-founded Visual Cortek in 2006, an Ottawa-based video analytics start-up that was later acquired by iwatchlife.com in 2009. He is also a consultant in computer vision and has assumed the role of Chief Scientist in a number of start-up companies such as Cognivue Corp, iWatchlife, and Tempo Analytics. Robert has a Bachelor of Electrical Engineering degree from Ecole Polytechnique in Montreal (1987) and MSc and PhD degrees from INRS-Telecommunications, Montreal (1996). You can visit the author’s website at laganiere.name.

    Browse publications by this author
Latest Reviews (5 reviews total)
OpenCV Computer Vision Application Programming Cookbook
Unlock this book and the full library FREE for 7 days
Start now