OpenCV Essentials

OpenCV can be downloaded from http://opencv.org/, and is available for the most popular operating systems, such as Unix (Linux/Mac), Microsoft Windows (Windows), Android, and iOS. In this book, the last stable release (2.4.9) of OpenCV for Windows 7 (SP1) has been used. For Windows, this release comes in the form of a self-extracting archive (opencv-2.4.9.exe), which should be extracted to the desired location (for example, OPENCV_SCR for C:\opencv-src). It should be noted that in Windows it is strongly recommended to allocate the source and binaries at absolute paths without white spaces because errors might appear later.

After extracting the archive, the obtained files are organized in two subdirectories under OPENCV_SCR: build and sources. The first one (build) includes precompiled (binaries) versions with Microsoft Visual C++ compilers (MSVC, v. 10, 11, and 12) for 32- and 64-bit architectures (located in the x 86 and x 64 subdirectories respectively). The sources subdirectory contains the source code of the OpenCV library. This code might be compiled with other compilers (for example, GNU g++).

This chapter covers a detailed installation process of OpenCV on Windows 7 (SP1). For Linux and other operating systems, you can have a look at the OpenCV online documentation (OpenCV Tutorials, Introduction to OpenCV section) available at http://docs.opencv.org/doc/tutorials/tutorials.html.

Compiled versus precompiled library

The OpenCV distribution includes the source code of the library that can be compiled when a different binary version is required. One such situation comes when we need to use the Qt-based user interface functions available in OpenCV (which are not included in the precompiled versions). Besides, the build process (compilation) for the OpenCV library is required if our compiler (for example, GNU g++) doesn't match the precompiled version of the library.

The requirements that have to be met in order to compile OpenCV with Qt are as follows:

• A compatible C++ compiler: We use the GNU g++ compiler included with MinGW (Minimal GNU GCC for Windows). This is a standard compiler on Unix and it is appropriate to guarantee code compatibility. Prior to the build process, it is quite convenient to add the location of the compiler binaries (g++ and gmake) to the Path environment variable (for example, in our local system, the location is C:\Qt\Qt5.2.1\Tools\mingw48_32\bin).

• The Qt library: In particular, the Qt 5.2.1 bundle (available at http://qt-project.org/) is customized for an easy setup because it includes the Qt library and the complete development IDE Qt Creator with MinGW 4.8 and OpenGL. Qt Creator is a full-fledged IDE with free software license that we recommend. The Qt binaries location must also be added to the Path environment variable (for example, C:\Qt\Qt5.2.1\5.2.1\mingw48_32\bin).

• The CMake build system: This cross-platform build system is available at http://www.cmake.org/. It consists of a set of tools that help the user prepare and generate the suitable configuration files used for building (compiling), testing, and packaging a large code project such as OpenCV.

Configuring OpenCV with CMake

In this section, we illustrate the configuration steps for OpenCV with CMake, with the help of screenshots of the steps involved:

1. The first step involves the selection of directories and compilers. Once CMake is launched, both the source directory (OPENCV_SCR) and the build directory (OPENCV_BUILD) can be set in the proper text fields in the CMake main window. Also, the checkboxes labeled as Grouped and Advanced should be marked in the CMake main window. We continue clicking on the Configure button. At this point, the tool prompts the user to specify the desired compiler and we choose MinGW Makefiles using the native compilers. If we choose the Specify native compilers option, it is possible to specify a particular location for the compiler and make tools. After clicking on the Finish button, the configuration step continues checking the settings of the system. The following screenshot shows the CMake window at the end of this preconfiguration process:

   

   CMake at the end of the preconfiguration step

   Note

   For the purpose of simplicity, we use in this text OPENCV_BUILD and OPENCV_SCR to denote respectively the target and source directories of the OpenCV local setup. Keep in mind that all directories should match the current local configuration.

2. The next step is the selection of the build options. At the center of the main CMake window, the red entries might be changed if desired. In our setup, we open the entries grouped with the label WITH and there we set the WITH_QT entry to ON, and then we click on Configure again to obtain a new set of options.

3. Now, the next stage is to set the Qt directories. In the main CMake window, a few entries are marked in red. These are the required directories to build OpenCV with Qt. The next entries to be set are: Qt5Concurrent_DIR, Qt5Core_DIR, Qt5Gui_DIR, Qt5OpenGL_DIR, Qt5Test_DIR, and Qt5Widgets_DIR (refer to the following figure). In our setup, these directories can be found under C:/Qt/Qt5.2.1/5.2.1/mingw48_32/lib/cmake.

   By clicking on the Configure button once, we obtain no further red entries and the configuration process is finally done, as shown in the following screenshot:

   

   Setting Qt directories for CMake

4. The last step is to generate the project. In this step, we click on the Generate button to obtain the suitable project files to build OpenCV in the target platform. Then, the CMake GUI should be closed to continue with the compilation.

In the process just described, it is possible to change the configuration options as many times as desired before the generation step. Some other convenient options to be set are listed as follows:

• BUILD_EXAMPLES: This option is used to compile the source code of several examples included in the distribution

• BUILD_SHARED_LIBS: Uncheck this option to get a static version of the libraries

• CMAKE_BUILD_TYPE: Set this to Debug to get a version for debugging purposes and so on

• WITH_TBB: Set this option to activate the use of Intel® Threading Building Block that lets you easily write parallel C++ code

• WITH_CUDA: Set this option to use processing by GPU through CUDA libraries

Building and installing the library

The compilation should be launched from the console at the target directory (OPENCV_BUILD) set during the configuration with CMake (that is, step 1 from the previous list). The command should be as follows:

OPENCV_BUILD>mingw32-make

This command launches a build process using the generated files by CMake. Compilation typically takes several minutes. If the compilation ends without errors, the installation continues with the execution of the following command:

OPENCV_BUILD>mingw32-make install

This command copies the OpenCV binaries to the following directory:

C:\opencv-buildQt\install

If something goes wrong during the compilation, we should return to CMake to change the options selected in the previous steps. Installation ends by adding the location of the library binaries (DLL files) to the Path environment variable. In our setup, this directory is located at OPENCV_BUILD\install\x64\mingw\bin.

To check the success of the installation process, it is possible to run some of the examples compiled along with the library (if the BUILD_EXAMPLES option was set with CMake). The code samples can be found at OPENCV_BUILD\install\x64\mingw\samples\cpp.

Canny edge detection sample

The preceding screenshot shows the output window for the sample cpp-example-edge.exe file, which demonstrates the Canny edge detection on the fruits.jpg input file included with the source OpenCV distribution.

In the next section, we summarize the recipe used to set up OpenCV 2.4.9 in our Windows 7-x32 platform with Qt 5.2.1 (MinGW 4.8).

After installation, preparing a new OpenCV code project is quite a straightforward process that requires including the header files and instructing the compiler to find the files and libraries used in the project.

OpenCV is composed of several modules, grouping related functionalities. Each module has an associated header file (for example, core.hpp) located in the directory with the same name as that of the module (that is, OPENCV_BUILD\install\include\opencv2\<module>). The supplied modules with the current version of OpenCV are as follows:

All OpenCV classes and functions are in the cv namespace. Consequently, we will have the following two options in our source code:

The DataType class defines the primitive datatypes for OpenCV. The primitive datatypes can be bool, unsigned char, signed char, unsigned short, signed short, int, float, double, or a tuple of values of one of these primitive types. Any primitive type can be defined by an identifier in the following form:

CV_<bit depth>{U|S|F}C(<number of channels>)

In the preceding code, U, S, and F stand for unsigned, signed, and float respectively. For the single channel arrays, the following enumeration is applied describing the datatypes:

enum {CV_8U=0, CV_8S=1, CV_16U=2, CV_16S=3, CV_32S=4, CV_32F=5, CV_64F=6};

The following diagram shows a graphical representation of a single channel (4 x 4) array with 8 bits of unsigned integers (CV_8U). In this case, each element should have a value from zero to 255, which may be represented by a grayscale image.

Single channel array of 8 bit unsigned integers for a greyscale image

We can define all of the preceding datatypes for multichannel arrays (up to 512 channels). The following diagram illustrates a graphical representation of three channels 4 x 4 array of 8 bits of unsigned integers (CV_8UC3). In this example, the array consists of tuples of three elements corresponding to an RGB image.

A three-channel array of 8 bit unsigned integers for an RGB image

The OpenCV Mat class is used for dense n-dimensional single or multichannel arrays. It can store real or complex-valued vectors and matrices, colored or grayscale images, histograms, point clouds, and so on. There are many different ways to create a Mat object, the most popular being the constructor where the size and type of the array are specified as follows:

Mat(nrows, ncols, type[, fillValue])

The initial value for the array elements might be set by the Scalar class as a typical four-element vector (for the RGB and transparency components of the image stored in the array). Next, we show some usage examples of Mat as follows:

Mat img_A(640, 480, CV_8U, Scalar(255)); // white image
// 640 x 480 single-channel array with 8 bits of unsigned integers
// (up to 255 values, valid for a grayscale image, for example,
// 255=white)
…
Mat img_B(Size(800, 600), CV_8UC3, Scalar(0,255,0)); // Green image
// 800 x 600 three channel array with 8 bits of unsigned integers
// (up to 24 bits color depth, valid for a RGB color image)

The Mat class is the main data structure that stores and manipulates images. OpenCV has implemented mechanisms to allocate and release memory automatically for these data structures. However, the programmer should still take special care when data structures share the same buffer memory.

Many functions in OpenCV process dense single or multichannel arrays usually using the Mat class. However, in some cases, a different datatype may be convenient, such as std::vector<>, Matx<>, Vec<>, or Scalar. For this purpose, OpenCV provides the proxy classes, InputArray and OutputArray, which allow any of the previous types to be used as parameters for functions.

To prepare the examples for this book, we used the Qt Creator IDE (included in the Qt 5.2 bundle) and OpenCV 2.4.9 compiled with MinGW g++ 4.8 and Qt functionality. Qt Creator is a free multiplatform IDE with very helpful features for C++ programming. However, the user can choose the tool chain to build the executables that best fit its needs.

Our first Qt Creator project with OpenCV will be quite a simple flip-image tool, named flipImage. This tool reads a color image file and transforms it into a grayscale image, flipped and saved into an output file.

For this application, we choose to create a new code project by navigating to File | New File or File | Project…, and then navigate to Non-Qt Project | Plain C++ Project. Then, we have to choose a project name and location. The next step is to pick a kit (that is, compiler) for the project (in our case, Desktop Qt 5.2.1 MinGW 32 bit) and location for the binaries generated. Usually, two possible build configurations (profiles) are used: debug and release. These profiles set the appropriate flags to build and run the binaries.

When a Qt Creator project is created, two special files (with the .pro and .pro.user extension) are generated to configure the build and run processes. The build process is determined by the kit chosen during the creation of the project. With the Desktop Qt 5.2.1 MinGW 32 bit kit, this process relies on the qmake and mingw32-make tools. With the .pro files as input, qmake generates the makefiles for Make (that is, mingw32-make) that drive the build process for each profile (that is, release and debug).

The qmake project file

For our flipImage sample project, the flipImage.pro file looks like the following code:

TARGET: flipImage
TEMPLATE = app
CONFIG += console
CONFIG -= app_bundle
CONFIG -= qt

SOURCES += \
    flipImage.cpp
INCLUDEPATH += C:\\opencv-buildQt\\install\\include
LIBS += -LC:\\opencv-buildQt\\install\\x64\mingw\\lib \
    -lopencv_core249.dll \
    -lopencv_highgui249.dll

The preceding file illustrates the options that qmake needs to generate the appropriate makefiles to build the binaries for our project. Each line starts with a tag indicating an option (TARGET, CONFIG, SOURCES, INCLUDEPATH, and LIBS) followed with a mark to add (+=) or remove (-=) the value of the option. In this sample project, we deal with the non-Qt console application. The executable file is flipImage.exe (TARGET) and the source file is flipImage.cpp (SOURCES). Since this project is an OpenCV application, the two last tags point out to the location of the header files (INCLUDEPATH) and the OpenCV libraries (LIBS) used by this particular project (for example, core and highgui). Note that a backslash at the end of the line denotes continuation in the next line. In Windows, path backslashes should be duplicated, as shown in the preceding example.

The following code shows the source code for the flipImage project:

#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <iostream>

using namespace std;
using namespace cv;

int main(int argc, char *argv[])
{
    int flip_code=0;
    Mat out_image; // Output image

    if (argc != 4) {//Check args. number
        cout << "Usage: <cmd> <flip_code> <file_in> <file_out>\n";
        return -1;
    }
    Mat in_image = imread(argv[2], CV_LOAD_IMAGE_GRAYSCALE);
    if (in_image.empty()) { // Check if read
        cout << "Error! Input image cannot be read...\n";
        return -1;
    }
    sscanf(argv[1], "%d", &flip_code); // Read flip code
    flip(in_image, out_image, flip_code);
    imwrite(argv[3], out_image); // Write image to file
    namedWindow("Flipped…"); // Creates a window
    imshow(win, out_image); // Shows output image on window
    cout << "Press any key to exit...\n";
    waitKey(); // Wait infinitely for key press
    return 0;
}

After building the project, we can run the flipImage application from the following command line:

CV_SAMPLES/flipImage_build/debug>flipImage.exe -1 lena.jpg lena_f.jpg

The following screenshot shows the window with the output image after flipping on both the axes (horizontal and vertical):

Input image (left) and output image after the flipImage tool has been applied (right)

The source code starts with the inclusion of the header files (core.hpp and highgui.hpp) associated with the modules used by the application. Note that it is also possible to include only the opencv.hpp header since it will in turn include all the header files of OpenCV.

The flipImage example gets the flip code and two file names (for the input and output images) as the command-line arguments. These arguments are obtained from the argv[] variable. The following example illustrates several essential tasks in an OpenCV application:

1. Read an image from the file (imread) to a Mat class and check whether the target variable is not empty (Mat::empty).

2. Call a procedure (for example, flip) with the proxy classes, InputArray (in_image) and OutputArray (out_image).

3. Write an image to a file (imwrite).

4. Create an output window (namedWindow) and show (imshow) an image on it.

5. Wait for a key (waitKey).

The code explanation is given as follows:

• Mat imread(const string& filename, int flags=1): This function loads an image from the specified file and returns it. It also returns an empty matrix if the image cannot be read. It supports the most usual image formats of the files, detected by their content rather than by their extension. The flags parameter indicates the color of the image loaded in the memory, which may differ from the original color of the image stored in the file. In the example code, this function is used as follows:

  Mat in_image = imread(argv[2], CV_LOAD_IMAGE_GRAYSCALE);

  Here, the filename is obtained from the command-line arguments (the second argument after the command name). The CV_LOAD_IMAGE_GRAYSCALE flag indicates that the image should be loaded in the memory as an 8 bit grayscale image. For a description of the available tags, it is recommended to read the OpenCV online documentation (available at http://docs.opencv.org/).

• bool imwrite(const string& filename, InputArray img, const vector<int>& params=vector<int>()): This function writes an image to a given file where some optional format parameters are specified after the second argument. The format of the output file is determined by the file extension. In our example code, this function is used without the format parameters as follows:

  imwrite(argv[3], out_image);

• void namedWindow(const string& winname, int flags=WINDOW_AUTOSIZE): This function creates a window without displaying it. The first argument is a string used as a name for the window and its identifier. The second argument is a flag or flag combination, which controls some window properties (for example, enable resize). Next, we show how this function is used in the example using a constant string as a name for the created window, as follows:

  namedWindow("Flipped …"); // Creates a window

  Compiling OpenCV with Qt adds some new functionality to the highgui module (more on that later). Then, the window created with Qt and the namedWindow function uses the default flags: CV_WINDOW_AUTOSIZE, CV_WINDOW_KEEPRATIO, or CV_GUI_EXPANDED.

• void imshow(const string& winname, InputArray mat): This function displays an array (image) in a window with the properties set previously with the specified flags when the window was created. In the example, this function is used as follows:

  imshow(win, out_image); // Shows output image on window

• int waitKey(int delay=0): This function waits for a key press or the milliseconds specified by delay (if delay is greater than zero). If delay is less than or equal to zero, it waits infinitely. It returns the key code if pressed or -1 if a key is not pressed after the delay. This function has to be used after creating and activating a window. In the example code, it is used as follows:

  waitKey(); // Wait infinitely for key press

A video deals with moving images rather than still images, that is, display of a frame sequence at a proper rate (FPS or frames per second). The following showVideo example illustrates how to read and play a video file with OpenCV:

//… (omitted for simplicity)
int main(int argc, char *argv[])
{
    Mat frame; // Container for each frame

    VideoCapture vid(argv[1]); // Open original video file
    if (!vid.isOpened()) // Check whether the file was opened
        return -1;
    int fps = (int)vid.get(CV_CAP_PROP_FPS);
    namedWindow(argv[1]); // Creates a window
    while (1) {
        if (!vid.read(frame)) // Check end of the video file
            break;
        imshow(argv[1], frame); // Show current frame on window
        if (waitKey(1000/fps) >= 0)
            break;
    }
    return 0;
}

The code explanation is given as follows:

In the preceding example, the waitKey function is used with a computed number of milliseconds (1000/fps) trying to play the video file at the same rate it was originally recorded. Playing a video at a faster/slower rate (more/less fps) than that will produce a faster/slower playback.

Usually, the computer vision problems we face are related with processing live video input from one or several cameras. In this section, we will describe the recLiveVid example, which grabs a video stream from a webcam (connected to our computer), displays the stream in a window, and records it in a file (recorded.avi). By default, in the following example, the video capture is taken from the camera with cam_id=0. However, it is possible to handle a second camera (cam_id=1) and grab the video from it, setting an argument at the command line:

//… (omitted for brevity)
int main(int argc, char *argv[])
{
    Mat frame;
    const char win_name[]="Live Video...";
    const char file_out[]="recorded.avi";
    int cam_id=0; // Webcam connected to the USB port
    double fps=20;

    if (argc == 2)
        sscanf(argv[1], "%d", &cam_id);

    VideoCapture inVid(cam_id); // Open camera with cam_id
    if (!inVid.isOpened())
        return -1;

    int width = (int)inVid.get(CV_CAP_PROP_FRAME_WIDTH);
    int height = (int)inVid.get(CV_CAP_PROP_FRAME_HEIGHT);
    VideoWriter recVid(file_out, CV_FOURCC('F','F','D','S'), fps, Size(width, height));
    if (!recVid.isOpened()) 
        return -1;

    namedWindow(win_name);
    while (1) {
        inVid >> frame; // Read frame from camera
        recVid << frame; // Write frame to video file
        imshow(win_name, frame); // Show frame
        if (waitKey(1000/fps) >= 0)
            break;
    }
    inVid.release(); // Close camera
    return 0;
}

The code explanation is given as follows:

This chapter started with an explanation of how to build and install the OpenCV library with Qt (using CMake, the GNU g++ compiler, and GNU Make). Then, it is given a quick look to the modules organization of the library with an easy explanation of its basic API concepts. The chapter follows up with a more detailed revision of the fundamental data structures to store arrays and manipulate images. Also, three examples of code, such as flipImage, showVideo , and recLiveVid are explained to illustrate the basic use of the OpenCV library. The next chapter will introduce you to the two mainstream options available to provide graphical user interface capabilities for OpenCV programs.