This article written by Gloria Bueno García, Oscar Deniz Suarez, José Luis Espinosa Aranda, Jesus Salido Tercero, Ismael Serrano Gracia, and Noelia Vállez Enanois, the authors of Learning Image Processing with OpenCV, is intended as a first contact with OpenCV, its installation, and first basic programs. We will cover the following topics:

A brief introduction to OpenCV for the novice, followed by an easy step-by-step guide to the installation of the library

A quick tour of OpenCV's structure after the installation in the user's local disk

Quick recipes to create projects using the library with some common programming frameworks

How to use the functions to read and write images and videos

(For more resources related to this topic, see here.)

An introduction to OpenCV

Initially developed by Intel, OpenCV (Open Source Computer Vision) is a free cross-platform library for real-time image processing that has become a de facto standard tool for all things related to Computer Vision. The first version was released in 2000 under BSD license and since then, its functionality has been very much enriched by the scientific community. In 2012, the nonprofit foundation OpenCV.org took on the task of maintaining a support site for developers and users.

At the time of writing this article, a new major version of OpenCV (Version 3.0) is available, still on beta status. Throughout the article, we will present the most relevant changes brought with this new version.

OpenCV is available for the most popular operating systems, such as GNU/Linux, OS X, Windows, Android, iOS, and some more. The first implementation was in the C programming language; however, its popularity grew with its C++ implementation as of Version 2.0. New functions are programmed with C++. However, nowadays, the library has a full interface for other programming languages, such as Java, Python, and MATLAB/Octave. Also, wrappers for other languages (such as C#, Ruby, and Perl) have been developed to encourage adoption by programmers.

In an attempt to maximize the performance of computing intensive vision tasks, OpenCV includes support for the following:

Multithreading on multicore computers using Threading Building Blocks (TBB)—a template library developed by Intel.

A subset of Integrated Performance Primitives (IPP) on Intel processors to boost performance. Thanks to Intel, these primitives are freely available as of Version 3.0 beta.

Interfaces for processing on Graphic Processing Unit (GPU) using Compute Unified Device Architecture (CUDA) and Open Computing Language (OpenCL).

The applications for OpenCV cover areas such as segmentation and recognition, 2D and 3D feature toolkits, object identification, facial recognition, motion tracking, gesture recognition, image stitching, high dynamic range (HDR) imaging, augmented reality, and so on. Moreover, to support some of the previous application areas, a module with statistical machine learning functions is included.

Downloading and installing OpenCV

OpenCV is freely available for download at http://opencv.org. This site provides the last version for distribution (currently, 3.0 beta) and older versions.

Special care should be taken with possible errors when the downloaded version is a nonstable release, for example, the current 3.0 beta version.

On http://opencv.org/releases.html, suitable versions of OpenCV for each platform can be found. The code and information of the library can be obtained from different repositories depending on the final purpose:

The main repository (at http://sourceforge.net/projects/opencvlibrary), devoted to final users. It contains binary versions of the library and ready-to-compile sources for the target platform.

The test data repository (at https://github.com/itseez/opencv_extra) with sets of data to test purposes of some library modules.

The contributions repository (at http://github.com/itseez/opencv_contrib) with the source code corresponding to extra and cutting-edge features supplied by contributors. This code is more error-prone and less tested than the main trunk.

With the last version, OpenCV 3.0 beta, the extra contributed modules are not included in the main package. They should be downloaded separately and explicitly included in the compilation process through the proper options. Be cautious if you include some of those contributed modules, because some of them have dependencies on third-party software not included with OpenCV.

The documentation site (at http://docs.opencv.org/master/) for each of the modules, including the contributed ones.

The development repository (at https://github.com/Itseez/opencv) with the current development version of the library. It is intended for developers of the main features of the library and the "impatient" user who wishes to use the last update even before it is released.

Rather than GNU/Linux and OS X, where OpenCV is distributed as source code only, in the Windows distribution, one can find precompiled (with Microsoft Visual C++ v10, v11, and v12) versions of the library. Each precompiled version is ready to be used with Microsoft compilers. However, if the primary intention is to develop projects with a different compiler framework, we need to compile the library for that specific compiler (for example, GNU GCC).

The fastest route to working with OpenCV is to use one of the precompiled versions included with the distribution. Then, a better choice is to build a fine-tuned version of the library with the best settings for the local platform used for software development. This article provides the information to build and install OpenCV on Windows. Further information to set the library on Linux can be found at http://docs.opencv.org/doc/tutorials/introduction/linux_install and https://help.ubuntu.com/community/OpenCV.

Getting a compiler and setting CMake

A good choice for cross-platform development with OpenCV is to use the GNU toolkit (including gmake, g++, and gdb). The GNU toolkit can be easily obtained for the most popular operating systems. Our preferred choice for a development environment consists of the GNU toolkit and the cross-platform Qt framework, which includes the Qt library and the Qt Creator Integrated Development Environment(IDE). The Qt framework is freely available at http://qt-project.org/.

After installing the compiler on Windows, remember to properly set the Path environment variable, adding the path for the compiler's executable, for example, C:QtQt5.2.15.2.1mingw48_32bin for the GNU/compilers included with the Qt framework. On Windows, the free Rapid Environment Editor tool (available at http://www.rapidee.com) provides a convenient way to change Path and other environment variables.

To manage the build process for the OpenCV library in a compiler-independent way, CMake is the recommended tool. CMake is a free and open source cross-platform tool available at http://www.cmake.org/.

Configuring OpenCV with CMake

Once the sources of the library have been downloaded into the local disk, it is required that you configure the makefiles for the compilation process of the library. CMake is the key tool for an easy configuration of OpenCV's installation process. It can be used from the command line or in a more user-friendly way with its Graphical User Interface (GUI) version.

The steps to configure OpenCV with CMake can be summarized as follows:

Choose the source (let's call it OPENCV_SRC in what follows) and target (OPENCV_BUILD) directories. The target directory is where the compiled binaries will be located.

Mark the Grouped and Advanced checkboxes and click on the Configure button.

Choose the desired compiler (for example, GNU default compilers, MSVC, and so on).

Set the preferred options and unset those not desired.

Click on the Configure button and repeat steps 4 and 5 until no errors are obtained.

Click on the Generate button and close CMake.

The following screenshot shows you the main window of CMake with the source and target directories and the checkboxes to group all the available options:

handling-image-and-video-files-img-0

The main window of CMake after the preconfiguration step

For brevity, we use OPENCV_BUILD and OPENCV_SRC in this text to denote the target and source directories of the OpenCV local setup, respectively. Keep in mind that all directories should match your current local configuration.

During the preconfiguration process, CMake detects the compilers present and many other local properties to set the build process of OpenCV. The previous screenshot displays the main CMake window after the preconfiguration process, showing the grouped options in red.

It is possible to leave the default options unchanged and continue the configuration process. However, some convenient options can be set:

BUILD_EXAMPLES: This is set to build some examples using OpenCV.

BUILD_opencv_<module_name>: This is set to include the module (module_name) in the build process.

OPENCV_EXTRA_MODULES_PATH: This is used when you need some extra contributed module; set the path for the source code of the extra modules here (for example, C:/opencv_contrib-master/modules).

WITH_QT: This is turned on to include the Qt functionality into the library.

WITH_IPP: This option is turned on by default. The current OpenCV 3.0 version includes a subset of the Intel Integrated Performance Primitives (IPP) that speed up the execution time of the library.

If you compile the new OpenCV 3.0 (beta), be cautious because some unexpected errors have been reported related to the IPP inclusion (that is, with the default value of this option). We recommend that you unset the WITH_IPP option.

If the configuration steps with CMake (loop through steps 4 and 5) don't produce any further errors, it is possible to generate the final makefiles for the build process. The following screenshot shows you the main window of CMake after a generation step without errors:

handling-image-and-video-files-img-1

Compiling and installing the library

The next step after the generation process of makefiles with CMake is the compilation with the proper make tool. This tool is usually executed on the command line (the console) from the target directory (the one set at the CMake configuration step). For example, in Windows, the compilation should be launched from the command line as follows:

OPENCV_BUILD>mingw32-make

This command launches a build process using the makefiles generated by CMake. The whole compilation typically takes several minutes. If the compilation ends without errors, the installation continues with the execution of the following command:

OPENCV_BUILD>mingw32-make install

This command copies the OpenCV binaries to the OPENCV_BUILDinstall directory.

If something went wrong during the compilation, we should run CMake again to change the options selected during the configuration. Then, we should regenerate the makefiles.

The installation ends by adding the location of the library binaries (for example, in Windows, the resulting DLL files are located at OPENCV_BUILDinstallx64mingwbin) to the Path environment variable. Without this directory in the Path field, the execution of every OpenCV executable will give an error as the library binaries won't be found.

To check the success of the installation process, it is possible to run some of the examples compiled along with the library (if the BUILD_EXAMPLES option was set using CMake). The code samples (written in C++) can be found at OPENCV_BUILDinstallx64mingwsamplescpp.

The short instructions given to install OpenCV apply to Windows. A detailed description with the prerequisites for Linux can be read at http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html. Although the tutorial applies to OpenCV 2.0, almost all the information is still valid for Version 3.0.

The structure of OpenCV

Once OpenCV is installed, the OPENCV_BUILDinstall directory will be populated with three types of files:

Header files: These are located in the OPENCV_BUILDinstallinclude subdirectory and are used to develop new projects with OpenCV.

Library binaries: These are static or dynamic libraries (depending on the option selected with CMake) with the functionality of each of the OpenCV modules. They are located in the bin subdirectory (for example, x64mingwbin when the GNU compiler is used).

Sample binaries: These are executables with examples that use the libraries. The sources for these samples can be found in the source package (for example, OPENCV_SRCsourcessamples).

OpenCV has a modular structure, which means that the package includes a static or dynamic (DLL) library for each module. The official documentation for each module can be found at http://docs.opencv.org/master/. The main modules included in the package are:

core: This defines the basic functions used by all the other modules and the fundamental data structures including the important multidimensional array Mat.

highgui: This provides simple user interface (UI) capabilities. Building the library with Qt support (the WITH_QT CMake option) allows UI compatibility with such a framework.

imgproc: These are image processing functions that include filtering (linear and nonlinear), geometric transformations, color space conversion, histograms, and so on.

imgcodecs: This is an easy-to-use interface to read and write images.

Pay attention to the changes in modules since OpenCV 3.0 as some functionality has been moved to a new module (for example, reading and writing images functions were moved from highgui to imgcodecs).

photo: This includes Computational Photography including inpainting, denoising, High Dynamic Range (HDR) imaging, and some others.

stitching: This is used for image stitching.

videoio: This is an easy-to-use interface for video capture and video codecs.

video: This supplies the functionality of video analysis (motion estimation, background extraction, and object tracking).

features2d: These are functions for feature detection (corners and planar objects), feature description, feature matching, and so on.

objdetect: These are functions for object detection and instances of predefined detectors (such as faces, eyes, smile, people, cars, and so on).

Some other modules are calib3d (camera calibration), flann (clustering and search), ml (machine learning), shape (shape distance and matching), superres (super resolution), video (video analysis), and videostab (video stabilization).

As of Version 3.0 beta, the new contributed modules are distributed in a separate package (opencv_contrib-master.zip) that can be downloaded from https://github.com/itseez/opencv_contrib. These modules provide extra features that should be fully understood before using them. For a quick overview of the new functionality in the new release of OpenCV (Version 3.0), refer to the document at http://opencv.org/opencv-3-0-beta.html.

Creating user projects with OpenCV

In this article, we assume that C++ is the main language for programming image processing applications, although interfaces and wrappers for other programming languages are actually provided (for instance, Python, Java, MATLAB/Octave, and some more).

In this section, we explain how to develop applications with OpenCV's C++ API using an easy-to-use cross-platform framework.

General usage of the library

To develop an OpenCV application with C++, we require our code to:

Include the OpenCV header files with definitions

Link the OpenCV libraries (binaries) to get the final executable

The OpenCV header files are located in the OPENCV_BUILDinstallincludeopencv2 directory where there is a file (*.hpp) for each of the modules. The inclusion of the header file is done with the #include directive, as shown here:

#include <opencv2/<module_name>/<module_name>.hpp>
// Including the header file for each module used in the code

With this directive, it is possible to include every header file needed by the user program. On the other hand, if the opencv.hpp header file is included, all the header files will be automatically included as follows:

#include <opencv2/opencv.hpp>
// Including all the OpenCV's header files in the code

Remember that all the modules installed locally are defined in the OPENCV_BUILDinstallincludeopencv2opencv_modules.hpp header file, which is generated automatically during the building process of OpenCV.

The use of the #include directive is not always a guarantee for the correct inclusion of the header files, because it is necessary to tell the compiler where to find the include files. This is achieved by passing a special argument with the location of the files (such as I<location> for GNU compilers).

The linking process requires you to provide the linker with the libraries (dynamic or static) where the required OpenCV functionality can be found. This is usually done with two types of arguments for the linker: the location of the library (such as -L<location> for GNU compilers) and the name of the library (such as -l<module_name>).

You can find a complete list of available online documentation for GNU GCC and Make at https://gcc.gnu.org/onlinedocs/ and https://www.gnu.org/software/make/manual/.

Tools to develop new projects

The main prerequisites to develop our own OpenCV C++ applications are:

OpenCV header files and library binaries: Of course we need to compile OpenCV, and the auxiliary libraries are prerequisites for such a compilation. The package should be compiled with the same compiler used to generate the user application.

A C++ compiler: Some associate tools are convenient as the code editor, debugger, project manager, build process manager (for instance CMake), revision control system (such as Git, Mercurial, SVN, and so on), and class inspector, among others. Usually, these tools are deployed together in a so-called Integrated Development Environment (IDE).

Any other auxiliary libraries: Optionally, any other auxiliary libraries needed to program the final application, such as graphical, statistical, and so on will be required.

The most popular available compiler kits to program OpenCV C++ applications are:

Microsoft Visual C (MSVC): This is only supported on Windows and it is very well integrated with the IDE Visual Studio, although it can be also integrated with other cross-platform IDEs, such as Qt Creator or Eclipse. Versions of MSVC that currently compatible with the latest OpenCV release are VC 10, VC 11, and VC 12 (Visual Studio 2010, 2012, and 2013).

GNU Compiler Collection GNU GCC: This is a cross-platform compiler system developed by the GNU project. For Windows, this kit is known as MinGW (Minimal GNU GCC). The version compatible with the current OpenCV release is GNU GCC 4.8. This kit may be used with several IDEs, such as Qt Creator, Code::Blocks, Eclipse, among others.

For the examples presented in this article, we used the MinGW 4.8 compiler kit for Windows plus the Qt 5.2.1 library and the Qt Creator IDE (3.0.1). The cross-platform Qt library is required to compile OpenCV with the new UI capabilities provided by such a library.

For Windows, it is possible to download a Qt bundle (including Qt library, Qt Creator, and the MinGW kit) from http://qt-project.org/. The bundle is approximately 700 MB.

Qt Creator is a cross-platform IDE for C++ that integrates the tools we need to code applications. In Windows, it may be used with MinGW or MSVC. The following screenshot shows you the Qt Creator main window with the different panels and views for an OpenCV C++ project:

handling-image-and-video-files-img-2

The main window of Qt Creator with some views from an OpenCV C++ project

Creating an OpenCV C++ program with Qt Creator

Next, we explain how to create a code project with the Qt Creator IDE. In particular, we apply this description to an OpenCV example.

We can create a project for any OpenCV application using Qt Creator by navigating to File | New File or File | Project… and then navigating to Non-Qt Project | Plain C++ Project. Then, we have to choose a project name and the location at which it will be stored. The next step is to pick a kit (that is, the compiler) for the project (in our case, Desktop Qt 5.2.1 MinGW 32 bit) and the location for the binaries generated. Usually, two possible build configurations (profiles) are used: debug and release. These profiles set the appropriate flags to build and run the binaries.

When a project is created using Qt Creator, two special files (with .pro and .pro.user extensions) are generated to configure the build and run processes. The build process is determined by the kit chosen during the creation of the project. With the Desktop Qt 5.2.1 MinGW 32 bit kit, this process relies on the qmake and mingw32-make tools. Using the *.pro file as the input, qmake generates the makefile that drives the build process for each profile (that is, release and debug). The qmake tool is used from the Qt Creator IDE as an alternative to CMake to simplify the build process of software projects. It automates the generation of makefiles from a few lines of information.

The following lines represent an example of a *.pro file (for example, showImage.pro):

TARGET: showImage
TEMPLATE = app
CONFIG += console
CONFIG -= app_bundle
CONFIG -= qt
SOURCES += 
   showImage.cpp
INCLUDEPATH += C:/opencv300-buildQt/install/include
LIBS += -LC:/opencv300-buildQt/install/x64/mingw/lib 
   -lopencv_core300.dll 
   -lopencv_imgcodecs300.dll
   -lopencv_highgui300.dll
   -lopencv_imgproc300.dll

The preceding file illustrates the options that qmake needs to generate the appropriate makefiles to build the binaries for our project. Each line starts with a tag indicating an option (TARGET, CONFIG, SOURCES, INCLUDEPATH, and LIBS) followed with a mark to add (+=) or remove (-=) the value of the option. In this sample project, we use the non-Qt console application. The executable file is showImage.exe (TARGET) and the source file is showImage.cpp (SOURCES). As this project is an OpenCV-based application, the two last tags indicate the location of the header files (INCLUDEPATH) and the OpenCV libraries (LIBS) used by this particular project (core, imgcodecs, highgui, and imgproc). Note that a backslash at the end of the line denotes continuation in the next line.

For a detailed description of the tools (including Qt Creator and qmake) developed within the Qt project, visit http://doc.qt.io/.

Summary

This article has now covered an introduction to OpenCV, how to download and install OpenCV, the structure of OpenCV, and how to create user projects with OpenCV.