Using OpenGL for the High-Speed Filtering of Images

In the previous chapters, we learned a lot about how to use OpenCV to deal with images and videos. Most of these processes are done by the CPU. In this chapter, we will explore another way to process images, that is, by moving the image filtering from the CPU to the graphics processing unit (GPU) using OpenGL.

In many types of software, such as the Google Chrome browser, you may see an option for hardware acceleration, or similar, on the Settings page. Usually, these settings mean that the graphics card (or the GPU) is being used to do rendering or computing. This approach, which uses another processor rather than the CPU to do the computing or rendering, is called heterogeneous computing. There are many ways to do heterogeneous computing, including OpenCL, which we mentioned in Chapters 6, Object Detection in Real Time while...

Technical requirements

A basic knowledge of the C and C++ programming languages is a requirement to follow this chapter. Since OpenGL will be a predominant part of this chapter, a good understanding of OpenGL will also be a big advantage.

Considering that we will use Qt and OpenCV along with OpenGL, readers are required, at the very least, to have Qt version 5 and OpenCV 4.0.0 installed, in the same way as for previous chapters.

All the code for this chapter can be found in our code repository at https://github.com/PacktPublishing/Qt-5-and-OpenCV-4-Computer-Vision-Projects/tree/master/Chapter-08.

You can check out the following video to see the code in action: http://bit.ly/2Fj4z3Y

Hello OpenGL

OpenGL is not a typical programming library like OpenCV or Qt. Its maintainer, the Khronos group, only designs and defines the API of OpenGL as a specification; however, it is not responsible for the implementation. Instead, it is the graphics card manufacturer's responsibility to give the implementation. Most manufacturers, such as Intel, AMD, and Nvidia, give their implementation in the drivers of their graphics cards. On Linux, there's an OpenGL implementation called Mesa, which can do software rendering while hardware rendering is also supported if the graphics card is driven correctly.

Nowadays, OpenGL has a steep learning curve; this is because you need to understand the heterogeneous architecture and another programming language, called the OpenGL Shading Language, as well as C and C++. In this chapter, we will use the new style API, which was introduced...

OpenGL in Qt

In the early days, Qt had a module named OpenGL, but in Qt 5.x, that module is deprecated. A new version of facilities for OpenGL supports is put into the gui module; if you search classes whose names start with QOpenGL in the Qt docs, you will find them. Besides the facilities that reside in the gui module, there's another important class named QOpenGLWidget in the widgets module. In this section, we will use some of these facilities to draw a triangle with OpenGL in Qt.

First, let's create the required Qt projects:

$ pwd
/home/kdr2/Work/Books/Qt5-And-OpenCV4-Computer-Vision-Projects/Chapter-08
$ mkdir QtGL
$ cd QtGL/
$ touch main.cpp
$ qmake -project
$ ls
QtGL.pro main.cpp
$

Then, we change the content of the QtGL.pro project file to the following:

TEMPLATE = app
TARGET = QtGL

QT += core gui widgets

INCLUDEPATH += .

DEFINES += QT_DEPRECATED_WARNINGS

# Input
HEADERS...

Filtering images with OpenGL

So far, we have learned how to draw a simple triangle in OpenGL. In this section, we will learn how to draw images and filter them with OpenGL.

We will do this work in a copy of the QtGL project, that is, a new project named GLFilter. The creation of the project simply involves straightforward copying and a little bit of renaming, just as we did in the previous chapters. I won't repeat it again here, so please copy it by yourself.

Drawing images with OpenGL

In order to draw an image on the OpenGL viewport, we should introduce another concept of OpenGL—the texture. A texture in OpenGL is usually a 2D image, and it is generally used to add visual detail to an object, which is mainly a...

Using OpenGL with OpenCV

In the preceding section, when we load the source image and flip it, we use Qt to do the work. This work can also be done by using the OpenCV library:

        img = cv::imread("./images/lizard.jpg");
        cv::Mat tmp;
        cv::flip(img, tmp, 0);
        cvtColor(tmp, img, cv::COLOR_BGR2RGB);
        // ...
        glTexImage2D(
            GL_TEXTURE_2D, 0, GL_RGB,
            img.cols, img.rows, 0, GL_RGB, GL_UNSIGNED_BYTE, img.data);
        // ...

Similarly, when saving the resulting image, we can do it like this:

        cv::Mat output(img.rows, img.cols, CV_8UC3);
        glReadPixels(
            0, 0, img.cols, img.rows, GL_RGB, GL_UNSIGNED_BYTE, output.data);
        cv::Mat tmp;
        cv::flip(output, tmp, 0);
        cvtColor(tmp, output, cv::COLOR_RGB2BGR);
        cv::imwrite(path.toStdString(), output);

Both QImage and cv::Mat represent...

Summary

OpenGL is a specification for developing 2D and 3D graphics applications, and it has many implementations. In this chapter, we learned a lot about it, including drawing primitives such as triangles, integrating it with the Qt library, using a texture to render images, and filtering images in the fragment shaders. Additionally, we used OpenGL in a non-typical way, that is, we didn't use it for graphics rendering, but for heterogeneous computing and processing images on the GPU in parallel.

Finally, we learned how to integrate OpenCV and OpenGL, and, in my opinion, by comparing this approach to using the raw OpenGL API, this is not a recommended way for a production application, but feel free to use it in your attempts.

With the end of this chapter, we have finished the book. I hope that all the projects we developed with Qt, OpenCV, Tesseract, the many DNN models,...

Further reading

There is a lot more to OpenGL than what we have covered in this chapter. Since we mainly focused on image processing in this book, we only showed how to use it to filter images. If you are interested in OpenGL, you can find more resources on its official website at https://www.opengl.org. There are also a number of awesome tutorials on the internet, for example, https://learnopengl.com and https://open.gl/.

If you are not so interested in OpenGL, which is mainly for 2D and 3D graphics development, but are interested in heterogeneous computing, then you can refer to OpenCL or CUDA. OpenCL is very similar to OpenGL; it is a specification maintained by the Khronos group. Additionally, the next generations of OpenGL and OpenCL are now consolidated into one specification named Vulkan, so Vulkan is also a good option to choose. CUDA is a proprietary solution for heterogeneous...