Chapter 2. Handling Matrices, Files, Cameras, and GUIs

This chapter will enable you to perform basic operations required in computer vision, such as dealing with matrices, opening files, capturing videos from a camera, playing videos, and creating GUIs for prototype applications.

In this chapter, the following topics will be covered:

• Basic matrix manipulation

• Pixel manipulation

• How to load and display images from files

• How to capture a video from a camera

• Video playback

• Swing GUI's integration with OpenCV

By the end of this chapter, you should be able to get this computer vision application started by loading images and creating nice GUIs to manipulate them.

Basic matrix manipulation

From a computer vision background, we can see an image as a matrix of numerical values, which represents its pixels. For a gray-level image, we usually assign values ranging from 0 (black) to 255 (white) and the numbers in between show a mixture of both. These are generally 8-bit images. So, each element of the matrix refers to each pixel on the gray-level image, the number of columns refers to the image width, as well as the number of rows refers to the image's height. In order to represent a color image, we usually adopt each pixel as a combination of three basic colors: red, green, and blue. So, each pixel in the matrix is represented by a triplet of colors.

Note

It is important to observe that with 8 bits, we get 2 to the power of eight (2⁸), which is 256. So, we can represent the range from 0 to 255, which includes, respectively the values used for black and white levels in 8-bit grayscale images. Besides this, we can also represent these levels as floating points...

Pixel manipulation

Pixel manipulation is often required for one to access pixels in an image. There are several ways to do this and each one has its advantages and disadvantages. A straightforward method to do this is the put(row, col, value) method. For instance, in order to fill our preceding matrix with values {1, 2, 3}, we will use the following code:

for(int i=0;i<image.rows();i++){
  for(int j=0;j<image.cols();j++){ 
    image.put(i, j, new byte[]{1,2,3});
  }
}

Tip

Note that in the array of bytes {1, 2, 3}, for our matrix, 1 stands for the blue channel, 2 for the green, and 3 for the red channel, as OpenCV stores its matrix internally in the BGR (blue, green, and red) format.

It is okay to access pixels this way for small matrices. The only problem is the overhead of JNI calls for big images. Remember that even a small 640 x 480 pixel image has 307,200 pixels and if we think about a colored image, it has 921,600 values in a matrix. Imagine that it might take around 50ms to make...

Loading and displaying images from files

Most computer vision applications need to retrieve images from some where. In case you need to get them from files, OpenCV comes with several image file loaders. Unfortunately, some loaders depend on codecs that sometimes aren't shipped with the operating system, which might cause them not to load. From the documentation, we see that the following files are supported with some caveats:

• Windows bitmaps: *.bmp, *.dib

• JPEG files: *.jpeg, *.jpg, *.jpe

• JPEG 2000 files: *.jp2

• Portable Network Graphics: *.png

• Portable image format: *.pbm, *.pgm, *.ppm

• Sun rasters: *.sr, *.ras

• TIFF files: *.tiff, *.tif

Note that Windows bitmaps, the portable image format, and sun raster formats are supported by all platforms, but the other formats depend on a few details. In Microsoft Windows and Mac OS X, OpenCV can always read the jpeg, png, and tiff formats. In Linux, OpenCV will look for codecs supplied with the OS, as stated by the documentation, so remember to install the relevant...

Displaying an image with Swing

OpenCV developers are used to a simple cross-platform GUI by OpenCV, which was called as HighGUI, and a handy method called imshow. It constructs a window easily and displays an image within it, which is nice to create quick prototypes. As Java comes with a popular GUI API called Swing, we had better use it. Besides, no imshow method was available for Java until its 2.4.7.0 version was released. On the other hand, it is pretty simple to create such functionality. Refer to the reference code in chapter2/swing-imageshow.

Let's break down the work in to two classes: App and ImageViewer. The App class will be responsible for loading the file, while ImageViewer will display it. The application's work is simple and will only need to use Imgcodecs's imread method, which is shown as follows:

package org.javaopencvbook;

import java.io.File;
…
import org.opencv.imgcodecs.Imgcodecs;

public class App
{
  static{ System.loadLibrary(Core.NATIVE_LIBRARY_NAME); }

public static...

Capturing a video from a camera

The process of capturing frames from a webcam is very complex and it involves hardware details as well as heavy decoding or decompression algorithms. Fortunately, OpenCV has wrapped it all in a simple, yet powerful class called VideoCapture. This class not only grabs an image from a webcam, but also reads video files. In case more advanced access to a camera is required, you may want to use its specialized drivers.

You can think of a video stream as a series of pictures and you can retrieve each image in Mat and process it as you like. In order to use the VideoCapture class to capture a webcam stream, you need to instantiate it using the VideoCapture(int device) constructor. Note that the constructor parameter refers to the camera index in case you have several cameras. So, if you have one built-in camera and one USB camera and you create a videocapture object, such as new VideoCapture(1), then this object will refer to your built-in camera, while new VideoCapture...

Video playback

Another important I/O task in computer vision is being able to open and process a video file. Fortunately, OpenCV can easily deal with videos through the VideoCapture class. Instead of constructing it with a device number, as was done previously, we need to create it with the file path. We can also use the empty constructor and make the open(String filename) method responsible for pointing to the file.

The videoplayback project available in the chapter's source code has the same structure as the swing-imageshow project, explained previously. It only differs when you initialize the VideoCapture instance:

VideoCapture capture = new VideoCapture("src/main/resources/videos/tree.avi");

We have also put a 50ms delay between each frame so that the whole video doesn't play too fast. There is also code that you can use to manipulate InterruptedException. Note that the video files won't play with the same velocity as seen in a video player device. This is because the capture.read(webcamMatImage...

Swing GUI's integration with OpenCV

It is important to have rich graphical user interfaces while debugging or experimenting with computer vision projects, since some tasks might require a lot of tuning. This way, dealing with sliders, buttons, labels, and mouse events should be in the backpack of any computer vision researcher. Thankfully, you can work with all of these components in a relatively easy way in Swing. In this section, we will cover the most important parts of creating an application that loads an image and blurs it at several levels through a slider. This application also makes use of mouse events to highlight details in the image as well as a nice button to click and clear everything. The next screenshot gives us a good idea of how the application works. The code can be found in the opencv-gui project within the code bundle for this book.

The code to load an image is not new to us and can be found in the Displaying an image with Swing section. We will pay closer attention to...

Summary

Wow! A lot of details have been covered in this chapter, but we have finally grasped the development of a complete application for computer vision. We touched on the topic of core structure from OpenCV, which is the Mat class for basic pixel manipulation, and its close relation to Swing's BufferedImage class. Besides this, we covered important tasks such as opening image files and displaying them in a Swing application. The important area of live video streaming has been covered with the VideoCapture class, which shows you how to obtain frames from a webcam as well as from video files. Finally, we created a rich graphical user interface application with sliders, labels, buttons and by handling mouse events in Java.

The foundations of working with a Java OpenCV API have been set and we are ready to go on to the next chapter, which will deal with core operators in image processing, such as smoothing filters to remove noise, using morphological operators to isolate elements, using bucket...