Chapter 7. Extracting Image Features and Descriptors

In this chapter, we will discuss feature detectors and descriptors, along with various applications of different types of feature detectors/extractors in image processing. We will start by defining feature detectors and descriptors. We will then continue our discussion on a few popular feature detectors such as Harris Corner/SIFT and HOG, and then their applications in important image processing problems such as image matching and object detection, respectively, with scikit-image and python-opencv (cv2) library functions.

 The topics to be covered in this chapter are as follows:

• Feature detectors versus descriptors, to extract features/descriptors from images
• Harris Corner Detector and the application of Harris Corner features in image matching (with scikit-image)
• Blob detectors with LoG, DoG, and DoH (with scikit-image)
• Extraction of Histogram of Oriented Gradients features
• SIFT, ORB, and BRIEF features and their application in image matching...

Feature detectors versus descriptors

In image processing, (local) features refer to a group of key/salient points or information relevant to an image processing task, and they create an abstract, more general (and often robust) representation of an image. A family of algorithms that choose a set of interest points from an image based on some criterion (for example, cornerness, local maximum/minimum, and so on, that detect/extract the features from an image) are called feature detectors/extractors.

 

 

On the contrary, a descriptor consists of a collection of values to represent the image with the features/interest points (for example, HOG features). Feature extraction can also be thought of as an operation that transforms an image into a set of feature descriptors, and, hence, a special form of dimensionality reduction. A local feature is usually formed by an interest point and its descriptor together.

Global features from the whole image (for example, image histogram) are often not desirable...

Harris Corner Detector

This algorithm explores the intensity changes within a window as the window changes location inside an image. Unlike an edge, for which intensity values change abruptly in only one direction, there is a significant change in intensity values at a corner in all directions. Hence, a large change in intensity value should result when the window is shifted in any direction at the corner (with good localization); this fact is exploited in the Harris Corner Detector algorithm. It is invariant to rotation, but not to scale (that is, the corner points found from an image remain unchanged when the image undergoes a rotation transformation, but change when the image is resized). In this section, we shall discuss how to implement a Harris Corner Detector with scikit-image.

With scikit-image

The next code snippet shows how to detect corner points in an image using the Harris Corner Detector with the corner_harris() function from the scikit-imagefeature module:

image = imread('.....

Blob detectors with LoG, DoG, and DoH

In an image, a blob is defined as either a bright on a dark region, or a dark on a bright region. In this section, we will discuss how to implement blob features detection in an image using the following three algorithms. The input image is a colored (RGB) butterfly image.

Laplacian of Gaussian (LoG)

In the Chapter 3, Convolution and Frequency Domain Filtering, we saw that the cross correlation of an image with a filter can be viewed as pattern matching; that is, comparing a (small) template image (of what we want to find) against all local regions in the image. The key idea in blob detection comes from this fact. We have already seen how an LoG filter with zero crossing can be used for edge detection in the last chapter. LoG can also be used to find scale invariant regions by searching 3D (location + scale) extrema of the LoG with the concept of Scale Space. If the scale of the Laplacian (σ of the LoG filter) gets matched with the scale of the blob, the...

Histogram of Oriented Gradients

A popular feature descriptor for object detection is the Histogram of Oriented Gradients (HOG). In this section, we will discuss how HOG descriptors can be computed from an image.

Algorithm to compute HOG descriptors

The following steps describe the algorithm:

1. If you wish to, you can globally normalize the image
2. Compute the horizontal and vertical gradient images
3. Compute the gradient histograms
4. Normalize across blocks
5. Flatten into a feature descriptor vector

HOG descriptors are the normalized block descriptors finally obtained by using the algorithm. 

Compute HOG descriptors with scikit-image

Let's now compute the HOG descriptors using the scikit-image feature module's hog() function and visualize them:

from skimage.feature import hog
from skimage import exposure
image = rgb2gray(imread('../images/cameraman.jpg'))
fd, hog_image = hog(image, orientations=8, pixels_per_cell=(16, 16), cells_per_block=(1, 1), visualize=True) 
print(image.shape, len(fd))
# ((256L, 256L),...

Scale-invariant feature transform

Scale-invariant feature transform (SIFT) descriptors provide an alternative representation for image regions. They are very useful for matching images. As demonstrated earlier, simple corner detectors work well when the images to be matched are similar in nature (with respect to scale, orientation, and so on). But if they have different scales and rotations, the SIFT descriptors are needed to be used to match them. SIFT is not only just scale invariant, but it still obtains good results when rotation, illumination, and viewpoints of the images change as well. 

 

 

Let's discuss the primary steps involved in the SIFT algorithm that transforms image content into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters.

Algorithm to compute SIFT descriptors

• Scale-space extrema detection: Search over multiple scales and image locations, the location and characteristic scales are given by DoG detector
• Keypoint localization...

Haar-like features

Haar-like features are very useful image features used in object detection. They were introduced in the first real-time face detector by Viola and Jones. Using integral images, Haar-like features of any size (scale) can be efficiently computed in constant time. The computation speed is the key advantage of a Haar-like feature over most other features. These features are just like the convolution kernels (rectangle filters) introduced in Chapter 3, Convolution and Frequency Domain Filtering. Each feature corresponds to a single value computed by subtracting a sum of pixels under a white rectangle from a sum of pixels under a black rectangle. The next diagram shows different types of Haar-like features, along with the important Haar-like features for face detection:

The first and the second important feature for face detection shown here seems to focus on the fact that the region of the eyes is often darker than the region of the nose and cheeks, and that the eyes are darker...

Summary

In this chapter, we discussed a few important feature detection and extraction techniques to compute different types of feature descriptors from an image using Python's scikit-image and cv2 (python-opencv) libraries. We started with the basic concepts of local feature detectors and descriptors for an image, along with their desirable properties. Then we discussed the Harris Corner Detectors to detect corner interest points of an image and use them to match two images (with the same object captured from different viewpoints). Next, we discussed blob detection using LoG/DoG/DoH filters. Next, we discussed HOG, SIFT, ORB, BRIEF binary detectors/descriptors and how to match images with these features. Finally, we discussed Haar-like features and face detection with the Viola—Jones algorithm. By the end of this chapter, you should be able to compute different features/descriptors of an image with Python libraries. Also, you should be able to match images with different types of feature...

Questions

1. Implement the Harris Corner Detector with sub-pixel accuracy with cv2.
2. Play with a few different pre-trained Haar Cascade classifiers with cv2 and try to detect multiple faces from images.
3. Use the FLANN-based approximate nearest-neighborhood matcher instead of BFMatcher to match the images with books with cv2.
4. Compute the SURF keypoints and use them for image matching with cv2. 

Further reading

• http://scikit-image.org/docs/dev/api/skimage.feature.html
• https://docs.opencv.org/3.1.0/da/df5/tutorial_py_sift_intro.html
• https://sandipanweb.wordpress.com/2017/10/22/feature-detection-with-harris-corner-detector-and-matching-images-with-feature-descriptors-in-python/
• https://sandipanweb.wordpress.com/2018/06/30/detection-of-a-human-object-with-hog-descriptor-features-using-svm-primal-quadprog-implementation-using-cvxopt-in-python/
• http://vision.stanford.edu/teaching/cs231b_spring1213/slides/HOG_2011_Stanford.pdf
• http://cvgl.stanford.edu/teaching/cs231a_winter1415/lecture/lecture10_detector_descriptors_2015.pdf
• https://www.cis.rit.edu/~cnspci/references/dip/feature_extraction/harris1988.pdf
• https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf
• https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf
• https://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdf