Chapter 9. Learning Object Tracking

In the previous chapter, we learned about video surveillance, background modeling, and morphological image processing. We discussed how we can use different morphological operators to apply cool visual effects to input images. In this chapter, we will learn how to track an object in a live video. We will discuss the different characteristics of an object that can be used to track it. We will also learn about different methods and techniques used for object tracking. Object tracking is used extensively in robotics, self-driving cars, vehicle tracking, player tracking in sports, video compression, and so on.

By the end of this chapter, you will learn:

• How to track colored objects

• How to build an interactive object tracker

• What is a corner detector

• How to detect good features to track

• How to build an optical flow-based feature tracker

Tracking objects of a specific color

In order to build a good object tracker, we need to understand what characteristics can be used to make our tracking robust and accurate. So, let's take a baby step in this direction, and see how we can use colorspaces to come up with a good visual tracker. One thing to keep in mind is that the color information is sensitive to lighting conditions. In real-world applications, you need to do some preprocessing to take care of this. But for now, let's assume that somebody else is doing this and we are getting clean color images.

There are many different colorspaces and picking up a good one will depend on what people use for different applications. While RGB is the native representation on the computer screen, it's not necessarily ideal for humans. When it comes to humans, we give names to colors that are based on their hue. This is why HSV (Hue Saturation Value) is probably one of the most informative colorspaces. It closely aligns with how we perceive...

Building an interactive object tracker

A colorspace-based tracker gives us the freedom to track a colored object, but we are also constrained to a predefined color. What if we just want to randomly pick an object? How do we build an object tracker that can learn the characteristics of the selected object and track it automatically? This is where the CAMShift algorithm, which stands for Continuously Adaptive Meanshift, comes into the picture. It's basically an improved version of the Meanshift algorithm.

The concept of Meanshift is actually nice and simple. Let's say we select a region of interest, and we want our object tracker to track that object. In this region, we select a bunch of points based on the color histogram and compute the centroid of spatial points. If the centroid lies at the center of this region, we know that the object hasn't moved. But if the centroid is not at the center of this region, then we know that the object is moving in some direction. The movement of the centroid...

Detecting points using the Harris corner detector

Corner detection is a technique used to detect interest points in the image. These interest points are also called feature points or simply features in Computer Vision terminology. A corner is basically an intersection of two edges. An interest point is basically something that can be uniquely detected in an image. A corner is a particular case of an interest point. These interest points help us characterize an image. These points are used extensively in applications, such as object tracking, image classification, visual search, and so on. Since we know that the corners are interesting, let's see how we can detect them.

In Computer Vision, there is a popular corner detection technique called the Harris corner detector. We construct a 2 x 2 matrix based on partial derivatives of the grayscale image, and then analyze the eigenvalues. Now what does this mean? Well, let's dissect it so that we can understand it better. Let's consider a small...

Shi-Tomasi Corner Detector

The Harris corner detector performs well in many cases, but it can still be improved. Around six years after the original paper by Harris and Stephens, Shi-Tomasi came up with something better and they called it Good Features To Track. You can read the original paper at: http://www.ai.mit.edu/courses/6.891/handouts/shi94good.pdf. They used a different scoring function to improve the overall quality. Using this method, we can find the N strongest corners in the given image. This is very useful when we don't want to use every single corner to extract information from the image. As discussed earlier, a good interest point detector is very useful in applications, such as object tracking, object recognition, image search, and so on.

If you apply the Shi-Tomasi corner detector to an image, you will see something like this:

As you can see here, all the important points in the frame are captured. Let's take a look at the following code to track these features:

int main(int...

Feature-based tracking

Feature-based tracking refers to tracking individual feature points across successive frames in the video. The advantage here is that we don't have to detect feature points in every single frame. We can just detect them once and keep tracking them after that. This is more efficient as compared to running the detector on every frame. We use a technique called optical flow to track these features. Optical flow is one of the most popular techniques in Computer Vision. We choose a bunch of feature points, and track them through the video stream. When we detect the feature points, we compute the displacement vectors and show the motion of those keypoints between consecutive frames. These vectors are called motion vectors.

A motion vector for a particular point is just a directional line that indicates where that point has moved as compared to the previous frame. Different methods are used to detect these motion vectors. The two most popular algorithms are the Lucas-Kanade...

Summary

In this chapter, we learned about object tracking. We learned how to use the HSV colorspace to track colored objects. We discussed clustering techniques used for object tracking and how we can build an interactive object tracker using the CAMShift algorithm. We learned about corner detectors and how to track corners in a live video. We discussed how to track features in a video using optical flow. We also learned the concepts behind Lucas-Kanade and Farneback algorithms and implemented them as well.

In the next chapter, we will discuss segmentation algorithms and see how we can use them for text recognition.