Tracking Objects

In this chapter, we will explore a selection of techniques from the vast topic of object tracking, which is the process of locating a moving object in a movie or a video feed from a camera. Real-time object tracking is a critical task in many computer vision applications such as surveillance, perceptual user interfaces, augmented reality, object-based video compression, and driver assistance.

Tracking objects can be accomplished in several ways, with the most optimal technique being largely dependent on the task at hand. We will take the following route in our study of this topic:

• Detect moving objects based on differences between the current frame and a frame that represents the background. First, we will try a simple implementation of this approach. Then, we will use OpenCV's implementations of more advanced algorithms, namely, the Mixture of Gaussians...

Technical requirements

This chapter uses Python, OpenCV, and NumPy. Please refer to Chapter 1, Setting Up OpenCV, for installation instructions.

The complete code and sample videos for this chapter can be found in this book's GitHub
repository, https://github.com/PacktPublishing/Learning-OpenCV-4-Computer-Vision-with-Python-Third-Edition, in the chapter08 folder.

Detecting moving objects with background subtraction

To track anything in a video, first, we must identify the regions of a video frame that correspond to moving objects. Many motion detection techniques are based on the simple concept of background subtraction. For example, suppose that we have a stationary camera viewing a scene that is also mostly stationary. In addition to this, suppose that the camera's exposure and the lighting conditions in the scene are stable so that frames do not vary much in terms of brightness. Under these conditions, we can easily capture a reference image that represents the background or, in other words, the stationary components of the scene. Then, any time the camera captures a new frame, we can subtract the frame from the reference image, and take the absolute value of this difference in order to obtain a measurement of motion at each pixel...

Tracking colorful objects using MeanShift and CamShift

We have seen that background subtraction can be an effective technique for detecting moving objects; however, we know that it has some inherent limitations. Notably, it assumes that the current background can be predicted based on past frames. This assumption is fragile. For example, if the camera moves, the entire background model could suddenly become outdated. Thus, in a robust tracking system, it is important to build some kind of model of foreground objects rather than just the background.

We have already seen various ways of detecting objects in Chapter 5, Detecting and Recognizing Faces, Chapter 6, Retrieving Images and Searching Using Image Descriptors, and Chapter 7, Building Custom Object Detectors. For object detection, we favored algorithms that could deal with a lot of variation within a class of objects, so that...

Finding trends in motion using the Kalman filter

The Kalman filter is an algorithm developed mainly (but not exclusively) by Rudolf Kalman in the late 1950s. It has found practical applications in many fields, particularly navigation systems for all sorts of vehicles from nuclear submarines to aircraft.

The Kalman filter operates recursively on a stream of noisy input data to produce a statistically optimal estimate of the underlying system state. In the context of computer vision, the Kalman filter can smoothen the estimate of a tracked object's position.

Let's consider a simple example. Think of a small red ball on a table and imagine you have a camera pointing at the scene. You identify the ball as the subject to be tracked, and flick it with your fingers. The ball will start rolling on the table in accordance with the laws of motion.

If the ball is rolling at a...

Tracking pedestrians

Up to this point, we have familiarized ourselves with the concepts of motion detection, object detection, and object tracking. You are probably anxious to put this newfound knowledge to good use in a real-life scenario. Let's do just that by tracking pedestrians in a video from a surveillance camera.

You can find a surveillance video inside the OpenCV repository at samples/data/vtest.avi. A copy of this video is located inside this book's GitHub repository at chapter08/pedestrians.avi.

Let's lay out a plan and then implement the application!

Planning the flow of the application

The application will adhere to the following logic:

1. Capture frames from a video file.
2. Use the first 20 frames...

Summary

This chapter has dealt with video analysis and, in particular, a selection of useful techniques for tracking objects.

We began by learning about background subtraction with a basic motion detection technique that calculates frame differences. Then, we moved on to more complex and efficient background subtraction algorithms – namely, MOG and KNN – which are implemented in OpenCV's cv2.BackgroundSubtractor class.

We then proceeded to explore the MeanShift and CamShift tracking algorithms. In the course of this, we talked about color histograms and back-projections. We also familiarized ourselves with the Kalman filter and its usefulness in smoothing the results of a tracking algorithm. Finally, we put all of our knowledge together in a sample surveillance application, which is capable of tracking pedestrians (or other moving objects) in a video.

By now...