Chapter 6. Detecting Foreground and Background Regions and Depth with a Kinect Device

In the field of video security applications, one often needs to notice the differences between frames because that's where the action happens. In other fields, it is also very important to isolate the objects from the background. This chapter shows several techniques to achieve this goal, comparing their strengths and weaknesses. Another completely different approach for detecting foreground or background regions is using a depth device like a Kinect. This chapter also deals with how to accomplish this goal with this device.

In this chapter, we will be covering:

• Background subtraction

• Frame differencing

• Averaging background method

• Mixture of Gaussian's method

• Contour finding

• Kinect depth maps

By the end of this chapter, you will have several approaches solving the problem of finding foreground/background regions, either through direct image processing or using a depth-compatible device such as a Kinect.

Background subtraction

When working with surveillance cameras, it's easy to see that most of the frame keeps still, while the moving objects, the ones we are interested in, are the areas that vary most over time. Background subtraction is defined as the approach used to detect moving objects from static cameras, also known as foreground detection, since we're mostly interested in the foreground objects.

In order to perform some valuable background subtraction, it is important to account for varying luminance conditions, taking care always to update our background model. Although some techniques extend the idea of background subtraction beyond its literal meaning, such as the mixture of Gaussian approach, they are still named like this.

In order to compare all the solutions in the following sections, we will come up with a useful interface, which is called VideoProcessor. This interface is made of a simple method called process. The whole interface is given in the following piece of code:

public...

Frame differencing

It should be straightforward to think of a simple background subtraction in order to retrieve foreground objects. A simple solution could look similar to the following line of code:

Core.absdiff(backgroundImage,inputImage , foregroundImage);

This function simply subtracts each pixel of backgroundImage from inputImage and writes its absolute value in foregroundImage. As long as we have initialized the background to backgroundImage and we have that clear from objects, this could work as a simple solution.

Here follows the background subtraction video processor code:

public class AbsDifferenceBackground implements VideoProcessor {
  private Mat backgroundImage;

  public AbsDifferenceBackground(Mat backgroundImage) {
    this.backgroundImage = backgroundImage;
  }

  public Mat process(Mat inputImage) {
    Mat foregroundImage = new Mat();
    Core.absdiff(backgroundImage,inputImage , foregroundImage);
    return foregroundImage;
  }

}

The main method, process, is really simple...

Averaging a background method

The problem with the background subtractor from the previous section is that the background will generally change due to illumination and other effects. Another fact is that the background may not be readily available, or the concept of background can change, for instance, when someone leaves a luggage in a video surveillance application. The luggage might be a foreground object for the first frames, but afterwards, it should be forgotten.

An interesting algorithm to deal with these problems uses the running average concept. Instead of always using the first frame as a clear background, it will update it constantly by calculating a moving average of it. Consider the following equation, which will be executed, updating each pixel from the old average and considering each pixel from the recently acquired image:

Note that is the new pixel value; is the value of the average background at time t-1, which would be the last frame; is the new value for the background...

The mixture of Gaussians method

Although we can get very good results with the previous idea, some more advanced methods have been proposed in literature. A great approach, proposed by Grimson in 1999, is to use not just one running average, but more averages so that if a pixel fluctuates between the two orbit points, these two running averages are calculated. If it does not fit any of them, it is considered foreground.

Besides, Grimson's approach also keeps the variance of the pixels, which is a measure of how far a set of numbers is spread out, taken from statistics. With a mean and a variance, a Gaussian model can be calculated and a probability can be measured to be taken into consideration, yielding a Mixture of Gaussians model (MOG). This can be very useful when branches and leaves are moving in the background.

Unfortunately, Grimson's method suffers from slow learning in the beginning and it can not distinguish between the moving shadows and moving objects. Therefore, an improved technique...

Contour finding

When dealing with the binary images removed from the background, it is important to transform pixels into useful information, such as by grouping them into an object or making it very clear for the user to see. In this context, it is important to know the concept of connected components, which are a set of connected pixels in a binary image, and OpenCV's function used to find its contours.

In this section, we will examine the findContours function, which extracts contours of connected components in an image as well as a helper function that will draw contours in an image, which is drawContours. The findContours function is generally applied over an image that has gone through a threshold procedure as well as some canny image transformation. In our example, a threshold is used.

The findContours function has the following signature:

public static void findContours(Mat image,
                java.util.List<MatOfPoint> contours,
                Mat hierarchy,
            ...

Kinect depth maps

From the beginning of this chapter until now, we have focused on the background subtraction approaches that try to model the background of the scene using ordinary cameras and then on applying frame differencing.

Note

Although the Kinect is reported to work with Linux and OSX, this section deals only with Windows setup on OpenCV 2.4.7 version.

In this section, we will take a different approach. We will set how far we want our objects to be considered foreground and background, which means removing the background by selecting a depth parameter. Unfortunately, this can not be done using a single ordinary camera in a single shot, so we will need a sensor that tells us the depth of objects or try to determine depth from stereo, which is not in the scope of this chapter. Thanks to both gamers and several efforts from all around the world, this device has become a commodity and it is called a Kinect. Some attempts can be made to use two cameras and try to get depth from stereo,...

Summary

This chapter has really covered several areas that deal with background removal as well as some details that arise from this problem, such as the need to use connected components to find their contours. Firstly, the problem of background removal itself was established. Then, a simple algorithm such as frame differencing was analyzed. After that, more interesting algorithms, such as averaging background and mixture of Gaussian (MOG) were covered.

After using algorithms to deal with background removal problems, an insight about connected components was explored. Core OpenCV algorithms such as findContours and drawContours were explained. Some properties of contours were also analyzed, such as their area as well as convex hulls.

The chapter finished with complete explanations of how to use the Kinect's depth sensor device as a background removal tool, for OpenCV 2.4.7. After depth instructions on the device setup, a complete application was developed, making it clear to deal with the...