Reconstructing 3D Scenes

In the previous chapter, we learned how a camera captures a 3D scene by projecting light rays on a 2D sensor plane. The image produced is an accurate representation of what the scene looks like from a particular point of view in the instant that the image is captured. However, by its nature, the process of image formation eliminates all of the information concerning the depth of the represented scene elements. This chapter will examine how, under specific conditions, the 3D structure of the scene and the 3D pose of the cameras that captured it can be recovered. We will demonstrate how a good understanding of the concepts of projective geometry allows us to devise methods that enable 3D reconstruction. We will, therefore, revisit the principle of image formation that was introduced in the previous chapter; in particular, we will now take into consideration...

Digital image formation

Let's redraw a new version of the diagram shown in Chapter 10, Estimating Projective Relations in Images, describing the pinhole camera model. More specifically, we want to demonstrate the relationship between a point in 3D at its position (X, Y, Z) and its image (x, y) on a camera, specified in pixel coordinates:

Notice the changes that have been made to the original diagram. First, we added a reference frame that we positioned at the center of the projection. Second, we have the y axis pointing downward in order to ensure that the coordinate system is compatible with the usual convention that places the image origin in the upper-left corner of the image. Finally, we also identified a special point on the image plane—considering the line coming from the focal point is orthogonal to the image plane, then the point (u0, v0) is the pixel position...

Calibrating a camera

Camera calibration is the process by which the different camera parameters are obtained. You can obviously use the specifications provided by the camera manufacturer, but for some tasks, such as 3D reconstruction, these specifications are not accurate enough. Camera calibration works by showing known patterns to the camera and analyzing the obtained images. An optimization process will then determine the optimal parameter values that explain the observations. This is a complex process that has been made easy by the availability of OpenCV's calibration functions.

Getting ready

To calibrate a camera, you show it a set of scene points where the 3D positions are known. Then, you need to observe where...

Recovering the camera pose

When a camera is calibrated, it becomes possible to relate the captured images to the outside world. We previously explained that if the 3D structure of an object is known, then you can predict how the object will be projected on to the sensor of the camera. The process of image formation is described by the projective equation presented at the beginning of this chapter. When most of the terms of this equation are known, then it becomes possible to infer the value of the other elements (2D or 3D) through the observation of some images. In this recipe, we will look at the camera pose recovery problem when a known 3D structure is observed.

How to do it...

Let's consider a simple object—...

Reconstructing a 3D scene from calibrated cameras

In the previous recipe, we saw that it is possible to recover the position of a camera that is observing a 3D scene when the camera is calibrated. The approach that was described took advantage of the fact that, sometimes, the coordinates of some 3D points visible in the scene might be known. We will now learn that if a scene is observed from more than one point of view, a 3D pose and structure can be reconstructed even if no information about the 3D scene is available. This time, we will use correspondences between image points in the different views in order to infer 3D information. We will introduce a new mathematical entity encompassing the relationship between two views of a calibrated camera, and we will discuss the principle of triangulation in order to reconstruct 3D points from 2D images.

...

Computing depth from a stereo image

Humans view the world in three dimensions using their two eyes. Robots can do the same when they are equipped with two cameras; this is called stereo vision. A stereo rig is a pair of cameras mounted on a device, looking at the same scene and separated by a fixed baseline (that is, the distance between the two cameras). This recipe will demonstrate how a depth map can be computed from two stereo images by computing the depth correspondence between the two views.

Getting ready

A stereo vision system is generally made of two side-by-side cameras looking toward the same direction. The following diagram illustrates such a stereo system in a perfectly-aligned configuration:

Under this ideal...