The Correpondence Problem

It is obvious that the depth can be perceived by binocular disparity based on the angles formed by objects and their images on the two retinas. However, when the scene is more complex than a single point, we have to make the assumption that prior to depth computation a one-to-one correspondence relationship between the two sets of points on the retinas is established, which may not be a trivial process. One suggested approach is to assume that the objects in the scene are first recognized before the corresponding parts or points can be identified and matched. The following three-step procedure for depth perception is just based on such an assumption:

• Analysis of the shape/recognition of the objects;
• Identification of corresponding parts (points, edges, etc.) of the objects on the two retinas;
• Depth perception of the object, by neurons tuned to different binocular disparities of the corresponding parts.

But this suggested procedure has some serious challenges. For example, as illustrated in the figure, two identical objects (e.g., points or vertical lines for simplicity) project two identical images on each of the retinas. However, the rays projected outward from these two pairs of images form four intersections, two of them are the actual objects, while the other two are ghost objects that do not exist. Or, equivalently, we have two possible solutions, one real, one ghost. But these ghosts are never seen. How does the visual system know what are real and what are not?

This correspondence problem is more generally illustrated by the random-dot experiment (Julesz 1971) as demonstrated by this web demo.