Seeing Behind the Camera

Imagine sitting in a room with a single open window. Across from this window is a white wall. Why isn’t an image of the outside world projected onto the wall? After all, the room and window are just a larger version of a camera (the window is the aperture and the wall is the sensor). The outside world is in fact imaged onto the wall — it is just very blurry. With some clever tricks, however, this image can be reconstructed, providing a view of the scene completely outside of the image frame.

At its most basic level, a photographic image is formed when light passes through an aperture and is focused onto a sensor. The traditional film or digital camera is, of course, the most common and convenient implementation of this process. This basic arrangement, however, can occur in many unexpected places. For example, an image of a solar eclipse can appear on a sidewalk when gaps in a tree canopy, acting as an aperture, focus the light onto the sidewalk. Or, an entire room can be converted into a camera by sealing the room from all light except for a small aperture at a window. In each case, a focused image is formed because the aperture is relatively small.

When the aperture is large, as in the case of a open window, the image formed is very blurry. Consider, however, the following scenario. The image shown in the top left is of a room with an open window. Shown in the bottom left is an image taken with somone standing in front of the window. If you look closely you will see that the pattern of light on the door is slightly different. The difference between these two images produces the same image as would have been achieved with an aperture that was the size and shape of just the person — that is, a much smaller aperture than the entire window. This effectively smaller aperture has the affect of producing a less blurry image. By simply differencing these two images and applying some simple processing (inverting to produce an upright image, removing noise, and enhancing contrast), the scene outside of the window can be reconstructed. (See “Accidental pinhole and pinspeck cameras: revealing the scene outside the picture” by Torralba and Freeman for complete technical details.)

[Photo credit: Antonio Torralba & William T. Freeman, reproduced with permission]

Shown in the upper right panel is the reconstructed scene of the outside of the window. For comparison the scene from outside the window is shown in the lower right. While the reconstruction is of quite low quality, it does give some general sense of the outside world.

This very clever technique is only applicable when presented with two images (or a video) of a room with some variation in the occlusion of the window. Although somewhat limited, this technique manages to perform the seemingly impossible — reconstructing part of the scene that is completely outside of the image frame. 

Comments

[Ryan]

Very cool. And like you said, very clever. I enjoy seeing technical experts perform "magic".

[Spoo]

Pull back, Wait a minute, Go right, Stop Enhance 5719. Track 45 left, Stop Enhance 15 to 23. Give me a hard copy right there.