The Story in Your Eyes

In an earlier post (“Photo forensics from eyes”) I described how the reflection of a light source on a person’s eye can be used to detect inconsistencies in lighting in an image. The reason why this works is that the surface of the eye is partially reflective and so the surrounding environment is reflected on the surface of the eye. Because the reflection of the light is typically very bright, it remains visible even in low resolution photos. If you have a high resolution image, it may be possible to take this approach one step further and reconstruct a fairly detailed image of what a person is looking at.

Though the surface of an eye behaves somewhat like a mirror, the curvature of the eye significantly distorts the reflection. And, the eye is not a perfect reflector, so the reflection is reduced in overall contrast and quality. Because the shape and reflective properties of the human eye are well understood, however, these distortions can be modeled and removed. With a sufficiently high resolution image, this means that an image of what a person sees can be “peeled” from the surface of an eye (see “The World in an Eye“ by Nishino and Nayar for complete details).

Shown below, for example, are three images of a person’s eye and a reconstruction of what they are looking at. The reconstruction is of sufficient quality that it is even possible to read the large text on the computer screen that the person on the far left was reading.  

[Photo credit: Ko Nishino and Shree Nayar, reproduced with permission]

With the ever-increasing resolution of even lower-end cell phone cameras, this forensic technique could prove incredibly useful in a variety of forensic cases including most notably child abductions in which a victim is photographed. In such a situation, the victim’s eyes will reflect back to us an image of their surroundings and, most likely, an image of the person taking the picture.