EyeTap: The eye itself as display and camera

EyeTap is a device which allows, in a sense, the eye itself to function as both a display and a camera. EyeTap is at once the eye piece that displays computer information to the user and a device which allows the computer to process and possibly alter what the user sees. That which the user looks at is processed by the EyeTap. This allows the EyeTap to, under computer control, augment, diminish, or otherwise alter a user's visual perception of their environment, which creates a Computer Mediated Reality. Furthermore, ideally, EyeTap displays computer-generated information at the appropriate focal distance, and tonal range. The figure below depicts and describes the basic functional principle of EyeTap. Note from the diagram that the rays of light from the environment are collinear with the rays of light entering the eye (denoted by the dotted lines) which are generated by a device known as the aremac. "aremac" is the word camera spelled backwards and is the device which generates a synthetic ray of light which is collinear with an incoming ray of light. Ideally, the aremac will generate rays of light to form an image which appears to be spatially aligned, and appears at with the same focus as the real world scene.

Fundamental operating principle of EyeTap: Rays of light which would have otherwise entered the eye are instead reflected by the diverter. The diverter is typically a double-sided mirror or a beamsplitter. These rays of light are collected by a sensor, typically a CCD camera. This camera data is processed, and the aremac redisplays the image as rays of light. The aremac (the word `camera' backwards), is a display device which is capable of displaying information at an appropriate depth. These rays reflect again off the diverter, and are then collinear with the rays of light from the scene. Thus the rays of light hitting the sceneward side of the diverter in a sense "pass through" the diverter by passing through the processor and aremac. The user perceives the virtual light. This virtual light can be either the same image, or a computer mediated version of the real world scene, since the virtual light is altered under computer control.

EyeTap differs from head-mounted-displays (HMDs) because HMDs are typically used only to provide information, or add information into what a user perceives. HMDs with head-trackers have been used to spatially register virtual information with the real world to create Augmented Reality (AR). Roland discusses a variety of HMD designs, and classifies AR display devices into two categories: Video and Optical. Optical see-through HMDs use partially silvered mirrors to optically combine computer-generated information with the real world view. These display types are similar to the display half of the EyeTap design. However, optical see-through HMDs can only add in information, since the partially silvered mirrors do not block out any portions of the real world; that is, computer-generated information does not occlude real world objects (except in some specific configurations where the virtual signal was displayed much brighter than light from the scene).

In a Computer Mediated Reality, images from a camera are seen by the user, and, by altering the video, real world objects can be completely occluded by computer-generated information. The Video see-through HMD system classification in Roland Fuchs describes systems which display video from a head-mounted camera to the user. However, the Video see-through HMD systems do not necessarily place the camera at the centre of projection of the lens of an eye of a wearer. Rather, the cameras are placed close to the eyes of a wearer, and the wearer perceives only the video images. The Video HMD system of Edwards' Video see-through presents one such system. The displacement of the cameras caused mismatch between the user's normal perception of the world, and what was seen through the Video HMD Roland's virtual eyes.

By contrast, in EyeTap devices, the diverter (which may be a double-sided mirror or a beamsplitter) places the centre of projection of the camera at (in an optical sense) the centre of projection of the lens of an eye of the wearer. When no computer mediation is used, EyeTap video can be displayed to the user in such a way that the user perceives what he/she would otherwise have in the absence of the device. EyeTap mediates a portion of the user's vision, in such a way that it is integrated with the un-mediated portion of the user's field of view, without any mismatch between the mediated area and the real world.