Scientists Receive NEI Grant Aiding Blind Interaction with Physical Objects

Photo of CamIO ("Camera Input-Output") system

James M. Coughlan, PhD, Senior Scientist at the Smith-Kettlewell Eye Research Institute, San Francisco, California, was recently awarded a four-year grant from NIH-NEI (R01EY025332) entitled, “Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired Summary,” which will support research and development of the CamIO (“Camera Input-Output”) project.    This project is designed to help blind students explore, learn about, and interact with complex two-dimensional (2D) and three-dimensional (3D) shapes and objects.  CamIO was conceived by Dr. Joshua Miele, a blind scientist with extensive experience in developing and evaluating audio/haptic interfaces for access to spatial information by people with visual disabilities.  Other Smith-Kettlewell researchers working on the project are Dr. Donald C. Fletcher and Dr. Huiying Shen.

Present computer vision-based projects for accessing physical objects have been limited to flat (2D) documents and display screens.  By contrast, CamIO is designed to provide access to general 3D objects, including relief maps, 3D models (e.g., anatomical models used in medicine or biology courses), appliances/devices used at home or work (e.g., microwave ovens, glucometers) as well as flat documents. The CamIO system requires a minimum of hardware (an inexpensive mounted camera and computer or smartphone or tablet), and no modifications are needed to make an object accessible to it.  In addition to providing audio or haptic feedback about hotspots that the user is pointing to, CamIO also has the potential to support a rich set of interactions with the object using multiple finger/hand gestures (e.g., circling a hotspot could trigger one type of action for a hotspot while double-clicking it would trigger another).

Compared with other approaches to making objects accessible to people who are blind or visually impaired, CamIO has several advantages: (a) there is no need to modify or augment existing objects (e.g., with Braille labels or special touch-sensitive buttons); (b) CamIO is accessible even to those who are not fluent in Braille; and (c) it permits natural exploration of the object with all fingers.

The project has the potential to have wide ranging impact on access to graphics, tactile literacy, STEM education, independent travel and wayfinding, access to devices, and other applications to increase the independent functioning of blind, low vision and deaf-blind individuals.  Blind and low vision persons will test the system on an ongoing basis to maximize the system’s effectiveness and ease of use.