Coughlan Lab

L to R: Huiying Shen, Ali Cheraghi, Brandon Biggs, James Coughlan, Charity Pitcher-Cooper, Giovanni Fusco

Contact Information

Email:
coughlan@ski.org
Lab Phone:
(415) 345-2146
2318 Fillmore St.
San Francisco, CA 94115

The goal of our laboratory is to develop and test assistive technology for blind and visually impaired persons that is enabled by computer vision and other sensor technologies.

Read More +

Publications

Journal Articles

Hong, J., & Coughlan, J. (2023). VR Training to Facilitate Blind Photography for Navigation. Journal On Technology And Persons With Disabilities. (Original work published 06/2023 C.E.)
Crabb, R., Cheraghi, S. A., & Coughlan, J. (2023). A Lightweight Approach to Localization for Blind and Visually Impaired Travelers. Sensors, 23(5). https://doi.org/10.3390/s23052701 (Original work published 03/2023 C.E.)
Biggs, B., Pitcher-Cooper, C., & Coughlan, J. (2022). Getting in Touch with Tactile Map Automated Production: Evaluating Impact and Areas for Improvement. Journal On Technology And Persons With Disabilities, 10. (Original work published 05/2022 C.E.)
Ahmetovic, D., Manduchi, R., Coughlan, J., & Mascetti, S. (2017). Mind your crossings: Mining GIS imagery for crosswalk localization. Acm Transactions On Accessible Computing (Taccess), 9(4). 10.1145/3046790
Coughlan, J., & Shen, H. (2013). Crosswatch: a system for providing guidance to visually impaired travelers at traffic intersections. Journal Of Assistive Technologies, 7.
Tekin, E., Vásquez, D., & Coughlan, J. (2013). S-K Smartphone Barcode Reader for the Blind. Journal On Technology And Persons With Disabilities, 1.
Manduchi, R., & Coughlan, J. (2012). (Computer) vision without sight. Communications Of The Acm, 55, 96–104.
Shen, H., Coughlan, J., & Ivanchenko, V. (2009). Figure-ground segmentation using factor graphs. Image And Vision Computing, 27, 854–863.
Yuille, A. L., Coughlan, J., & Konishi, S. (2003). The KGBR viewpoint-lighting ambiguity. Journal Of The Optical Society Of America (Josa) A, 20(1). (Original work published 01/2003 C.E.)
Shen, H., Chan, K. -Y., Coughlan, J., & Brabyn, J. (2008). A mobile phone system to find crosswalks for visually impaired pedestrians. Technology And Disability, 20, 217–224.

Conference Papers

Biggs, B., Toth, C., Stockman, T., Coughlan, J., & Walker, B. N. (2022). Evaluation of a Non-Visual Auditory Choropleth and Travel Map Viewer. International Conference on Auditory Display (ICAD) 2022. Presented at the. Virtual. Retrieved de PMCID: PMC10010675 (Original work published 07/2022 C.E.)
Coughlan, J., Biggs, B., & Shen, H. (2022). Non-Visual Access to an Interactive 3D Map. Joint International Conference on Digital Inclusion, Assistive Technology & Accessibility (ICCHP-AAATE '22). Presented at the. (Original work published 07/2022 C.E.)
Coughlan, J., Biggs, B., & Shen, H. (2021). Point and Listen: Bringing a 3D Map to Life with Audio-Based AR. 6th Annual Frameless XR Symposium. Presented at the. (Original work published 11/2021 C.E.)
Biggs, B., Coughlan, J., & Coppin, P. (2021). Design and evaluation of an interactive 3D map. RESNA 2021 Conference. Presented at the. (Original work published 07/2021 C.E.)
Coughlan, J., Biggs, B., Rivière, M. -A., & Shen, H. (2020). An Audio-Based 3D Spatial Guidance AR System for Blind Users. 17th International Conference on Computers Helping People with Special Needs. Presented at the. (Original work published 09/2020 C.E.)
Fusco, G., Cheraghi, S. A., Neat, L., & Coughlan, J. (2020). An Indoor Navigation App using Computer Vision and Sign Recognition. 17th International Conference on Computers Helping People with Special Needs. Presented at the. (Original work published 09/2020 C.E.)
Fusco, G., & Coughlan, J. (2020). Indoor Localization for Visually Impaired Travelers Using Computer Vision on a Smartphone. 17th International Web for All Conference: Automation for Accessibility. Presented at the. (Original work published 04/2020 C.E.)
Biggs, B., Coughlan, J., & Coppin, P. (2019). Design and Evaluation of an Audio Game-Inspired Auditory Map Interface. The 25th International Conference on Auditory Display (ICAD 2019). Presented at the. Northumbria University, Newcastle-upon-Tyne, UK. (Original work published 06/2019 C.E.)
Fusco, G., & Coughlan, J. (2018). Indoor Localization using Computer Vision and Visual-Inertial Odometry. International Conference on Computers Helping People with Special Needs (ICCHP '18). Presented at the. Linz, Austria. (Original work published 07/2018 C.E.)
Coughlan, J., & Miele, J. (2017). AR4VI: AR as an Accessibility Tool for People with Visual Impairments. AR for Good, part of ISMAR 2017. Presented at the. Nantes, France: IEEE. (Original work published 10/2017 C.E.)

Projects

Active

Magic Map

The Magic Map is an interactive 3D map installed at the Magical Bridge Playground in Palo Alto, California. It consists of a 1/100 scale 3D bronze representation of the playground, which includes over seventy play structures organized into multiple play zones and paths. When the tip of the "Magic Wand" tethered to the map is pointed to a specific feature on the map, the name and description of the feature are read aloud in audio. This interactivity makes the map accessible to visitors with visual impairments, and without requiring them to read braille.

ZoomBoard: an Affordable, Portable System to Improve Access to Presentations and Lecture Notes for Low Vision Viewers

The goal of the project is to develop a “ZoomBoard” system that students with low vision can use to better access visual material on a whiteboard or blackboard. The prototype version of the system that we plan to develop in this grant will consist of a dedicated camera system placed by the teacher to capture a view of the board, which wirelessly transmits a video stream that will be displayed on a student’s iPad. The student will use the ZoomBoard app to view this video stream, zoom in on any region of interest using a pinch gesture on the iPad, and apply image enhancements such as contrast…

Computer Vision Journal Club

The Computer Vision Journal Club meets periodically to discuss papers on topics in computer vision, machine learning and other topics of interest such as assistive technologies for persons who are blind or visually impaired, dual sensory loss (hearing and vision loss), neuroscience and psychophysics. All are welcome to attend.

2018

Fri Mar 23, 2018
Jason Jo, Yoshua Bengio. Measuring the tendency of CNNs to Learn Surface Statistical Regularities. Link: https://arxiv.org/abs/1711.11561

Thurs Feb 15, 2018
Tulsiani, Gupta, Fouhey, Efros, Malik. Factoring Shape, Pose, and Layout from…

CamIO

CamIO (short for “Camera Input-Output”) is a system to make physical objects (such as documents, maps, devices and 3D models) accessible to blind and visually impaired persons, by providing real-time audio feedback in response to the location on an object that the user is touching. CamIO currently works on iOS using the built-in camera and an inexpensive hand-held stylus, made out of materials such as 3D-printed plastic, paper or wood.

See a video demonstration of CamIO here, showing how the user can trigger audio labels by pointing a stylus at "hotspots" on a 3D map of a playground. See the Ma…

Completed

Sign Finder

This project seeks to develop a computer vision-based system that allows a visually impaired traveler to find and read informational signs, such as signs labeling office doors, streets, restrooms and Exit signs.

Link to open source code

Tactile Graphics Helper (TGH)

Tactile graphics use raised lines, textures, and elevations to provide individuals with visual impairments access to graphical materials through touch. Tactile graphics are particularly important for students in science, technology, engineering, and mathematics (STEM) fields, where educational content is often conveyed using diagrams and charts. However, providing a student who has a visual impairment with a tactile graphic does not automatically provide the student access to the graphic's educational content. Instead, the student may struggle to decipher subtle differences between textures or…

Workshop Series on Computer Vision and Sensor-Enabled Assistive Technology for Visual Impairment

Recent workshop:

Workshop on Environmental Sensing Technologies for Visual Impairment (ESTVI '13 in San Francisco)

ESTVI '13 focused on emerging technologies capable of sensing environmental features for applications in access technologies for persons with visual impairment, including low vision and blindness. The development of environmental sensing technologies (ESTs) and the study of their potential to support the activities of daily living for visually impaired persons is progressing at a rapid pace, and engages many disparate research fields, including computer vision, wearable sensors…

Display Reader

The goal of the Display Reader project is to develop a computer vision system that runs on smartphones and tablets to enable blind and visually impaired persons to read appliance displays. Such displays are found on an increasing array of appliances such as microwave ovens, thermostats and home medical devices.

Click here for Display Reader software download.

BLaDE

BLaDE (Barcode Localization and Decoding Engine) is an Android smartphone app designed to enable a blind or visually impaired user find and read product barcodes. The primary innovation of BLaDE, relative to most commercially available smartphone apps for reading barcodes, is that it provides real-time audio feedback to help visually impaired users find a barcode, which is a prerequisite to being able to read it.

Link to BLaDE software download: http://legacy.ski.org/Rehab/Coughlan_lab/BLaDE/

Click here for YouTube video demo of BLaDE in action.

 

Crosswatch

Crosswatch is a smartphone-based system developed for providing real-time guidance to blind and visually impaired travelers at traffic intersections. Using the smartphone's built-in camera and other sensors, Crosswatch is designed to tell blind and visually impaired travelers what kind of traffic intersection they are near, how to align themselves properly to the crosswalk, and when the walk light or other traffic light indicates it is time to cross.

Click here for link to zip file containing Crosswatch code (open source) and documentation.

Talking Signs

Created by William Loughborough in 1979, Talking Lights was a system of infrared transmitters and receivers allowing blind and visually impaired travelers to quickly and easily "read signs" at a distance.

Centers

Video Description Research and Development Center

The Smith-Kettlewell Video Description Research and Development Center (VDRDC) investigates innovative technologies and techniques for making online video more accessible to blind and visually-impaired students and consumers. Through collaboration with a broad array of partners and stakeholders in the Description Leadership Network, we are developing advanced video annotation methods for use in a wide variety of educational settings, as well as helping educators and other description providers make better use of the tools already available.

People

Current

Past

Ender Tekin

Ender Tekin

Associate Scientist at The University of Wisconsin-Madison
Photo of Giovanni Fusco

Giovanni Fusco

Engineering Manager and Lead Machine Learning Eng., Pixofarm
Photograph of Laura Walker

Laura Walker

Senior Visual Experience Engineer, Apple, Cupertino CA
Photo of Pannag Sanketi

Pannag Sanketi

Tech Lead Manager/Senior Staff Software Engineer at Google
Photo of Vidya Murali

Vidya Murali

Technical Expert in Applied Machine Learning at Ford Research

News

SKERI Receives Rehabilitation Engineering Research Center (RERC) grant on Blindness and Low Vision

Smith-Kettlewell is proud to announce the newly awarded Rehabilitation Engineering Research Center (RERC) grant on Blindness and Low Vision. This is a five-year grant from the National Institute on Disability, Independent Living, and Rehabilitation Research, establishing Smith-Kettlewell as a center promoting the independence and well-being of people with visual impairments through research and development to improve the understanding of, and provide solutions for, challenges facing the blind and low-vision community.

Audiom allows for accessible, virtual visits to the Magical Bridge playground

The Magical Bridge Foundation mission is to bring inclusive, accessive playgrounds that can be enjoyed by all children. For the first time, it is now possible for blind and visually impaired visitors to view a nonvisual map of the Magical Bridge Playground in Palo Alto, at home, directly from their web browser using Audiom. Audiom was developed by Smith-Kettlewell researcher, Brandon Biggs, is a map viewing tool for the visually impaired. It allows non-visual access to route, landmark, and survey knowledge -- critical information needed for navigation.

Brandon Biggs brings his ideas for accessibility to SKERI

We’re excited to welcome Brandon Biggs to the Smith-Kettlewell Eye Research Institute. Brandon arrived in August to collaborate with Dr. Coughlan, Dr. Shen and Dr. Fusco to work on accessibility research and development. This collaboration started last year, when Brandon met Dr. Coughlan at the Association for Education and Rehabilitation of the Blind and Visually Impaired (AER) International Conference in Reno, Nevada. Brandon was then enrolled in a Master’s program in Inclusive Design at OCAD University in Toronto, Canada, which he completed in June.

Scientists Receive NEI Grant Aiding Blind Interaction with Physical Objects

James Coughlan, PhD, Senior Scientist at the Smith-Kettlewell Eye Research Institute, San Francisco, California, was recently awarded a four-year grant from NIH-NEI (R01EY025332) entitled, “Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired Summary".