Home / Labs / Coughlan Lab

Coughlan Lab

Principal Investigator:

Contact Information:

2318 Fillmore St.
San Francisco, CA 94115

The goal of our laboratory is to develop and test assistive technology for blind and visually impaired persons that is enabled by computer vision and other sensor technologies.

L to R: Huiying Shen, Ali Cheraghi, Brandon Biggs, James Coughlan, Charity Pitcher-Cooper, Giovanni Fusco

Publications
Projects
Centers
People
Collaborators
News
Events

Schematic showing a graph with shaded areas around modes and edges indicating possible navigation areas. The current intermediate target is a node enclosed in a circle.

S. Mascetti, Dr. Ahmetovic, G. Galimberti and J. Coughlan. “NavGraph: Enhancing Blind Travelers’ Navigation Experience and Safety.” Proc. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), Volume 9, Issue 3, Article No.: 117, Pages 1 – 27. 2025.

Manzoni, M. ., Mascetti, S. ., Ahmetovic, D. ., Crabb, R. ., & Coughlan, J. . (2025). MapIO: a Gestural and Conversational Interface for Tactile Maps. IEEE Access.

Biggs, B. ., Coughlan, J. ., & Walker, B. (2025). Systematically Evaluating Digital Map Tools Based on the WCAG. Journal on Technology and Persons With Disabilities, 13.

Biggs, B. ., Agbaroji, H. ., Toth, C. ., Stockman, T. ., Coughlan, J. ., & Walker, B. . (2024). Co-Designing Auditory Navigation Solutions for Traveling as a Blind Individual During The COVID-19 Pandemic. Journal of Blindness Innovation and Research, 14(1). (Original work published 2024)

Hong, J. ., & Coughlan, J. . (2023). VR Training to Facilitate Blind Photography for Navigation. Journal On Technology And Persons With Disabilities. (Original work published 2023)

Crabb, R. ., Cheraghi, S. A., & Coughlan, J. . (2023). A Lightweight Approach to Localization for Blind and Visually Impaired Travelers. Sensors, 23(5). https://doi.org/10.3390/s23052701 (Original work published 2023)

Biggs, B. ., Pitcher-Cooper, C. ., & Coughlan, J. . (2022). Getting in Touch with Tactile Map Automated Production: Evaluating Impact and Areas for Improvement.Journal on Technology and Persons With Disabilities, 10. (Original work published 2022)

Tekin, E. ., Vásquez, D. ., & Coughlan, J. . (2013). S-K Smartphone Barcode Reader for the Blind. Journal on Technology and Persons With Disabilities, 1.

Coughlan, J. ., & Shen, H. . (2013). Crosswatch: a system for providing guidance to visually impaired travelers at traffic intersections. Journal of Assistive Technologies, 7.

Manduchi, R. ., & Coughlan, J. . (2012). (Computer) vision without sight. Communications of the ACM, 55, 96–104.

Shen, H. ., Coughlan, J. ., & Ivanchenko, V. . (2009). Figure-ground segmentation using factor graphs. Image and Vision Computing, 27, 854–863.

Coughlan, J. ., & Manduchi, R. . (2009). Functional assessment of a camera phone-based wayfinding system operated by blind and visually impaired users. International Journal on Artificial Intelligence Tools, 18, 379–397.

12 Next »

Biggs, B. ., Toth, C. ., Stockman, T. ., Tupy, S. ., Coughlan, J. ., & Walker, B. N. (2022). Audiom: an Auditory Web-Based Geographic Map Viewer Showing COVID-19 State Data and a Travel Map. (Original work published 2022)

Fusco, G. ., Tekin, E. ., & Coughlan, J. . (2016). An Appliance Display Reader for People with Visual Impairments.

Hand pointing to the wing of an airplane shown in a tactile graphic. The hand skeleton detected by computer vision is shown in green.

Ahmetovic, D., Coughlan, J., Dal Santo, G., Ezrouri, K., Manzoni, M., & Mascetti, S. (2025). CamIO in the Browser: A Cross-Platform Audio Label Tool for Tactile Graphics. MobileHCI 2025

Biggs, B. ., Toth, C. ., Stockman, T. ., Coughlan, J. ., & Walker, B. N. (2022). Evaluation of a Non-Visual Auditory Choropleth and Travel Map Viewer. International Conference on Auditory Display (ICAD) 2022. Presented at the International Conference on Auditory Display (ICAD) 2022. Virtual. Retrieved from PMCID: PMC10010675 (Original work published 2022)

Coughlan, J. ., Biggs, B. ., & Shen, H. . (2021). Point and Listen: Bringing a 3D Map to Life with Audio-Based AR. 6th Annual Frameless XR Symposium. Presented at the 6th Annual Frameless XR Symposium. (Original work published 2021)

Biggs, B. ., Coughlan, J. ., & Coppin, P. . (2021). Design and evaluation of an interactive 3D map. RESNA 2021 Conference. Presented at the RESNA 2021 Conference. (Original work published 2021)

Coughlan, J. ., Biggs, B. ., Rivière, M.-A. ., & Shen, H. . (2020). An Audio-Based 3D Spatial Guidance AR System for Blind Users. 17th International Conference on Computers Helping People With Special Needs. Presented at the 17th International Conference on Computers Helping People with Special Needs. (Original work published 2020)

Fusco, G. ., Cheraghi, S. A., Neat, L. ., & Coughlan, J. . (2020). An Indoor Navigation App using Computer Vision and Sign Recognition. 17th International Conference on Computers Helping People With Special Needs. Presented at the 17th International Conference on Computers Helping People with Special Needs. (Original work published 2020)

Biggs, B. ., Coughlan, J. ., & Coppin, P. . (2019). Design and Evaluation of an Audio Game-Inspired Auditory Map Interface. The 25th International Conference on Auditory Display (ICAD 2019). Presented at the The 25th International Conference on Auditory Display (ICAD 2019). Northumbria University, Newcastle-upon-Tyne, UK. (Original work published 2019)

Fusco, G. ., & Coughlan, J. . (2018). Indoor Localization using Computer Vision and Visual-Inertial Odometry. International Conference on Computers Helping People With Special Needs (ICCHP ’18). Presented at the International Conference on Computers Helping People with Special Needs (ICCHP ’18). Linz, Austria. (Original work published 2018)

Fusco, G. ., & Morash, V. S. (2015). The Tactile Graphics Helper: Providing Audio Clarification for Tactile Graphics Using Machine Vision. ASSETS. Presented at the ASSETS. (Original work published 2015)

Manduchi, R. ., & Coughlan, J. . (2014). The last meter: blind visual guidance to a target. Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, 3113–3122. ACM.

Murali, V. N., & Coughlan, J. . (2013). Smartphone-based crosswalk detection and localization for visually impaired pedestrians. Multimedia and Expo Workshops (ICMEW), 2013 IEEE International Conference on, 1–7. IEEE. (Original work published 2013)

Coughlan, J. ., & Shen, H. . (2012). The crosswatch traffic intersection analyzer: a roadmap for the future. Computers Helping People With Special Needs, 25–28. Springer Berlin Heidelberg.

12 Next »

Zaman, N., Potluri, V., Biggs, B., & Coughlan, M. J. (2025). WhatsAI: Transforming Meta Ray-Bans into an Extensible Generative AI Platform for Accessibility. arXiv preprint arXiv:2505.09823.

Shen, H. ., & Coughlan, J. . (2012). Towards a real-time system for finding and reading signs for visually impaired users. In Computers Helping People with Special Needs (pp. 41–47). Springer Berlin Heidelberg. (Original work published 2012)

Coughlan, J. . (2012). Markov random fields and techniques for performing inference with them.

Tekin, E. ., & Coughlan, J. . (2012). BLaDE: Barcode localization and decoding engine.

Coughlan, J. . (2011). Mechanisms for propagating surface information in 3-D reconstruction. In C. W. Tyler (Ed.), Computer Vision: From Surfaces to 3D Objects. Chapman and Hall/CRC Press.

Coughlan, J. . (2010). Maximum Entropy Distributions and Their Relationship to Maximum Likelihood.

Coughlan, J. . (2009). A Tutorial Introduction to Belief Propagation.

Coughlan, J. . (2009). Basic 3D Geometry for One and Two Cameras.

Coughlan, J. . (1999). A Transparent Interpretation of the EM Algorithm.

Projects

Active

Empowering Data Vision: A Self-Paced Introduction to the Convergence of AI and Data Science

Active

Audiom Map of Smith-Kettlewell

An Audiom map of the main Smith-Kettlewell building at 2318 Fillmore St., San Francisco has been created by Brandon Biggs. This is an audio-visual map that allows users to explore a detailed map of the building with or without vision. The map runs in any browser and is available to anyone who visits the building.

Active

MapIO: a Gestural and Conversational Interface for Tactile Maps

For individuals who are blind or have low vision, tactile maps provide essential spatial information but are limited in the amount of data they can convey. Digitally augmented tactile maps enhance these capabilities with audio feedback, thereby combining the tactile feedback provided by the map with an audio description of the touched elements. In this context, we explore an embodied interaction paradigm to augment tactile maps with conversational interaction based on Large Language Models, thus enabling users to obtain answers to arbitrary questions regarding the map. We analyze the type of…

Person with visual impairment using a walk light detector app at a traffic intersection. Left photo shows user holding smartphone, right photo shows image acquired by app of traffic intersection including walk light.

Active

Using VR to Help Train Visually Impaired Users to Aim a Camera

People with visual impairments increasingly rely on camera-enabled smartphone apps for tasks like photography, navigation, and text recognition. Despite the growing use of these applications, precise camera aiming remains a significant challenge. This project explores the impact of virtual reality (VR) exploration in the context of learning to use a camera-based app. So far we have studied this approach in the context of training a visually impaired person to use walk-light detector app at traffic intersections.

Close-up photo showing person pointing right index finger to Ireland on a map of the British Isles with other fingers closed.

Active

CamIO Hands

This project builds on the CamIO project to provide point-and-tap interactions allowing a user to acquire detailed information about tactile graphics and 3D models. The interface uses an iPhone’s depth and color cameras to track the user’s hands while they interact with a model. When the user points to a feature of interest on the model with their index finger, the system reads aloud basic information about that feature. For additional information, the user lifts their index finger and taps the feature again. This process can be repeated multiple times to access additional levels of…

Active

Magic Map

The Magic Map is an interactive 3D map installed at the Magical Bridge Playground in Palo Alto, California. It consists of a 1/100 scale 3D bronze representation of the playground, which includes over seventy play structures organized into multiple play zones and paths. When the user's index fingertip touches a specific feature on the map, the name and description of the feature are read aloud in audio. This interactivity allows visitors with visual impairments to navigate the map without requiring them to read braille.

Active

Outreach at Smith-Kettlewell

Smith-Kettelwell is deeply committed to supporting our community. Please see the news tab for our upcoming and ongoing events. In the News tab, you will find information about our involvement in past outreach activities.

Active

Audiom

Audiom is a tool that allows blind and visually impaired individuals to view maps completely in audio. It is a web component and can be embedded into any webpage, similar to Google Maps. It allows non-visual use of route, landmark, and survey knowledge, which is the critical information needed for navigation.

Active

Blindness and Low Vision Support Group

Join Dr. Don Fletcher, one of the world’s leading authorities on Low Vision Rehabilitation, to share experiences and learn about the things that help you maintain a full and happy life while living with low vision.

Active

A Computer Vision-Based Indoor Wayfinding Tool

The ability to navigate safely and confidently is a fundamental requirement for independent travel and access to many settings such as work, school, shopping, transit and healthcare. Navigation is particularly challenging for people with visual impairments, who have limited ability to see signs, landmarks or maps posted in the environment.

12 Next »

Completed

Crowd-Sourced Description for Web-Based Video (CSD)

The Descriptive Video Exchange Project (funded by the National Eye Institute of the National Institutes of Health, (grant # R01 EY020925-01) focuses on crowd-sourced techniques for describing DVD media. CSD will expand DVX to include Internet-based media such as YouTube, iTunes U, and other streamed video found on a wide variety of web sites. Many streamed Internet-based video sources provide well-defined, public APIs for accessing all the information DVX requires. Using these APIs will allow the VDRDC to expand DVX to include streamed content so that seamless, simple, crowd-sourced descriptions can be added to Internet-based video by volunteers or professionals anywhere.

Completed

Talking Signs

Created by William Loughborough in 1979, Talking Lights was a system of infrared transmitters and receivers allowing blind and visually impaired travelers to quickly and easily "read signs" at a distance.

Completed

ZoomBoard: an Affordable, Portable System to Improve Access to Presentations and Lecture Notes for Low Vision Viewers

The goal of the project is to develop a “ZoomBoard” system that students with low vision can use to better access visual material on a whiteboard or blackboard. The prototype version of the system that we plan to develop in this grant will consist of a dedicated camera system placed by the teacher to capture a view of the board, which wirelessly transmits a video stream that will be displayed on a student’s iPad. The student will use the ZoomBoard app to view this video stream, zoom in on any region of interest using a pinch gesture on the iPad, and apply image enhancements such as contrast…

Graphic of lines of braille with finger trace trajectory superimposed over braille

Completed

Regressions in Braille Reading

This project explores regressions (movements to re-read text) in braille reading. The image on the right plots the braille reading finger movements in blue and regressions in black.

Sample indoor sign showing automatic detection in yellow

Completed

Sign Finder

This project seeks to develop a computer vision-based system that allows a visually impaired traveler to find and read informational signs, such as signs labeling office doors, streets, restrooms and Exit signs. Link open source code

Completed

Tutorials and Reference

These are tutorials and reference materials I have written on various topics in probability and geometry over the years.

Completed

Tactile Graphics Helper (TGH)

Tactile graphics use raised lines, textures, and elevations to provide individuals with visual impairments access to graphical materials through touch. Tactile graphics are particularly important for students in science, technology, engineering, and mathematics (STEM) fields, where educational content is often conveyed using diagrams and charts. However, providing a student who has a visual impairment with a tactile graphic does not automatically provide the student access to the graphic's educational content. Instead, the student may struggle to decipher subtle differences between textures or…

Completed

Workshop Series on Computer Vision and Sensor-Enabled Assistive Technology for Visual Impairment

Recent workshop: Workshop on Environmental Sensing Technologies for Visual Impairment (ESTVI '13 in San Francisco) ESTVI '13 focused on emerging technologies capable of sensing environmental features for applications in access technologies for persons with visual impairment, including low vision and blindness. The development of environmental sensing technologies (ESTs) and the study of their potential to support the activities of daily living for visually impaired persons is progressing at a rapid pace, and engages many disparate research fields, including computer vision, wearable sensors…

Zoomed-in view of appliance display partially obscured by glare

Completed

Display Reader

The goal of the Display Reader project is to develop a computer vision system that runs on smartphones and tablets to enable blind and visually impaired persons to read appliance displays. Such displays are found on an increasing array of appliances such as microwave ovens, thermostats and home medical devices. Click here for Display Reader software download.

Completed

BLaDE

BLaDE (Barcode Localization and Decoding Engine) is an Android smartphone app designed to enable a blind or visually impaired user find and read product barcodes. The primary innovation of BLaDE, relative to most commercially available smartphone apps for reading barcodes, is that it provides real-time audio feedback to help visually impaired users find a barcode, which is a prerequisite to being able to read it. Link to BLaDE software download: http://legacy.ski.org/Rehab/Coughlan_lab/BLaDE/ Click here for YouTube video demo of BLaDE in action.

12 Next »