Multisensory Information access and AI: Advancing opportunities for the Visually Impaired

Dr. Hari Palani Principal Researcher & CEO

Event Date:

Friday, July 19th, 2024 – 12:00 PM to 1:00 PM

Speaker:

Dr. Hari Palani Principal Researcher & CEO

Host:

James Coughlan

Abstract

Lack of timely access to information is a significant problem for the nearly 24 million blind or visually impaired (BVI) individuals in the U.S. and 285 million globally. Significant progress has been made in ensuring BVI individuals get the necessary accommodations they need. However, very little work has been done towards making critical materials accessible in real-time, especially for non-textual materials such as math expressions, graphical representations, and maps. Current methods for authoring, converting, and/or producing accessible versions of non-textual materials requires significant human, time, and financial resources. The process also requires people with rare and specialized knowledge such as braille transcribers and tactile graphic designers.
My research program is aimed at addressing these issues and is unified by two complementary strands of applied science, (1) multisensory information access, and (2) AI. In this talk, I will first introduce the notion of multisensory information access, with a focus on how it impacts the perception, cognition, and behavioral characteristics of humans. Then, using the findings as foundation, I will show how we can (and are) using AI to enable multisensory information access in educational and navigational settings to address the long-standing accessibility issues of BVI individuals. https://www.unarlabs.com/

Event Category

Scientific

Event type

Colloquium

Publications

A cartoon of two individuals walking along a street with thought bubbles imagining a fall and an ambulance for one and consternation about foot placement in the other.

Shanidze, N.M., Agathos, C.P., Ellmers, T.J., & Young, W.R., (2026). Links Between Central Visual Field Loss and Movement Processing during Walking. Gait & Posture, 110095.

(2026). The best stereoacuity is rarely at the fovea. Vision Research, 240, 108748. https://doi.org/https://doi.org/10.1016/j.visres.2025.108748

Guénot, J. & Verghese, P. (2026). Cue combination for depth perception in macular degeneration: Motion parallax augments disparity. Journal of Vision, 26(1), 11-11. https://doi.org/10.1167/jov.26.1.11

diagram of a head rotation that does not produce an eye counterrotation and instead the eyes are rotated to a new target by the head

Shanidze, N.M. & Velisar, A. (2025). Cancellation of the Vestibulo-Ocular Reflex during Smooth Pursuit in Patients with Maculopathy. Frontiers in Neurology, 16, 1632527.

Zhou, Z., Hou, C.; Attentional Eye Selection Affects Perception During Binocular Rivalry. Trans. Vis. Sci. Tech. 2025;14(10):26.

Verghese P, Chopin A, Gomes-Tomaz Â, Alcalde NG, Levi DM. (2025). Vergence anomalies are associated with impaired stereopsis in amblyopia. Vision Res. 237:108696. doi: 10.1016/j.visres.2025.108696.

Bonneh, Y. ., & Tyler, C. W. (2025). The Canonical Deep Neural Network as a Model for Human Symmetry Processing. IScience. (Original work published 2025)

Good, W. V., Wong, R. J., Norcia, A. ., Hou, C. ., Cellucci, J. ., McGovern, M. ., … Bhutani, V. K. (2025). Effect of bilirubin on visuocortical development in preterm infants. Journal of Perinatology. 10.1038/s41372-025-02213-4 (Original work published 2025)

Rubinstein, J. F., Alcalde, N. G., Chopin, A. ., & Verghese, P. . (2025). Oculomotor challenges in macular degeneration impact motion extrapolation. Journal of Vision, 25(1). 10.1167/jov.25.1.17

García-Lázaro, H. G., & Teng, S. . (2025). Novel Stimuli to Benchmark and Train Echolocation Skills. The Journal on Technology & Persons With Disabilities, 13. 20.500.12680/b2774511d (Original work published 2025)

Hartelius, G., Likova, T. L., Haldeman, D. D., Sester, M., Elfers, J., & Tyler, W. C. (2025). Evidence that Attentional Stance Shift is Learned Rapidly, Reliably Induces Mindfulness, and Constitutes a Distinctive Marker of Cognitive Process. Mindfulness, 16(5), 1258–1272.

Tyler, W. C., Mineff, N. K., Liang, M., & Likova, T. L. (2025). Spatiomotor dynamics of hand movements during the drawing of memory-guided trajectories without visual feedback. Journal of neurophysiology, 133(6), 1665–1674.

Overview of Robin, a wearable echolocation device designed to aid navigation and perception for people who are blind or have low vision. A mannequin wears the headset, which emits ultrasound pulses toward a chair, records the reflected echoes, and slows them down for playback in audible frequencies. A web app lets users customize the signal parameters.

Bhattacharyya, P., Tam, R., Reynolds, I., Orsmond, P., & Teng, S. (2025). Optimizing Signal Parameters to Enhance Echoacoustic Perception in Humans. Durham, UK.

García-Lázaro, H. G., & Teng, S. . (2025). The role of motor control in echolocation performance: self-initiated vs. passive listening for object spatial localization. Durham, UK. Retrieved from https://imrf2025.sciencesconf.org/resource/page/id/30

Agathos, C. P., Velisar, A. ., & Shanidze, N. . (2025). Examining gaze-driven postural adaptations in older adults with and without central visual field loss. Maastricht, Netherlands.

Krasovskaya, S. ., Coughlan, J. ., Patel, A. ., & Teng, S. . (2025). Modeling target search in blind echolocators using a Kalman Filter with realistic exploratory behavior simulations. Boston, MA.

Bhattacharyya, P. ., Tam, R. ., Reynolds, I. ., Orsmond, P. ., & Teng, S. . (2025). Optimizing Signal Parameters to Enhance Echoacoustic Perception of Objects and Scenes in Humans. Durham, UK.

García-Lázaro, H. G., & Teng, S. . (2025). Auditory masking release indexes human click-based echolocation performance. Boston, MA.

Shanidze, N. ., Velisar, A. ., & Agathos, C. P. (2025). Lifetime Noise Exposure is Associated with Greater Postural Sway during Quiet Stance.

Krasovskaya, S. ., Coughlan, J. ., & Teng, S. . (2025). A Kalman Filter model of echo-guided head movement. Durham, UK.

Agathos, C. P., Velisar, A. ., & Shanidze, N. . (2025). Postural Adaptations to Eccentric Viewing Strategies in Age-Related Central Visual Field Loss. Durham, UK.

Rosengen, B. ., Agathos, C. P., & Shanidze, N. . (2025). Fall-Related Ruminations are Linked to Spatial Orientation and Cognitive Flexibility. San Francisco, CA, USA.

Bhattacharyya, P. ., Tam, R. ., Orsmond, P. ., Hicks, S. ., & Teng, S. . (2025). Time-resolved EEG decoding of neural text representations during naturalistic braille reading. Boston, MA.

Velisar, A. ., Agathos, C. P., Telaprolu, D. S. L., & Shanidze, N. . (2025). Lifetime Noise Exposure Affects Age-Related Gait Changes. Maastricht, Netherlands.

Hand pointing to the wing of an airplane shown in a tactile graphic. The hand skeleton detected by computer vision is shown in green.

Ahmetovic, D., Coughlan, J., Dal Santo, G., Ezrouri, K., Manzoni, M., & Mascetti, S. (2025). CamIO in the Browser: A Cross-Platform Audio Label Tool for Tactile Graphics. MobileHCI 2025

Pitcher-Cooper, C. ., Ihorn, S. ., & Yoon, I. . (2024). Access granted: Unlocking accessibility in computer science through collaboration. ASSETS 2024 Workshop. Presented at the ASSETS 2024 Workshop. Association for Computing Machinery. (Original work published 2024)

Narcisi, A. ., Shen, H. ., Ahmetovic, D. ., Mascetti, S. ., & Coughlan, J. . (2024). Accessible Point-and-Tap Interaction for Acquiring Detailed Information about Tactile Graphics and 3D Models. International Conference on Computers Helping People With Special Needs. Presented at the International Conference on Computers Helping People with Special Needs. Linz, Austria. Retrieved from https://link.springer.com/chapter/10.1007/978-3-031-62846-7_30 (Original work published 2024)

Seo, J. ., O’Modhrain, S. ., Xia, Y. ., Kamath, S. ., Lee, B. ., & Coughlan, J. . (2024). Born Accessible Data Science and Visualization Courses: Challenges of Developing Curriculum to be Taught by Blind Instructors to Blind Students. EUROVIS 2024. Presented at the EUROVIS 2024. https://doi.org/10.2312/eved.20241053 (Original work published 2024)

Hong, J. ., & Coughlan, J. . (2024). Enhancing Walk-Light Detector Usage for the Visually Impaired: A Comparison of VR Exploration and Verbal Instructions. Web4All 2024. Presented at the Web4All 2024. Singapore. (Original work published 2024)

Scott, A. ., Narins, L. ., Kulkarni, A. ., Castanon, M. ., Kao, B. ., Ihorn, S. ., … Yoon, I. . (2023). Improved Image Caption Rating – Datasets, Game, and Model. 2023 CHI Conference on Human Factors in Computing Systems. Presented at the 2023 CHI Conference on Human Factors in Computing Systems. Hamburg, Germany: Association for Computing Machinery. 10.1145/3544549.3585632 (Original work published 2023)

Biggs, B. ., Toth, C. ., Stockman, T. ., Coughlan, J. ., & Walker, B. N. (2022). Evaluation of a Non-Visual Auditory Choropleth and Travel Map Viewer. International Conference on Auditory Display (ICAD) 2022. Presented at the International Conference on Auditory Display (ICAD) 2022. Virtual. Retrieved from PMCID: PMC10010675 (Original work published 2022)

Coughlan, J. ., Biggs, B. ., & Shen, H. . (2022). Non-Visual Access to an Interactive 3D Map. Joint International Conference on Digital Inclusion, Assistive Technology & Accessibility (ICCHP-AAATE ’22). Presented at the Joint International Conference on Digital Inclusion, Assistive Technology & Accessibility (ICCHP-AAATE ’22). (Original work published 2022)

Sarker, P. ., Zaman, N. ., & Tavakkoli, A. . (2022). Vr-sft: Reproducing swinging flashlight test in virtual reality to detect relative afferent pupillary defect. 193-204. Springer Nature Switzerland Cham.

Coughlan, J. ., Biggs, B. ., & Shen, H. . (2021). Point and Listen: Bringing a 3D Map to Life with Audio-Based AR. 6th Annual Frameless XR Symposium. Presented at the 6th Annual Frameless XR Symposium. (Original work published 2021)

Biggs, B. ., Coughlan, J. ., & Coppin, P. . (2021). Design and evaluation of an interactive 3D map. RESNA 2021 Conference. Presented at the RESNA 2021 Conference. (Original work published 2021)

Binaee, K. ., Sinnott, C. B., Capurro, K. ., MacNeilage, P. R., & Lescroart, M. . (2021). Pupil tracking under direct sunlight. 2021 ACM Syposium on Eye Tracking Research and Applications ActivEye Workshop. Presented at the 2021 ACM Syposium on Eye Tracking Research and Applications ActivEye Workshop. Germany: Association for Computing Machinery. (Original work published 2021)

a picture describing an experimental setup where a robot bird is shown on a platform waiting for a live bird to land on a perch to start a birdsong

Araguas, A., Valencia, D., Chopin, A., Thomas, A., Gauthier, P., Richer, F., Guellai, B., & Deregnaucourt, S. (2026). Influence of behavioural contingency on developmental song learning in young zebra finches (Taeniopygia guttata) tutored by a robot bird. Proceedings of the Royal Society B: Biological Sciences, 293(2066), 20252834. https://doi.org/10.1098/rspb.2025.2834

Zaman, N., Potluri, V., Biggs, B., & Coughlan, M. J. (2025). WhatsAI: Transforming Meta Ray-Bans into an Extensible Generative AI Platform for Accessibility. arXiv preprint arXiv:2505.09823.

diagram if the cameras directed at robotic eyes with labels of individual components and placement in 3D space

Shanidze, M. N., Velisar, A., Lotze, A. W., & Love, K. K. (2025). Oculomotor simulator for assessing eye-tracking accuracy. US Patent 12,474,771

Likova, T. L., Mineff, N. K., Zhou, Z., Liang, M., & Tyler, W. C. (2025). Training-induced Reorganization of the Cross-modal Connectivity of the Human Motion Area in Severe Low Vision. Electronic Imaging, 37, 1–8.

Zaman, N. . (2024). Extending Vision with Extended Reality. University of Nevada, Reno.

Mukhtar, A. ., Henderson, K. ., Bourquin, S. ., Teng, S. ., & Puri, A. . (2024). Exploring Object Properties that Support the Transfer of Information Across Active Echolocation and Touch. New York, NY.

Child-robot interactions in different body and emotions oriented tasks: comparison with a human partner

Waisberg, E. ., Ong, J. ., Zaman, N. ., Tavakkoli, A. ., & Lee, A. . (2023). The Impact of COVID-19 on Ophthalmology Clinic and Surgical Volume. Ir Med J, 116, P884.

Waisberg, E. ., Ong, J. ., Masalkhi, M. ., Zaman, N. ., Sarker, P. ., Lee, A. G., & Tavakkoli, A. . (2023). GPT-4 and medical image analysis: Strengths, weaknesses and future directions. Journal of Medical Artificial Intelligence, 6.

Waisberg, E. ., Ong, J. ., Masalkhi, M. ., Zaman, N. ., Sarker, P. ., Lee, A. G., & Tavakkoli, A. . (2023). Chat Generative Pretrained Transformer to optimize accessibility for cataract surgery postoperative management. The Pan-American Journal of Ophthalmology, 5, 46.

Zaman, N. ., Sarker, P. ., & Tavakkoli, A. . (2023). Calibration of head mounted displays for vision research with virtual reality. Journal of Vision, 23, 7-7.

Waisberg, E. ., Ong, J. ., Masalkhi, M. ., Zaman, N. ., Kamran, S. A., Sarker, P. ., … Lee, A. G. (2023). The case for expanding visual assessments during spaceflight. Prehospital and Disaster Medicine, 38, 518-521.