AI-Based Multisensory Aid for the Blind and Deaf

Abstract

Individuals who are blind and deaf face major challenges in performing daily activities such as navigation, communication, reading printed text, and identifying surround- ing objects. Most existing assistive technologies are designed to support only a single sensory impairment and rely heavily on audio-based feedback, which is ineffective for users with hearing loss. This paper presents an AI-based multisensory assistive system developed to support individuals with dual sensory im- pairments. The proposed system integrates computer vision and machine learning techniques including object detection, currency detection, sign language recognition, and Optical Character Recognition (OCR). Visual inputs captured through a camera are processed using deep learning models, and the extracted information is conveyed to the user through tactile feedback using vibration alerts and Braille-based output. The system aims to improve independence, safety, and accessibility in everyday activities.

Introduction

The AI-Based Multisensory Aid for the Blind and Deaf is an integrated assistive system designed to improve independence, safety, and accessibility for individuals with combined visual and auditory impairments. Addressing limitations of existing assistive technologies—which often rely heavily on audio feedback and focus on single tasks—the proposed solution provides a unified, non-auditory–friendly framework that supports multiple daily-life activities.

The system uses a camera and a laptop as the central processing platform to capture and analyze real-time visual data using artificial intelligence and computer vision techniques. Core functionalities include object detection, currency recognition, sign language recognition, and Optical Character Recognition (OCR). YOLOv8 enables real-time object and currency localization, while CNN models classify currency denominations and recognize sign language gestures. MediaPipe is used for accurate hand landmark detection in sign language recognition, and Tesseract OCR extracts text from printed materials. Outputs are delivered through text, audio via bone conduction earphones, and optional haptic feedback, ensuring accessibility without blocking environmental awareness.

The literature review highlights that most existing systems address visual or auditory impairments separately, rely on audio-based feedback, and operate as standalone solutions. There is limited research on integrated multisensory systems tailored for deaf-blind users. This gap motivates the proposed unified, laptop-based framework that combines multiple assistive functions into a single, flexible platform.

Experimental results demonstrate that the system performs reliably in real-time indoor environments, with accurate object detection, currency recognition, sign language interpretation, and text extraction. The modular architecture allows efficient processing, easy enhancement, and scalability. Overall, the proposed system offers a practical and comprehensive assistive solution that enhances quality of life and independence for blind and deaf individuals by leveraging AI-driven technologies in a single, user-friendly framework.

Conclusion

This paper presented an AI-based multisensory assistive system designed to support blind and deaf individuals in performing everyday activities with greater independence and confidence. The proposed system integrates multiple computer vision and deep learning modules, including object detection, currency recognition, sign language interpretation, and Optical Character Recognition, into a unified laptop-based processing framework. By combining these functionalities, the system ad- dresses several accessibility challenges within a single assistive solution. The integration of advanced models such as YOLOv8, Convolutional Neural Networks, and MediaPipe enables the system to interpret complex visual information in real time. Object detection enhances environmental awareness by iden- tifying surrounding objects and obstacles, while currency recognition assists users during financial transactions. The sign language recognition module enables effective communication by converting hand gestures into audible output, and the OCR module allows users to access printed textual information independently. Together, these components demonstrate the effectiveness of artificial intelligence in bridging sensory limitations. A key strength of the proposed system is its wearable- oriented design and the use of bone conduction earphones for audio feedback. Bone conduction technology allows in- formation to be delivered without obstructing environmental sounds, thereby maintaining situational awareness and improv- ing safety. The system operates with acceptable latency and demonstrates stable performance during real-time evaluation, making it suitable for assistive applications in indoor environ- ments. Overall, the experimental results validate the feasibility and practicality of the proposed multisensory assistive system. The modular architecture allows flexibility and scalability, enabling future enhancements such as improved robustness under vary- ing lighting conditions, expanded gesture vocabularies, and further optimization of processing efficiency. The proposed solution highlights the potential of AI-driven assistive tech- nologies to improve accessibility, communication, and quality of life for blind and deaf users.

References

[1] S. Sagar, R. Kumar, and A. Sharma, “AI-Based Assistive System for Visually and Hearing Impaired Persons,” IEEE Access, vol. 10, pp. 118345–118356, 2022. [2] J. Redmon and A. Farhadi, “Real-Time Object Detection Using YOLO- Based Deep Learning Models,” IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 421–430, 2022. [3] G. Jocher, A. Chaurasia, and J. Qiu, “YOLOv8: A State-of-the-Art Real- Time Object Detection Framework,” Ultralytics Technical Report, 2023. [4] Z. Zhang, Y. Wang, and L. Chen, “Currency Recognition Using Deep Convolutional Neural Networks,” IEEE International Conference on Intelligent Systems, pp. 215–220, 2022. [5] C. Lugaresi et al., “MediaPipe: A Framework for Building Perception Pipelines,” IEEE Computer Vision and Pattern Recognition Workshops, [6] pp. 187–195, 2022. [7] M. Ahmed and S. Hassan, “Hand Gesture Recognition for Sign Lan- guage Translation Using CNN,” IEEE International Conference on Signal Processing and Communications, pp. 98–103, 2023. [8] R. Kaur and P. Singh, “Deep Learning-Based Sign Language Recog- nition Systems: A Survey,” IEEE Access, vol. 11, pp. 45612–45625, 2023. [9] A. Patel, D. Shah, and N. Mehta, “Optical Character Recognition for Assistive Applications Using Tesseract OCR,” IEEE International Conference on Artificial Intelligence and Data Engineering, pp. 145– 150, 2022. [10] S. Verma and K. Joshi, “Wearable Assistive Devices for Visually Impaired Users: A Review,” IEEE Sensors Journal, vol. 23, no. 4, pp. 3121–3132, 2023. [11] P. Rao, S. Iyer, and R. Nair, “Multisensory Assistive Technology Using Computer Vision and Audio Feedback,” IEEE International Conference on Smart Systems and Inventive Technology, pp. 601–606, 2024.

Copyright

Copyright © 2026 Abishek Binu, Ayisha S Anil, Dhaniyal Jose, Mithun Rajesh, Vyshna Suresh, Amritha S Nadarajan. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.