BlindAssist: A Near-Real-Time Object Detection and Voice-Based Assistive System for Visually Impaired Users

Abstract

BlindAssist is a Python prototype that uses a laptop webcam and a YOLOv8 object detector to tell a visually impaired user, by voice, which objects are in front of the camera and roughly where they are. Frames are captured with OpenCV and passed to a COCO-pretrained YOLOv8 model (Ultralytics), and the results are spoken offline with pyttsx3. Each announcement gives the object’s class name and whether it lies to the left, ahead, or to the right, based on the horizontal position of its bounding box. For seven object classes an uncalibrated estimate of distance is computed from the box width and shown on screen; it is not spoken. Objects are tracked by class name, so that a class is announced when it first appears, when its direction changes, and once when it has been missing for three consecutive detection passes. Face modules using YuNet and SFace (OpenCV) and the DeepFace library recognize known people from reference photographs, and a DeepFace emotion model announces the dominant facial expression of each detected face; a custom pen-and-pencil model is under development. The paper describes the design, gives the algorithms and configuration, compares the system with published assistive systems, and describes the tests we plan to run. No labeled test set or user study has been completed, so no accuracy, speed, or user-study figures are reported.

Introduction

BlindAssist is a Python-based assistive application designed to help visually impaired people understand their surroundings through spoken descriptions of objects and people detected by a webcam. The system uses YOLOv8 for object detection, OpenCV for image processing, pyttsx3 for offline text-to-speech, and facial recognition and expression-analysis libraries to provide additional information. Once the required models are downloaded, the application can operate without an internet connection.

The system identifies objects in a live video stream and announces their names and approximate directions as left, ahead, or right. It uses announcement-control logic to avoid unnecessary repetition, announcing objects when they first appear, when their direction changes, and when they disappear after three consecutive detection passes without being detected. For seven supported object classes, the application also estimates distance using the object's bounding-box width; however, these estimates are uncalibrated and are displayed only on the screen rather than spoken aloud.

BlindAssist incorporates face detection and recognition using OpenCV YuNet and SFace, along with DeepFace for additional identity recognition and facial-expression analysis. Recognized people can be announced by name, while the emotion-analysis module identifies and speaks a dominant facial-expression label. These features have been tested informally during development, but their accuracy has not been formally evaluated.

The application is developed using Python, YOLOv8, PyTorch, OpenCV, pyttsx3, and other open-source libraries. It can use a GPU when CUDA is available or fall back to the CPU. A custom dataset for detecting pens and pencils is being prepared using Roboflow because these objects are not included in the default COCO-trained model. Jenkins-based development automation is also in progress.

The main contribution of BlindAssist is the integration of object detection, directional voice announcements, repeat-control logic, face recognition, and facial-expression analysis into a single local application. Unlike conventional object-detection systems that primarily display visual labels, BlindAssist converts detected information into speech to make it more accessible to users with visual impairments.

However, the system remains a prototype with several limitations. Its detection-to-speech latency has not been measured, distance estimates have not been calibrated, and object tracking is based on class names rather than individual objects. Custom object detection and some development-automation features are still under development, while face-recognition and emotion-analysis accuracy remain unverified.

Conclusion

BlindAssist is a webcam-based prototype that turns YOLOv8 detections into spoken sentences giving the object’s name and direction. A class is spoken once when it appears, again if it moves, and once when it is gone. Face modules can name known people, and a DeepFace emotion stage announces the dominant facial expression. Distance is estimated for seven classes only and is shown on screen but not spoken. Compared with the published systems reviewed here, the prototype uses similar components but adds explicit announcement rules, a known-person module, and facial-expression announcements; it offers neither depth-based obstacle detection nor the breadth of a general-purpose app. It is a small aid and not a navigation system for blind users. The main limits are the COCO-only default model, the uncalibrated distance, and the lack of any measured results. The next steps are to build a labeled test set, calibrate the distance estimate, train the pen and pencil model, and test with visually impaired users before making any claims about accuracy or usefulness. The Jenkins setup is also still in progress.

References

[1] World Health Organization, World Report on Vision. Geneva, Switzerland: WHO, 2019. [2] R. R. A. Bourne et al., “Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis,” The Lancet Global Health, vol. 5, no. 9, pp. e888–e897, 2017, doi: 10.1016/S2214-109X(17)30293-0. [3] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, real-time object detection,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788, doi: 10.1109/CVPR.2016.91. [4] G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” 2023. [Online]. Available: https://github.com/ultralytics/ultralytics [5] W. Elmannai and K. Elleithy, “Sensor-based assistive devices for visually-impaired people: Current status, challenges, and future directions,” Sensors, vol. 17, no. 3, Art. no. 565, 2017, doi: 10.3390/s17030565. [6] R. Tapu, B. Mocanu, and T. Zaharia, “Wearable assistive devices for visually impaired: A state of the art survey,” Pattern Recognition Letters, vol. 137, pp. 37–52, 2020, doi: 10.1016/j.patrec.2018.10.031. [7] M. Poggi and S. Mattoccia, “A wearable mobility aid for the visually impaired based on embedded 3D vision and deep learning,” in Proc. IEEE Symp. Computers and Communications (ISCC), 2016, pp. 208–213, doi: 10.1109/ISCC.2016.7543741. [8] R. Tapu, B. Mocanu, and T. Zaharia, “DEEP-SEE: Joint object detection, tracking and recognition with application to visually impaired navigational assistance,” Sensors, vol. 17, no. 11, Art. no. 2473, 2017, doi: 10.3390/s17112473. [9] G. J. Ahmed, F. H. Khorsheed, and F. K. Zaidan, “Designing an assistive tool for visually impaired people based on object detection technique,” J. Informatics, Network, and Computer Science (JOINCS), vol. 8, no. 2, 2025. [Online]. Available: https://joincs.umsida.ac.id [10] A. Tavakoli Yaraki, “Seeing with sound: Object detection and localization by YOLOv8 and audio feedback for blind individuals,” M.S. thesis, Univ. of Padua, Padua, Italy, 2024. [Online]. Available: https://thesis.unipd.it/handle/20.500.12608/66485 [11] B. Mocanu, R. Tapu, and T. Zaharia, “DEEP-SEE FACE: A mobile face recognition system dedicated to visually impaired people,” IEEE Access, vol. 6, pp. 51975–51985, 2018, doi: 10.1109/ACCESS.2018.2870334. [12] Microsoft, “Seeing AI.” [Online]. Available: https://www.microsoft.com/garage/wall-of-fame/seeing-ai [13] T.-Y. Lin et al., “Microsoft COCO: Common objects in context,” in Computer Vision – ECCV 2014, LNCS vol. 8693. Springer, 2014, pp. 740–755, doi: 10.1007/978-3-319-10602-1_48. [14] W. Wu, H. Peng, and S. Yu, “YuNet: A tiny millisecond-level face detector,” Machine Intelligence Research, vol. 20, no. 5, pp. 656–665, 2023, doi: 10.1007/s11633-023-1423-y. [15] Y. Zhong, W. Deng, J. Hu, D. Zhao, X. Li, and D. Wen, “SFace: Sigmoid-constrained hypersphere loss for robust face recognition,” IEEE Trans. Image Processing, vol. 30, pp. 2587–2598, 2021, doi: 10.1109/TIP.2020.3048632. [16] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000. [17] A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. [18] pyttsx3 contributors, “pyttsx3: Offline text-to-speech library for Python.” [Online]. Available: https://pypi.org/project/pyttsx3/ [19] Roboflow, “Roboflow: Computer vision developer framework.” [Online]. Available: https://roboflow.com [20] N. Sakure, C. Pagare, Md. E. Sheikh, S. Debnath, S. Sharnagat, H. Keche, and V. Shrivastava, “BlindAssist,” GitHub repository, 2026. [Online]. Available: https://github.com/CoderNishchay/Blind-Assist [21] Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330–1334, 2000, doi: 10.1109/34.888718. [22] S. I. Serengil and A. Ozpinar, “LightFace: A hybrid deep face recognition framework,” in Proc. Innovations in Intelligent Systems and Applications Conf. (ASYU), 2020, pp. 23–27, doi: 10.1109/ASYU50717.2020.9259802. [23] S. I. Serengil and A. Ozpinar, “HyperExtended LightFace: A facial attribute analysis framework,” in Proc. Int. Conf. Engineering and Emerging Technologies (ICEET), 2021, pp. 1–4, doi: 10.1109/ICEET53442.2021.9659697. [24] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in Proc. British Machine Vision Conf. (BMVC), 2015, pp. 41.1–41.12, doi: 10.5244/C.29.41.

Copyright

Copyright © 2026 Nishchay Sakure, Chanchal Pagare, Md. Eliyas Sheikh, Sapna Debnath, Saurabh Sharnagat, Harsha Keche, Mr. Vaibhav Shrivastava. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.