HCI with Gesture, Eye Tracking and Voice

Abstract

The mouse, as an invention by Human-Computer Interaction (HCI) technology, is truly impressive. Still, wireless mouse or Bluetooth mouse uses things such as a battery and a mini dongle which means it isn’t truly device-free. In the system being suggested, this limitation can be addressed using computer vision and a camera to sense hand movements and their tips. The system uses a machine learning algorithm within its algorithm. With the hand gestures in place, the computer can be managed virtually and lets you left click, right click, scroll and move the cursor without a physical mouse. The technique uses deep learning to discover the hands in the video. So, the system is intended to decrease the risk of pandemics spread by lessening interaction and not requiring any additional devices to manage computer

Introduction

The text discusses the development of a gesture-controlled virtual mouse and keyboard system integrated with eye-tracking and voice commands to create a touchless, intuitive, and accessible human–computer interaction (HCI) framework. With increasing reliance on technology, traditional input devices such as mice and keyboards present limitations including reduced accessibility, hygiene concerns, hardware dependency, and inefficiency in mobile or shared environments.

Existing virtual input systems often rely on single-modality control, specialized hardware, or perform poorly under varying lighting and background conditions. To overcome these challenges, the proposed system uses only a standard webcam and combines hand gesture recognition, eye-controlled cursor movement, and voice-based commands to enable real-time, low-latency interaction without physical contact.

The methodology employs OpenCV for video capture and processing and MediaPipe for detecting 21 hand landmarks per hand. Gesture recognition is achieved by analyzing spatial relationships between finger landmarks to identify predefined gestures such as pinching, pointing, V-sign, and finger closures. These gestures are mapped to system actions like cursor movement, left/right click, double-click, scrolling, and system-level controls (volume and brightness) using PyAutoGUI and system APIs.

Results demonstrate that the multimodal system performs reliably with good accuracy, responsiveness, and usability. By integrating gesture control, eye tracking, a virtual keyboard, and a voice assistant into a unified interface, the system enhances accessibility, reduces physical contact, and supports natural interaction.

Conclusion

This project presents a more intuitive and touch-free way of interacting with computers using simple hand gestures. By combining computer vision with hand-tracking technology, it eliminates the need for physical peripherals, making digital interaction more accessible and hygienic. The system works in real time and offers decent accuracy, showcasing the potential of gesture-based interfaces in daily computing. It lays the groundwork for more natural and immersive human-computer interaction in the future. This system has broad real-world impact across multiple domains, including healthcare and medical rehabilitation, where AI-powered multimodal interfaces can help patients with neurological disorders, motor disabilities, and age-related limitations regain independence through touchless computer interaction; extended reality (XR) and spatial computing, enabling natural, hands-free control in VR/AR environments for gaming, training, virtual collaboration, and design; smart home and IoT ecosystems, where gesture, voice, and eye-based commands can serve as a universal controller for seamless, hygienic, and intuitive device management; accessibility-as-a-service platforms, offering cloud-based, customizable interfaces that promote inclusive technology access in education, workplaces, and public spaces; automotive and in-vehicle systems, enhancing safety and usability by allowing drivers and passengers to control infotainment, navigation, and vehicle functions without distraction; industrial automation and manufacturing, where touchless gesture control improves efficiency and safety in hazardous, sterile, or glove-restricted environments; and advanced biometric authentication and security, leveraging eye tracking and gesture recognition for secure, contactless user verification across financial, industrial, and high-security applications

References

[1] T. Moh R. Chitra, A. M. Anusha Bamini, D. Brindha, A. P. Reddy, N. Kezia and G. Beulah, \"Hand Gesture Recognition using Shape-based Image Features for Music Controller,\" 2022 3rd International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 2022, pp. 1540-1544, doi: 10.1109/ICOSEC54921.2022.9952146. [2] K. Aggarwal and A. Arora, \"An Approach to Control the PC with Hand Gesture Recognition using Computer Vision Technique,\" 2022 9th International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India, 2022, pp. 760-764, doi: 10.23919/INDIACom54597.2022.9763282 [3] Zhi-heng, Wang & Jiang-tao, Cao & Liu, Jinguo & Zi-qi, Zhao. (2017). Design of human-computer. interaction control system based on hand-gesture recognition. 143-147. 10.1109/YAC.2017.7967394. [4] Chengwei S. R. Chowdhury, S. Pathak and M. D. A. Praveena, \"Gesture Recognition Based Virtual Mouse and Keyboard,\" 2020 4th International Conference on Trends in Electronics and Informatics (ICOEI)(48184), Tirunelveli, India, 2020, pp. 585-589, doi: 10.1109/ICOEI48184.2020.9143016. [5] Ladwane, Avantika & Bhandeker, Sharayu & Dhaske, Suraksha & Ugalmugale, Dipali & Rane, Mugdha. (2023). OPERATING VIRTUAL MOUSE AND KEYBOARD USING GESTURE RECOGNITION. 10. 2349-5162. [6] K. H. Shibly, S. Kumar Dey, M. A. Islam and S. Iftekhar Showrav, \"Design and Development of Hand Gesture Based Virtual Mouse,\" 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), Dhaka, Bangladesh, 2019, pp. 1-5, doi: 10.1109/ICASERT.2019.8934612. [7] M. Vidya, S. Vineela, P. Sathish and A. S. Reddy, \"Gesture-Based Control of Presentation Slides using OpenCV,\" 2023 Second International Conference on Augmented Intelligence and Sustainable Systems (ICAISS), Trichy, India, 2023, pp. 1786-1791, doi: 10.1109/ICAISS58487.2023.10250520. [8] J. Hossain Gourob, S. Raxit and A. Hasan, \"A Robotic Hand: Controlled With Vision Based Hand Gesture Recognition System,\" 2021 International Conference on Automation, Control and Mechatronics for Industry 4.0 (ACMI), Rajshahi, Bangladesh, 2021, pp. 1-4, doi: 10.1109/ACMI53878.2021.9528192. [9] Shibly, Kabid & Dey, Samrat & Islam, Md. Aminul & Showrav, Shahriar. (2019). Design and Development of Hand Gesture Based Virtual Mouse. 10.1109/ICASERT.2019.8934612. [10] Kotti, J. ., Padmaja, B., & Deepa, D. (2024). Enhancing Gesture-Controlled Virtual Mouse and Virtual Keyboard Using AI Techniques. Journal of Mobile Multimedia, 20(02), 437–494. [11] Y. Zhao, X. Ren, C. Lian, K. Han, L. Xin and W. J. Li, \"Mouse on a Ring: A Mouse Action Scheme Based on IMU and Multi-Level Decision Algorithm,\" in IEEE Sensors Journal, vol. 21, no. 18, pp. 20512-20520, 15 Sept.15, 2021, doi: 10.1109/JSEN.2021.3096847. [12] N. Bhuvaneswary, M. Sathish, S. S. Abhayankar, K. S. V. Barath and P. S. N. Reddy, \"Real Time Eye Blink Recognition for Virtual Mouse Using Raspberry Pi,\" 2025 3rd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT), Bengaluru, India, 2025, pp. 972-975, doi: 10.1109/IDCIOT64235.2025.10915003. [13] L, Sai & L, Kiran. (2024). A1 Voice Assistant Using Python and API. 1-6. 10.1109/ADICS58448.2024.10533536.

Copyright

Copyright © 2025 Basava Darshan B S, Dr. Sankhya N Nayak, Benakesh S, Bhagath Mohan, Chethan J M. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.