A Multimodal Gesture and Voice Controlled Virtual Mouse for Accessible Desktop Interaction

Abstract

This paper presents a multimodal virtual mouse system that integrates real-time hand gesture recognition with voice command processing to facilitate touchless desktop interaction. Built upon Python, the system utilizes MediaPipe for hand landmark detection, OpenCV for visual capture, and PyAutoGUI for system automation. By combining intuitive gesture control for cursor manipulation and voice-driven commands for browserbased operations, the project creates a practical, low-cost, and accessible input interface. The system supports cursor movement, left click, right click, scrolling, idle-state detection, opening Google, and spoken web search. The paper expands the original project draft into a full IEEE-style manuscript by incorporating user-supplied experimental screenshots, a detailed implementation description, an evaluation framework for qualitative and quantitative analysis, and placeholders for measured metrics such as latency, recognition accuracy, false positives, false negatives, and voice-command success rate. The resulting manuscript is designed to be directly editable in Overleaf and to serve both as a submission-ready template and as a reproducible record of the implemented system.

Introduction

This paper presents a multimodal gesture-and-voice controlled virtual mouse that enables touchless desktop interaction using a standard webcam and microphone. The system is designed as an accessible, low-cost alternative to traditional input devices by combining vision-based hand tracking for cursor control and mouse actions with speech recognition for executing higher-level desktop and browser commands. The multimodal approach leverages the complementary strengths of gestures for spatial navigation and voice for semantic commands, making interaction more natural and efficient.

The proposed system addresses the limitations of conventional mouse and keyboard input in scenarios such as accessibility-focused computing, healthcare, public kiosks, classrooms, and hands-busy environments. Unlike many existing virtual mouse systems that rely solely on gesture recognition, this work integrates voice commands to support tasks such as opening websites and performing search queries without specialized hardware.

The architecture consists of six modular components: video capture, hand landmark detection, gesture interpretation, voice recognition, action execution, and user feedback. The system employs MediaPipe Hands to detect 21 hand landmarks in real time, OpenCV for video processing, SpeechRecognition for voice input, and PyAutoGUI for controlling desktop events such as cursor movement, clicking, scrolling, and keyboard automation. Gesture recognition is implemented using transparent rule-based logic based on fingertip positions and distances, while voice commands are processed through ambient-noise calibration and command parsing.

To improve usability and reliability, the paper proposes a calibration procedure that includes camera positioning, cursor mapping, gesture threshold tuning, and voice calibration. Mathematical models for fingertip distance calculation, coordinate mapping, and cursor smoothing are also provided to reduce jitter and improve interaction accuracy. The implementation emphasizes safety by using PyAutoGUI's built-in fail-safe mechanisms to prevent unintended cursor actions.

The study also reviews prior work on multimodal interaction, accessibility, and computer vision, identifying a gap in existing gesture-based systems that lack detailed implementation guidelines, accessibility considerations, and integrated voice functionality. The proposed system addresses these shortcomings by providing a transparent architecture, qualitative evaluation framework, and reproducible implementation details. Although quantitative performance metrics are yet to be completed, the work establishes a practical foundation for future evaluation and extension of touchless desktop interaction systems using commodity hardware.

Conclusion

This paper presented a multimodal gesture and voice controlled virtual mouse system for touchless desktop interaction. The system combines MediaPipe-based hand landmark detection, OpenCV-based video capture, PyAutoGUI-based desktop automation, and speech-based browser control to create a lowcost human-computer interaction framework. The manuscript preserves the core idea and wording of the original draft while expanding it into a full IEEE-style paper with architecture, implementation details, evaluation methodology, discussion, limitations, ethics, and future work. The uploaded screenshots provide clear qualitative evidence that the system can detect idle, move cursor, left click, right click, scroll, open Google, and search-by-voice states. At the same time, the paper deliberately avoids inventing missing quantitative values. Instead, it includes clearly marked placeholders for the measurements that must be supplied before formal submission. In this sense, the manuscript is both a complete write-up of the implemented system and a transparent template for rigorous final evaluation.

References

[1] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays,F. Zhang, C.-L. Chang, M. G. Yong, J. Lee, W.-T. Chang, W. Hua, M. Georg, and M. Grundmann, “MediaPipe: A Framework for Building Perception Pipelines,” arXiv preprint arXiv:1906.08172, 2019. [2] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, “MediaPipe Hands: On-device Real-Time Hand Tracking,” arXiv preprint arXiv:2006.10214, 2020. [3] Google, “MediaPipe Hands Documentation,” 2026. [Online]. Available: https://mediapipe.readthedocs.io/en/latest/solutions/hands.html. [4] Accessed: Apr. 29, 2026. [5] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000. [6] OpenCV, “cv::VideoCapture Class Reference,” 2026. [Online]. Available: https://docs.opencv.org/4.x/d8/dfe/classcv11VideoCapture.html. Accessed: Apr. 29, 2026. [7] A, Sweigart, “PyAutoGUI Documentation,” 2026. [Online]. Available: https://pyautogui.readthedocs.io/. Accessed: Apr. 29, 2026. [8] Uberi, “SpeechRecognition Documentation,” 2026. [Online]. Available: https://github.com/Uberi/speechrecognition and https://www.mintlify. com/Uberi/speechrecognition/api/recognizer. Accessed: Apr. 29, 2026. [9] H. Pham, “PyAudio: Python Bindings for PortAudio,” 2026. [Online]. Available: https://people.csail.mit.edu/hubert/pyaudio/. Accessed: Apr. 29, 2026. [10] J. O. Wobbrock, S. K. Kane, K. Z. Gajos, S. Harada, and J. Froehlich, “Ability-Based Design: Concept, Principles and Examples,” ACM Transactions on Accessible Computing, vol. 3, no. 3, Art. no. 9, 2011. [11] S. L. Oviatt and P. R. Cohen, “Perceptual User Interfaces: Multimodal Interfaces that Process What Comes Naturally,” Communications of the ACM, vol. 43, no. 3, pp. 45–53, 2000. [12] D. M. Krum, O. Omoteso, W. Ribarsky, T. Starner, and L. F. Hodges, “Speech and Gesture Multimodal Control of a Whole Earth 3D Visualization Environment,” in Proceedings of the Symposium on Data Visualisation 2002 (VisSym 2002), 2002, pp. 195–200. [13] S. Krishna and N. Sinha, “Gestop: Customizable Gesture Control of Computer Systems,” CoRR, abs/2010.13197, 2020.

Copyright

Copyright © 2026 S. Jaya Naga Chandrika, V. Swapna, K. Revathi, K. Ratna Kumar, G. Hanok. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.