A Vision-Based Human-Computer Interaction System Using Hand Gestures

Abstract

Human–Computer Interaction (HCI) traditionally depends on physical input devices such as a mouse, keyboard, touchpad, and touchscreen. Although these devices provide efficient interaction, they require direct physical contact and may not always be convenient in situations where touchless interaction is preferred. This paper presents a Virtual Mouse Using Hand Recognition System, a computer vision-based system that enables users to control the mouse cursor and perform basic mouse operations using hand gestures. The proposed system uses a standard webcam to capture real-time video frames and employs OpenCV for image processing and MediaPipe Hand Tracking for detecting and tracking hand landmarks. The detected landmark coordinates are analyzed to identify finger positions and recognize predefined gestures. PyAutoGUI is used to translate the recognized gestures into operating-system-level mouse actions such as cursor movement, left click, right click, and scrolling. A coordinate mapping mechanism converts the hand position from the webcam frame to the computer screen coordinates, while a smoothing technique is incorporated to reduce unwanted cursor movement. The proposed system eliminates the need for a physical mouse and provides a natural and contactless method of computer interaction. The system can be useful in accessibility applications, interactive presentations, smart environments, educational demonstrations, and hygienic touch-free computing environments. Experimental observations indicate that the combination of real-time hand landmark detection and gesture-based interaction can provide responsive and intuitive mouse control under suitable lighting and camera conditions.

Introduction

The text presents a Virtual Mouse Using Hand Recognition System, a contactless human–computer interaction system that allows users to control a computer mouse using hand gestures captured through a webcam.

Main idea

Traditional computer mice require physical contact. The proposed system replaces the physical mouse with a webcam-based hand-tracking interface. The webcam captures the user's hand movements, and computer-vision software recognizes gestures and converts them into mouse commands such as:

• Cursor movement

• Left clicking

• Right clicking

• Dragging

• Scrolling

The system is designed to be low-cost, contactless, and accessible without requiring specialized hardware.

Technologies used

The system combines several software technologies:

• OpenCV: captures and processes webcam video.

• MediaPipe Hand Tracking: detects the hand and extracts landmarks such as the wrist, joints, and fingertips.

• Python: used to implement the overall system.

• PyAutoGUI: converts recognized gestures into actual mouse operations.

• Mathematical coordinate mapping: converts webcam coordinates into computer-screen coordinates.

The overall pipeline is:

Webcam → OpenCV → MediaPipe Hand Detection → Landmark Extraction → Gesture Recognition → PyAutoGUI → Mouse Action

Methodology

The system operates through five main stages:

1. Video acquisition: The webcam continuously captures frames and mirrors them so that hand movement feels natural.

2. Hand detection and landmark extraction: MediaPipe identifies important hand points, including fingertips and finger joints.

3. Finger recognition: The relative positions of landmarks determine whether individual fingers are extended or folded.

4. Cursor movement: The index fingertip is used to control the cursor. Camera coordinates are mathematically converted into screen coordinates.

5. Gesture recognition and mouse-event generation: Specific hand configurations are mapped to mouse functions such as clicking, dragging, and scrolling.

To improve usability, the system incorporates smoothing, noise reduction, and threshold-based detection to reduce cursor instability and errors caused by hand tremors, lighting, or background disturbances.

Hardware and software requirements

The system requires only:

• A standard computer or laptop

• A built-in or external webcam

• At least a dual-core processor

• Around 4 GB RAM or more

No specialized gloves, sensors, depth cameras, or other dedicated hardware are required.

Advantages and applications

The proposed virtual mouse provides a touch-free alternative to conventional input devices. Potential applications include:

• Public interactive systems

• Presentations

• Gaming

• Virtual reality

• Laboratory demonstrations

• Accessibility applications

• Situations where reducing physical contact is desirable

Conclusion

This paper presented a Virtual Mouse Using Hand Recognition System that converts hand movements and selected gestures into computer mouse operations. The system uses a webcam as the input device, OpenCV for video processing, MediaPipe-based hand landmark detection for identifying hand structure, and PyAutoGUI for operating-system mouse control. The index finger is used for cursor positioning, while selected hand configurations provide click and drag commands. Coordinate mapping, a frame margin, and smoothing are incorporated to improve practical pointer control. The proposed approach demonstrates the potential of markerless hand recognition as a simple human-computer interaction technique. Its low hardware requirement and modular software structure make it suitable for academic projects, demonstrations, and further research. At the same time, camera conditions, gesture ambiguity, cursor stability, and user variation remain important considerations. Future work can address these limitations through calibration, adaptive filtering, improved gesture classification, multi-hand interaction, and systematic quantitative evaluation.

References

[1] C. Lugaresi et al., “MediaPipe: A Framework for Building Perception Pipelines,” arXiv preprint arXiv:1906.08172, 2019. [2] Google Research, “MediaPipe Solutions: Hand Landmarker,” Google Developers documentation. [3] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000. [4] OpenCV, “OpenCV-Python Tutorials and Documentation,” OpenCV documentation. [5] Al Sweigart, Automate the Boring Stuff with Python, 2nd ed., No Starch Press, 2019. [6] Python Software Foundation, “Python 3 Documentation,” Python Documentation. [7] R. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., Springer, 2022. [8] D. W. Stutz, “Computer Vision Based Hand Gesture Recognition: A Review,” literature on vision-based gesture interaction and human-computer interaction. [9] J. Y. Bouguet, “Camera Calibration Toolbox for Matlab,” technical documentation and methods for camera calibration. [10] R. Cipolla, S. Battiato, and G. M. Farinella, Machine Learning for Computer Vision, Springer, 2013. [11] J. K. Aggarwal and Q. Cai, “Human Motion Analysis: A Review,” Computer Vision and Image Understanding, vol. 73, no. 3, pp. 428–440, 1999. [12] M. Turk and A. Pentland, “Eigenfaces for Recognition,” Journal of Cognitive Neuroscience, vol. 3, no. 1, pp. 71–86, 1991. [13] F. Zhang, A. Bazarevsky, A. Vakunov, A. Tkachenka, K. Sung, M. Chang, A. L. Grundmann, and others, “MediaPipe Hands: On-device Real-time Hand Tracking,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020. [14] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu, F. Zhang, and M. Grundmann, “BlazePalm: On-device Real-time Hand Tracking,” Google Research, 2019. [15] S. S. Rautaray and A. Agrawal, “Vision Based Hand Gesture Recognition for Human Computer Interaction: A Survey,” Artificial Intelligence Review, vol. 43, pp. 1–54, 2015. [16] S. Mitra and T. Acharya, “Gesture Recognition: A Survey,” IEEE Transactions on Systems, Man, and Cybernetics, Part C, vol. 37, no. 3, pp. 311–324, 2007. [17] V. I. Pavlovic, R. Sharma, and T. S. Huang, “Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 677–695, 1997. [18] J. P. Wachs, M. Kölsch, H. Stern, and Y. Edan, “Vision-Based Hand-Gesture Applications,” Communications of the ACM, vol. 54, no. 2, pp. 60–71, 2011. [19] R. Poppe, “A Survey on Vision-Based Human Action Recognition,” Image and Vision Computing, vol. 28, no. 6, pp. 976–990, 2010. [20] M. Turk, “Computer Vision in the Interface,” Communications of the ACM, vol. 47, no. 1, pp. 60–67, 2004

Copyright

Copyright © 2026 Dr. S. Uma, Sathiyapriya A. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.