Hand Gesture Recognition-Based Virtual Mouse Using CNN

Abstract

Advancement sin the field of computing have beena crucial point in the historyofmankind.Inputandoutputdevices, which are of utmost importance, have undergone numerous changes. Human-Computer Interface (HCI) is an importantpart of computer systems. In this paper, we have developed a virtual mouse that detects hands in live video feeds, recognizes gestures, and uses a Convolutional Neural Network (CNN) to classify them. The system then performs appropriate mouse operations.This approach aims to replace traditional hardware- based input devices.

Introduction

1. Overview of Progress (Past Decade)

Hand gesture recognition has advanced dramatically due to:

• Deep learning (e.g., CNNs for spatial features, LSTMs for temporal patterns)

• Improved sensors (depth cameras like Kinect, IMUs in wearables)

• Edge computing and lightweight models (e.g., MobileNet)

These developments have enabled real-time, robust gesture recognition in diverse environments.

2. Background

• Early systems relied on heuristic and image processing techniques.

• CNNs revolutionized static gesture recognition.

• RNNs and LSTMs enhanced dynamic gesture modeling.

• Depth cameras and wearables improved 3D gesture accuracy.

• Integration of multimodal data (visual + sensor) increased robustness.

• Applications span VR/AR, robotics, HCI, healthcare, and sign language.

3. Related Research Highlights

Key innovations include:

• Hybrid models (CNN + LSTM)

• 3D gesture datasets and benchmarks (e.g., MSR Action3D)

• Multimodal fusion (vision + sensor)

• Real-time models for mobile/edge devices

• Sign language recognition using CNNs and NLP-inspired models

• Use of public datasets to foster benchmarking and innovation

4. Notable Datasets

Some key datasets:

• MSR Gesture 3D: depth sequences

• Leap Motion: skeletal finger tracking

• ChaLearn: multimodal, dynamic gestures

• 20bn Jester: real-world dynamic hand gestures

• DHG 14/28: depth, fine-grained gestures

• EgoGesture / First-Person: egocentric views

• NTU RGB+D: multimodal action and gesture data

5. Training & Testing Accuracy

Models show high accuracy on benchmark datasets. Example:

• NTU RGB+D (2020): 98.3% train / 95.4% test

• MSR Gesture 3D (2015): 96.5% / 92.3%

• Others: ~89%–95% test accuracy using CNNs, RNNs, and hybrid models.

6. Applications

• Human-Computer Interaction: control systems, VR/AR interfaces

• Sign Language Translation: for accessibility and communication

• Robotics: gesture-based robot control and teaching

• Healthcare: rehabilitation, touchless control in sterile areas

• Security: gesture-based biometric authentication

7. Challenges & Unresolved Issues

Despite progress, challenges remain:

• Gesture variability: User differences affect accuracy.

• Background clutter: Complex scenes reduce model performance.

• Hand occlusion: Partial visibility complicates detection.

• Data scarcity: Limited annotated datasets hinder model generalization.

• Real-time constraints: High computational load impairs latency-sensitive applications.

Solutions include:

• Diverse datasets, background subtraction, multi-camera setups, data augmentation, model optimization, and lightweight CNNs.

Conclusion

Gestures using Convolutional Neural Networks (CNNs) have shown remarkable progress in recent years, becoming one of the most promising approaches for human-computer interaction CNNs, capable of automatically recognizing sur- face features, have enabled more accurate and robust gesture recognition systems . CNN in hand gesture recognition has great potential in a variety of applications, including virtual reality,signlanguageinterpretation,roboticsanduserinterface design But despite the progress, many challenges remain un- solved, such as fluctuating gestures, background clutter, hand holding, and the need for large and diverse data sets These issues affect the overall performance of gesture recognition systems, especially in real-world applications where gestures exist vary widely can and environmental conditions can be unpredictable.(23) Future research will focus on improving data acquisition methods, controlling occlusion, and optimiz- ing the model. Research on hybrid models that combineCNNs with other methods such as recurrent neural networks (RNNs) or anti-generational networks (GANs) can also help address existing limitations, and improve CNN architectures for the balance of accuracy and computational efficiency is required to implement this system for real -time applications. In conclusion, while manual applications using CNN are still evolving,thecontinueddevelopmentofnewtechniquesand the availability of better hardware and datasets will overcome theexistingchallenges,whichinvolvemanualwhichare usedfortheapplicationsfoundincommunicationdomains It holds great promise to be it is the core technology of the systems.(29)

References

[1] X.Zhao,Y.Chen,andZ.Yang,”HandGestureRecogni- tion with Convolutional Neural Networks,” IEEE Trans- actions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 1-15, 2020. [2] Y. Li et al., ”Touchless Elevator Control Using CNN- BasedHandGestureRecognition,”inIEEETransactions on Consumer Electronics, 2023. [3] A. Kumar and R. Singh, ”A Robust CNN System for Hand Gesture Recognition in Sign Language,” in Pro- ceedings of the IEEE Conference on Human-Centered Computing, 2022. [4] M. Patel et al., ”Multimodal Continuous Gesture Recog- nitionUsingCNN-BiLSTMNetworks,”IEEEAc- cess, vol. 10, pp. 243-256, 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9862878. [5] S. Rao et al., ”Hand Gesture Recognition Leveraging CNNs and Depth Data,” in Proceedings of the Interna- tionalConferenceonArtificialIntelligenceandRobotics, 2023. [6] ”GestureRecognitionforSmartHealthcare UsingCNN-BasedModels,”IEEETransactions onBiomedicalEngineering,vol.70,no.4, pp. 1084-1095, 2022. [Online]. Available: https://ieeexplore.ieee.org/document/10248500. [7] Z. Wei et al., ”Improving Hand Gesture Detection in Complex Backgrounds Using CNNs,” IEEE Journal of Emerging Technologies in Computing Systems, 2023. [8] T. Wen and L. Liu, ”Gesture Recognition for Smart HomeAutomation:CNNApplications,”IEEEEmbedded Systems Letters, vol. 14, pp. 53-58, 2023. [9] P.Joshietal.,”DynamicHandGestureRecognitionwith Depth Sensors and CNN,” in Proceedings of the IEEE ConferenceonComputerVisionandPatternRecognition, 2023. [10] X. Zhao and J. Chen, ”Real-Time Gesture Recognition for AR Interfaces,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, pp. 2021-2028, 2023. [11] A. Smith et al., ”Applications of Lightweight CNN Models for Mobile Gesture Recognition,” IEEE Access, vol. 11, pp. 301-315, 2022. [12] T. Lee, ”Fusion of RGB and Depth Data for Enhanced Gesture Recognition,” IEEE Signal Processing Letters, vol. 29, no. 5, pp. 143-150, 2022. [13] J. Wu et al., ”Hand Gesture Recognition in Wearable DevicesUsingCNNs,”IEEEEmbeddedSystemsLetters, 2023. [14] K. Chang and L. Wang, ”Optimizing CNNs for Gesture Recognition in Edge Devices,” IEEE Transactions on Mobile Computing, vol. 25, no. 7, pp. 763-776, 2023. [15] ”GestureRecognitionUsingHybridCNN-RNNModels,” in Proceedings of the IEEE International Joint Confer- ence on Neural Networks, 2023. [16] S. Rao, ”A Comparative Study of Deep Learning Archi- tectures for Hand Gesture Recognition,” IEEE Access, vol. 10, pp. 1934-1950, 2023. [17] ”Advanced Datasets for CNN-Based Hand Gesture Recognition,” in IEEE Transactions on Machine Learn- ing, vol. 14, pp. 523-531, 2023. [18] L. Zhang et al., ”Gesture Recognition for Autonomous VehiclesUsingCNNs,”IEEETransactionsonIntelligent Transportation Systems, vol. 24, no. 1, pp. 33-42, 2023. [19] M.Pateletal.,”StaticandDynamicGestureRecognition with Lightweight CNN Models,” in IEEE Transactions on Systems, Man, and Cybernetics, vol. 53, pp. 672-680, 2023. [20] T.Zhangetal.,”TCCNN:TransformerConCatedConvo- lutionalNeuralNetworksforHandGestureRecognition,” IEEE VDE Conference, 2023. [21] M. Patel et al., ”A Deep CNN-Based Hand Gestures Recognition Using High-Resolution Thermal Imaging,” IEEE Conference Publication, 2023. [22] L. Wei et al., ”Hand Gesture Recognition with Deep ConvolutionalNeuralNetworks:AComparativeStudy,” IEEE Systems and Process Control Conference, 2023. [23] J. Kumar et al., ”Hand Gesture Recognition for Charac- ters Understanding Using Convex Hull Landmarks and Geometric Features,” IEEE JournalsMagazine, 2023. [24] R. Liu et al., ”Advancements in Prosthetic Hand Control UsingCNN-BasedGestureRecognition,”IEEETransac- tions on Biomedical Engineering, 2023. [25] A.Guptaetal.,”HandGestureRecognitionforRemote Control Applications Using CNNs,” IEEE Transactions onConsumerElectronics,vol.69,no.1,pp.75-83,2023. [26] T. Nguyen et al., ”Real-Time Hand Gesture Recognition for Virtual Environments Using Depth and RGB Data,” IEEE Access, vol. 11, pp. 15000-15014, 2023. [27] S. Bansal and R. Kumar, ”Hybrid CNN-RNN Architec- turesforMultimodalGestureRecognition,”IEEETrans- actions on Multimedia, vol. 25, pp. 333-345, 2023. [28] Y. Zhao et al., ”Dynamic Gesture Recognition Using Skeleton-Based CNN Models,” IEEE Transactions on Cybernetics, 2023. [29] J. Wu et al., ”Gesture Recognition in Augmented Re- ality: Leveraging CNNs for Intuitive Interaction,” IEEE Transactions on Visualization and Computer Graphics, 2023. [30] M. Singh et al., ”Dataset for Continuous Hand Gesture RecognitioninReal-WorldEnvironments,”IEEEAccess, vol. 11, pp. 21523-21535, 2023. [31] L. Han et al., ”Exploring Transformer Models for Hand GestureRecognition,”IEEETransactionsonNeuralNet- works and Learning Systems, 2023. [32] F. Delgado et al., ”High-Accuracy Hand Gesture Recog- nition Using CNNs and Thermal Data,” IEEE Sensors Journal, vol. 23, no. 3, pp. 243-254, 2023. [33] K.ChandrasekharandA.Vyas,”IntegrationofHand Gesture Recognition in Autonomous Vehicles Using CNN Models,” IEEE Intelligent Transportation Systems Magazine, vol. 15, no. 1, pp. 23-33, 2023. [34] Z. Lin et al., ”Robust Gesture Recognition in Low-Light Conditions Using Thermal CNN Models,” IEEE Trans- actions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2023-2034, 2023.

Copyright

Copyright © 2025 Sampreeth S. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.