With the rise of technology, computing has been embedded in portable electronics like cellphones and palm tops. The QWERTY keyboard is still the primary input method, though. This project proposes a virtual keyboard programme that generates a visual using image processing. This is a representation of a keyboard. When a camera records hand movements as typing inputs, the simulated keyboard will be operational and available. The creation of a virtual cursor that recognizes fingers is based on the same idea. The motions of the hands that move the mouse will be captured by the camera. Use the camera to take a picture of a physical keyboard or mouse so that you may create a virtual version of them.
Introduction
I. INTRODUCTION
To interact with the user, the majority of modern mobile phones feature touch screen technology. But utilising this kind of technology in laptops and desktops still comes at a premium price. Making a virtual mouse technology that is easy to use and can take the role of touch displays was our main objective. This was accomplished using a webcam. When someone sits close to of the computer, the camera on the computer records their image, which appears as a small green box in the center of the screen. The objects displayed in the box in green will be processed and matched by the code. When the computer detects an item, it produces a boundary that may be changed with the pointer of the mouse if it matches, turning it a red color. In addition to enhancing computer security, this offers a virtual computing environment. The pointer may be moved, clicked to the left or right, and other keyboard operations that are comparable to those done on physical keyboards can be done with hand gestures. The gesture will only show a green box if it is inconsistent with the box. When the motion is detected, a red boundary that respects the positioning of the finger caps will show. The intricacy of utilizing the left and right click features was the only problem. OpenCV is an entirely free software library for visual analysis and artificial intelligence. A Python package called Numpy provides simple yet elegant data structures. A machine learning framework called Tensorflow is open-source, whereas Keras is a library for neural networks.We'll use convolutional neural networks and SVM Hog in this project. For person detection, the SVM Hog histogram of directed gradients is utilised for feature extraction, and linear support vector machine (SVMs) are employed for classification.
II. LITERATURE SURVEY
A real-time video image is used to create an artificial mouse that recognises hand and fingertip gestures. Thumma Dhyanchand, Galla Vamsi Krishna, and Satish Maheshwaram, Virtual Mouse Control with Colored Fingertips and Hand Gesture Recognition Vantukala Vishnu Teja Reddy, 2020.
This work presents a new virtual mouse technology based on real-time fingertip sensing and RGB-D images. Images in RGB-D are recommended. and the 2000 denomination note's electrotype watermark, to name just a few of the characteristics that were taken away throughout this process. Real-time virtual mouse system with RGB-D images and fingertip sensing, Dinh-Son Tran, Ngoc-Huynh Ho, Hyung-Jeong Yang, Soo-Hyung Kim, and Guee Sang Lee, 2020.
A virtual keyboard that employs a multi-tab approach with three to four arrays is produced using convex-Hull and keypad layout optimization. This type of virtual keyboard is often utilized in mobile contexts. A Novel Virtual Keypad with Fingers Motion Detection for AR/VR Devices, Tae-Ho and Hyuk-Jae Lee, 2018.
The paper suggests a technique that uses a web-based or built-in camera to record hand motions and identify hand tips using computer vision. J. Nagaraj, P. Ajay, S. Jaya, B. Shriram, and Shankar. A Real-Time AI Artificial Mouse System Based on Deep Learning and Utilising Computer Vision to Prevent the Spread of COVID-19
Users may engage and exert control over the non-touch letter writing method described in this work. A healthier and safer option to physical keyboards are virtual keyboards that appear on screens. by distinguishing a small range of hand gestures. A non-touch character writing system using a virtual keyboard that uses hand gesture detection was developed by Md. Abdur Rahim, Jungpil Shin, and Md. Rashedul Islam in 2020.
Utilizing human input, the system analyses the real-time video data before utilizing facial recognition. In 2020, at the Akshada Dongre Department of IT at St. Francis Institute of Technology in Mumbai, India, Rodney Pinto, Ameya Patkar, and Minal Lopes presented their work on Computer Cursor Control using Eye and Facial Movements.
In August 2015, the International Journal of Electrical Electronics & Computer Science Engineering published an article titled "Virtual Mouse Implementation Using Colour Pointer Detection," written by Abdul Khaliq and A. Shahid Khan and published on pages 63–66.
Erdem, E. Yardimci, Y. Atalay, V. Cetin, and A. E., "Computer vision-based mouse", Acoustics, Speech, and Signal Processing, Proceedings (ICASS), IEEE International Conference, 2002 Chu-Feng Lien, "Portable Vision-Based HCI - A Realtime Hand Mouse System on Handheld Devices", National Taiwan University, Computer Science and Information Engineering.
III. PROBLEM FORMULATION
Internet services, home automation systems, robot abuse, and gaming may all benefit from the simple integration of computer vision-based mouse. Researchers studying computer vision are especially interested in following non-rigid movements in consecutive movies. We engage with the physical environment around us as we grow up. Through the use of movements to communicate with objects and each other, gesture recognition systems have helped us comprehend the interaction between humans and objects better. This study recognises gestures with a webcam. 3D or stereo cameras are not required, as the previously reported research was also carried out using an inexpensive, 1.3-megapixel laptop webcam.
In dynamic contexts, a computer's vision-based mouse facilitates communication between humans and computing equipment. The method for executing mouse activities without requiring a physical interface between people and computer equipment is demonstrated in this paper. This method works well for a variety of activities, such as flicking through photos, playing games, and switching TV stations. The distance (measured in metres) separating the person using it and the lens has a threshold value that may be changed in accordance with the resolution of the camera. In order for hand gestures to be recognised in a particular situation, the subject needs to go close to the camera. A 1.3 megapixels webcam with a 2m threshold was used for the study.
IV. METHODOLOGY
OpenCV is a module for Python that facilitates instantaneous computer vision app creation; the library included in OpenCV is used for the processing of video and images and evaluation, including face and item identification; it is a machine learning toolkit that contains image processing algorithms for object detection.
Given the camera feed, the first step is to eliminate the background. We employ a running average to obtain the average image, which also serves as the backdrop image
In background removal, a reference picture is created, every new frame is subtracted from it, and the resulting binary segmentation is thresholder to emphasize unpredictable elements.
The built-in edge extraction function in OpenCV is used for contour extraction. It employs a clever filtering technique. You can adjust parameters to improve edge detection.
Given that convex hull points are more frequently located on the extremities, this information may be used to determine the number of fingers. We are locating the contour's highest point of divergence.
We employ mathematical features of contours and regions to determine the positions of the user's fingertips in image space. Find out where the individual's fingers are. Check to see whether any fingers are in contact with the table surface.
Taking into account the user's expected postures, our job is to ascertain which fingers are most likely to come into touch with the computer's keyboard mat. We used a method known as shadow analysis. To Map Clicks to Touch Points on a Keypad We identified the letters and matched touch to keyboards.
In order to follow the movement of the mouse and identify fingertips, we number each one. Movement Identification: Clicks, Both One and Multiple Enter a keystroke to instruct the system to run.
The computer's screen will show a keypad.
Your fingerprint keypad will allow the camera to record live feeds.
Real-time word typing recognition on a computer keyboard will happen as an outcome of the picture analysis.
The computer's screen will display those phrases
V. RESULTS AND DISCUSSIONS
In conclusion, a major step towards improving human-computer interaction has been made with the creation of a virtual keyboard and the mouse system that makes use of gesture detection and image processing. By leveraging webcam technology and sophisticated algorithms, we have demonstrated the feasibility of substituting traditional input devices with intuitive, gesture-based interfaces. While challenges such as refining click functionality persist, the integration of OpenCV, Numpy, Keras, and TensorFlow underscores the potential for future advancements in this field. As technology continues to evolve, our work lays a foundation for more seamless and natural interaction paradigms between users and computing devices.
VI. FUTURE SCOPE
The project aims to improve the Fingertip Detection module by making it less sensitive to lighting fluctuations and identifying the panel's 3D posture for augmented reality.
In the future, events involving touch on the displayed screen will be detected by the HCI by means of visual elements like icons and character forms.
Future plans include for the addition of new features including multi-finger window enlargement and closure.
Additionally, we want to integrate speech recognition into the keyboard.
References
[1] Y. Chen, E.R. Hoffmann, R.S. Goonetilleke, Structure of Hand/Mouse Movements, IEEE Trans. Human-Machine Syst. 45 (2015) 790–798. doi:10.1109/THMS.2015.2430872.
[2] Erdem, E. Yardimci, Y. Atalay, V. Cetin, 2002. Computer vision based mouse, Proceedings. (ICASS). IEEE International Conference.
[3] Eckert, M. Lopez, M. ; Lazaro, C. ; Meneses, J. ; Martinez Ortega, J.F., 2015 Mokey - A motion based keyboard interpreter .Tech. Univ. of Madrid, Madrid, Spain.
[4] Chu-Feng Lien, Portable Vision-Based HCI - A Real-time Hand Mouse System on Handheld Devices.
[5] Jun Hu, Guolin Li, Xiang Xie, Zhong Lv, and Zhihua Wang, Senior Member, IEEE:Bare-fingers Touch Detection by the Button’s Distortion in a Projector–Camera System.
[6] Dikovita, C.R.; Abeysooriya, D.P., \"V-Touch: Markerless laser-based interactive surface,\" 2013 International Conference on Advances in ICT for Emerging Regions (ICTer), Dec. 2013: 248-252.
[7] AlKassim, Z., \"Virtual laser keyboards: A giant leap towards human-computer interaction,\" 2012 International Conference on Computer Systems and Industrial Informatics (ICCSII), Dec. 2012:1-5.
[8] Du H, Oggier T, Lustenberger F, et al. “A virtual keyboard based on true-3D optical ranging,”British Machine Vision Conference. 2005, pp.220-229.
[9] Hagara M, Pucik J. “Accuracy of 3D camera based virtual keyboard”. 2014 24th International Conference on Radioelektronika (RADIOELEKTRONIKA), IEEE, 2014: 1-4.
[10] Banerjee, A. Ghosh, K. Bharadwaj, H. Saikia, Mouse Control using a Web Camera based on Colour Detection, Int.J. Comput. Trends Technol. 9 (2014) 15–20. doi:10.14445/22312803/ijctt-v9p104.