AI Powered Smart Vision Calculator

Abstract

TheAIPoweredSmartVisionCalculatoris a gesture-based system developed to enhance mathematical problem-solving without the use of traditional input devices. It employs real-time hand tracking using MediaPipe and OpenCV to recognize user gestures, which are then interpreted into mathematical symbols through a fine-tuned Gemini-2B model implemented in TensorFlow. The front-end interface enables userstodraw,submit,andsolveequationsthroughintuitivehand movements in the air. This system aims to improve accessibility, support educational engagement, and promote a natural modeof human-computer interaction. Its accurate symbol recognition, real-time feedback, and seamless integration of AI technologies establish it as a user-friendly and innovative solution for inter- active learning and creative expression.

Introduction

The AI Powered Smart Vision Calculator is an innovative, hands-free mathematical tool designed to replace traditional input methods like keyboards and mice with natural hand gestures. It leverages AI technologies—including OpenCV and MediaPipe for real-time hand tracking and TensorFlow with a fine-tuned Gemini-2B model for symbol recognition and math reasoning—to allow users to write equations in the air and receive instant, step-by-step solutions.

This system addresses key limitations of existing calculators by enabling intuitive, interactive, and accessible math problem-solving without conventional devices, making it suitable for diverse users from students to teachers. It also ensures privacy and offline functionality by performing local inference without relying on internet connectivity.

The solution integrates gesture-based commands, a color-selection toolbar, and secure backend processing using Django and Hugging Face APIs, providing a scalable, responsive web interface. Tests with algebraic and calculus problems demonstrate the system’s accuracy and educational value.

The project builds upon prior research in gesture recognition, fine-tuned language models, and ethical AI deployment, improving on their shortcomings by offering robust, adaptive, real-time gesture interpretation and safe, context-aware AI explanations.

Conclusion

AI Powered Smart Vision Calculator provides a novel mathematical solution-discovery process using real-time ges- ture recognition and AI-supported symbol interpretation. The system, supported by software such as OpenCV, MediaPipe, andtunedGemini-2B,enablesuserstohand-drawandanalyze equationsfreefromthelimitationsoftraditionalinputdevicesin mid-air space. The system provides hands-off, intuitive interactions, increasing accessibility, accommodating various learning needs, and enabling collaborative interaction withthe math subject matter. In subsequent releases, the system will introduce adaptive learning capabilities that adapt to performance and user preference. Future planned develop- mentincludesmulti-languagesymbolrecognition,cloud-based session history storage, and improved model compression methods to further improve performance on lower-spec hard- ware. The features will increase the availability of the system usability, thereby making it an invaluable educational and productivity tool.

References

[1] S. M., D. R., S. Patil, and N. V., “AirCanvas using OpenCV andMediaPipe,” International Journal for Research in Applied Science &Engineering Technology, vol. 13, no. 1, pp. 1467–1470, Jan. 2024. [2] P. Bhande et al., “Gesture-Based Air Typing: A Machine LearningApproachforAccessibleTextInput,”JournalofScientificResearchandTechnology, vol. 3, no. 1, pp. 1–11, Jan. 2024. [3] P. Kumar et al., “A Canvas of Air and Signs: Integrating Voice Activated Hand Sign Recognition and Air Canvas for Hearing Impaired and Non-Verbal People,” International Journal of Innovative Science and Research Technology, vol. 9, no. 4, pp. 172–178, Apr. 2024. [4] V. Jain and R. Hebbalaguppe, “AirPen: A Touchless Fingertip Based Gestural Interface for Smartphones and Head-Mounted Devices,” Tata Consultancy Services Research, New Delhi, India, Tech. Rep., 2024. [5] Y.-H. Chen et al., “Egocentric-View Fingertip Detection for AirWriting Based on Convolutional Neural Networks,” Sensors, vol. 21, no. 13, p. 4382, Jun. 2021. [6] W. Hawkins, B. Mittelstadt, and C. Russell, “The Effect of Fine-Tuning on Language Model Toxicity,” Oxford Internet Institute, University of Oxford, Tech. Rep., 2023. [7] B. Zhang et al., “Encoder-Decoder Gemma: Improving the Quality- Efficiency Trade-Off via Adaptation,” arXiv:2403.12345, Mar. 2024. [8] K. Mo et al., “Fine-Tuning Gemma-7B for Enhanced Sentiment Analysis of Financial News Headlines,” arXiv:2403.67890, Mar. 2024. [9] M. Syromiatnikov, V. Ruvinskaya, and N. Komleva, “Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks,” arXiv:2403.54321, Mar. 2024.

Copyright

Copyright © 2025 K S Mahesh, Sudeep Venkatesh Naik, Rahul H Pujari, Anoop Shetty Y, Manjunatha S. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.