A Prototype Deep Learning-Based System for Indian Sign Language Alphabet Gesture Recognition

Abstract

Indian Sign Language (ISL) is a communication medium used by hearing and speech-impaired individuals. In this paper a prototype deep learning-based Indian Sign Language alphabet gesture recognition system using a MobileNetV2-based Convolutional Neural Network (CNN) model has been proposed. The proposed system recognizes static ISL alphabet gestures and converts them into readable text in real time. TensorFlow and Keras with transfer learning techniques have been used in developing the model. Image preprocessing and data augmentation methods were applied to improve model generalization and prediction accuracy. The MediaPipe Python framework and OpenCV were used for real-time hand detection, gesture extraction, and webcam-based prediction.The experimental results showed stable training and validation performance with minimal overfitting. During testing, the trained model achieved an accuracy of 99.90%. The developed system performed effectively under proper lighting conditions and clear hand positioning, demonstrating its suitability for real-time static ISL gesture recognition.

Introduction

The study presents a real-time Indian Sign Language (ISL) alphabet recognition system using deep learning and computer vision to improve communication between hearing/speech-impaired individuals and non-sign language users. The system captures hand gestures through a webcam and recognizes static ISL alphabet gestures using a MobileNetV2-based Convolutional Neural Network (CNN). It is implemented with TensorFlow, Keras, OpenCV, MediaPipe, and CustomTkinter, providing real-time text output with confidence scores. The Kaggle ISL alphabet dataset was divided into training, validation, and testing sets, while preprocessing techniques such as resizing, normalization, grayscale conversion, Gaussian blur, CLAHE enhancement, and data augmentation improved model robustness and reduced overfitting.

A review of previous research shows that CNN, LSTM, GRU, SVM, and Transformer-based models have achieved high recognition accuracy, but many systems focus either on static or dynamic gestures separately. The proposed methodology combines transfer learning with MobileNetV2, MediaPipe hand tracking, and efficient image preprocessing to accurately recognize static ISL alphabets. The CNN architecture includes a MobileNetV2 feature extractor, Global Average Pooling, Batch Normalization, Dense, Dropout, and Softmax layers. Training employed transfer learning, Early Stopping, Batch Normalization, Dropout, and data augmentation to improve generalization and prevent overfitting.

Experimental results demonstrate excellent performance, achieving approximately 99% training and validation accuracy and 99.90% testing accuracy with a loss value of 0.0292. Real-time testing confirmed fast and stable gesture recognition under proper lighting and clear hand positioning. Although performance decreases under poor lighting, cluttered backgrounds, or similar-looking gestures, the proposed system provides an effective foundation for future developments, including dynamic gesture recognition, word- and sentence-level ISL translation, speech synthesis, and advanced assistive communication systems.

Conclusion

This research journal paper presented a real-time static Indian Sign Language (ISL) alphabet gesture recognition system using Deep Learning and Computer Vision techniques. The developed system successfully recognizes static ISL alphabet gestures using a webcam input device and converts them into readable text output. The proposed MobileNetV2 model showed stable learning performance and achieved high prediction accuracy during training and testing. Experimental evaluation demonstrated effective classification performance with minimal overfitting and efficient feature extraction capability. Real-time testing also showed accurate gesture recognition under proper lighting conditions and clear hand positioning. MediaPipe hand tracking with webcam-based prediction improved the practical usability and efficiency of the system. Image preprocessing and data augmentation techniques also helped improve model generalization and recognition performance under different gesture variations and environmental conditions. The proposed prototype aims to reduce communication barriers between hearing and speech-impaired individuals and non-sign language users by providing a simple and efficient real-time ISL recognition solution. Although the system achieved high accuracy for static ISL alphabet recognition, the current implementation is limited to recognizing only static hand gestures. Dynamic gesture recognition, continuous sentence interpretation, and complete sign language translation are not currently supported. Future improvements can be done by modeling a dynamic gesture recognition system using sequence-based deep learning models such as LSTM and Transformer architectures, sentence-level ISL translation, speech synthesis integration, mobile application deployment, and larger dataset support for improved robustness and real-time communication performance. Additional enhancements may also include multilingual output support and improved performance under complex real-world environments.

References

[1] Rokade, Yogeshwar & Jadav, Prashant. (2017). Indian Sign Language Recognition System. International Journal of Engineering and Technology. 9. 189-196. 10.21817/ijet/2017/v9i3/170903S030. [2] Patil, Rachana & Patil, Vivek & Bahuguna, Abhishek & Datkhile, Gaurav. (2021). Indian Sign Language Recognition using Convolutional Neural Network. ITM Web of Conferences. 40. 03004. 10.1051/itmconf/20214003004. [3] Katoch, Shagun & Singh, Varsha & Tiwary, Uma Shanker. (2022). Indian Sign Language recognition system using SURF with SVM and CNN. Array. 14. 100141. 10.1016/j.array.2022.100141. [4] G, Dr & Goyal, Kaushal. (2023). Indian Sign Language Recognition Using Mediapipe Holistic. 10.48550/arXiv.2304.10256. [5] Rupali Kadwade, Akanksha Tangade, Neha Pakhare, Samiksha Kolhe, Hajara Waikar, S. J. Wagh, 2023, Indian Sign Language Recognition System, INTERNATIONAL JOURNAL OF ENGINEERING RESEARCH & TECHNOLOGY (IJERT) Volume 12, Issue 05 (May 2023), [6] Kumar, Anuj & Rawat, Prachi & Tamta, Vivek & Kumar, Papendra. (2025). A Comprehensive Approach to Indian Sign Language Recognition: Leveraging LSTM and MediaPipe Holistic for Dynamic and Static Hand Gesture Recognition. EAI Endorsed Transactions on AI and Robotics. 10.4108/airo.8693. [7] Vashisth, Harsh & Tarafder, Tuhin & Aziz, Rehan & Arora, Mamta. (2023). Hand Gesture Recognition in Indian Sign Language Using Deep Learning. Engineering Proceedings. 59. 96. 10.3390/engproc2023059096. [8] R. Awalkar, A. Sah, R. Barahate, Y. Kharche, and A. Magar, ‘Silent expressions: Two-handed Indian Sign Language recognition using MediaPipe and machine learning’, International Journal of Innovative Science and Research Technology (IJISRT), no. IJISRT25MAR598, pp. 587–595, Mar. 2025. [9] Sahil Sagar Khetam, Komal Murkute, Sagar Surve, Ruturaj Bhunje, Pratham Raut, and Anurag Kumar, “Indian Sign Language to Text/Speech Translation: A Deep Learning”, Int. J. Sci. Inno. Eng., vol. 2, no. 4, pp. 11–15, Apr. 2025, doi: 10.70849/IJSCI. [10] Rastogi, Umang & Mahapatra, Rajendra & Kumar, Sushil. (2025). Advanced gesture recognition in Indian sign language using a synergistic combination of YOLOv10 with Swin Transformer model. Scientific Reports. 15. 10.1038/s41598-025-18496-8. [11] Rushil Verma.\"Indian Sign Language Alphabet Dataset\" Kaggle. https://www.kaggle.com/datasets/rushilverma07/indian-sign-language-alphabet-dataset

Copyright

Copyright © 2026 Jeetumoni Barman, Nafisa Zaman, Nayana Das. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.