Hand Gesture Recognition for Sign Language Using ML

Abstract

Sign language is a main mean of communication among the people who are hearing and speech impaired. In spite of its significance, there is communication void between users of sign language and the general population. The study is aimed at creating a solution, which will automatically identify the sign language gestures based on the artificial intelligence approach. The suggested system is based on the image processing and deep learning models to identify hand gestures and translate them into the readable text. MobileNetV2 is a pre-trained deep learning model that is used with Teachable Machine to enhance the accuracy and allow real-time gesture recognition. The system is designed to offer a convenient, effective and easily accessible communication solution.

Introduction

The text describes a machine learning-based Sign Language Recognition (SLR) system designed to bridge communication gaps between deaf/mute individuals and people who do not understand sign language. It highlights how advancements in AI and computer vision enable real-time translation of hand gestures into text, improving accessibility and communication.

Earlier approaches relied on rule-based and traditional machine learning methods, which required manual feature extraction and had limited accuracy. Modern systems use deep learning, especially Convolutional Neural Networks (CNNs), which automatically learn features from images and improve recognition performance. SLR systems are generally camera-based or sensor-based, though challenges like lighting changes, complex backgrounds, and motion variations still affect accuracy.

The proposed system collects a large dataset of 35 gesture classes (A–Z and 0–9) using webcam images. The data is preprocessed through resizing, normalization, and noise removal to improve model quality. A lightweight CNN model, MobileNetV2, is used with transfer learning via Teachable Machine to enable efficient and real-time gesture recognition.

The model is trained and tested using a split dataset, then deployed for real-time prediction where captured gestures are processed and converted into readable text. The system uses a webcam-based architecture where images are classified by the CNN and displayed as text output.

Conclusion

Sign Language Recognition systems are a great way of enhancing communication among the hearing and speech impaired individuals. Gesture recognition can be accomplished with high accuracy and in real-time because of the incorporation of deep learning techniques. Teachable machine + MobileNetV2 is a practical and efficient solution to gesture recognition systems.

References

[1] arons, D. (1994). Aspects of the Syntax of American Sign Language. PhD dissertation, Boston University.Google Scholar [2] Aarons, D., Bahan, B., Kegl, J. & Neidle, C. (1992). Clausal structure and a tier for grammatical marking in American Sign Language. Nordic Journal of Linguistics, 15, 103–142.CrossRefGoogle Scholar [3] Aarons, D., Bahan, B., Kegl, J. & Neidle, C. (1995). Lexical tense markers in American Sign Language. In Emmorey, K. & Reilly, J. S. (eds.), Language, Gesture and Space (pp. 225–253). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar [4] Abdel-Fattah, M. (2005). Arabic Sign Language: A perspective. Journal of Deaf Studies and Deaf Education, 10, 212–221.CrossRefGoogle ScholarPubMed [5] Abercrombie, D. (1967). Elements of General Phonetics. Edinburgh: Edinburgh University Press.Google Scholar [6] Aboh, E. O., Pfau, R. & Zeshan, U. (2005). When a wh-word is not a wh-word: The case of Indian Sign Language. In Bhattacharya, T. (ed.), The Yearbook of South Asian Languages and Linguistics 2005 (pp. 11–43). Berlin: Mouton de Gruyter.Google Scholar [7] Adams, M. (2003). Historia de la Educación de los Sordos en México y Lenguaje por Señas Mexicano. Spring Valley, CA: Fundación de Sordos Hispanos de San Diego.Google Scholar [8] Aitchison, J. (1991). Language Change: Progress or Decay? Cambridge: Cambridge Unversity Press.Google Scholar [9] Ajavon, P. A. (2003). The Incorporation of Nigerian Signs in Deaf Education in Nigeria: A Pilot Study. Frankfurt am Main. Brussels: Peter Lang.Google Scholar [10] Ajello, R., Mazzoni, L. & Nicolai, F. (2001). Linguistic gestures: Mouthing in Italian Sign Language (LIS). In Braem, P. Boyes & Sutton-Spence, R. (eds.), The Hands Are the Head of the Mouth: The Mouth as an Articulator in Sign Languages (pp. 231–246). Hamburg: Signum.Google Scholar [11] Akach, A. O. (1993). Squibbles barriers. Sign Post, 6(1), 2–4.Google Scholar [12] Aldersson, R. & McEntee-Atalinanis, L. J. (2007). A lexical comparison of Icelandic and Danish Sign Language. Birkbeck Studies in Applied Linguistics, 2, 41–67.Google Scholar [13] Aldrete, G. S. (1999). Gestures and Acclamations in Ancient Rome (Ancient Society and History). Baltimore, MD/London: Johns Hopkins University Press.Google Scholar [14] Allen, G. D., Wilbur, R. B. & Schick, B. S. (1991). Aspects of rhythm in ASL. Sign Language Studies, 72, 297–320.CrossRefGoogle Scholar [15] Allott, R. (2000). Brain, Lexicon, Syntax. Rutgers, NJ: Language Origins Society.Google Scholar [16] ,American School for the Deaf (1818). Second Report of the Directors of the Connecticut Asylum for the Education and Instruction of Deaf and Dumb Persons. 5. Hartford: American School for the Deaf.Google Scholar [17] Anderson, J. M. (1971). The Grammar of Case: Towards a Localist Theory. London: Cambridge University Press.Google Scholar [18] Ann, J. (1993). A linguistic investigation of the relationship between physiology and handshape. PhD dissertation. University of Arizona, Tucson. [19] Ann, J. (1998). Contact between a sign language and a written language: Character signs in Taiwan Sign Langauge. In Lucas, C. (ed.), Pinky Extension and Eye Gaze: Language Use in Deaf Communities (pp. 59–99). Washington, DC: Gallaudet University Press.Google Scholar [20] Ann, J. (2006). Frequency of Occurrence and Ease of Articulation of Sign Language Handshapes: The Taiwanese example. Washington, DC: Gallaudet University Press.Google Scholar [21] Aramburo, A. (1989). Sociolinguistic aspects of the Black Deaf community. In Lucas, C. (ed.), The Sociolinguistics of the Deaf Community (pp. 103–119). San Diego, CA: Academic Press.CrossRefGoogle Scholar

Copyright

Copyright © 2026 Arjun Vinod Shinde, Aditya Jajoo, Aditya Jha, Ayush Kumar Shrivastava, Yugal Kishor. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.