Real Time Detection and Conversion of Gestures to Text and Speech to Sign System

Abstract

Millions of individuals throughout the world use sign language as their primary form of communication, particularly those who are deaf or hard of hearing. But because not everyone is proficient in sign language, it might be difficult to communicate and restrict access to information for sign language users This project represents a groundbreaking integration of technology in the field of communication and accessibility One of the many challenges is the ability to communicate with others through sign language. The goal of the system is to convert speech to text using NLP(natural language processing),Mediapipe for hand landmark detection and CNN for sign classification

Introduction

Sign language is a visual communication system used primarily by deaf and mute individuals, involving hand gestures, body movements, and facial expressions. This project focuses on Indian Sign Language, aiming to bridge the communication gap between hearing and non-hearing people through a real-time AI-based system that converts sign language into text using computer vision and machine learning.

The system has two main modules:

1. Speech to Sign Conversion: Spoken words are captured via microphone, transcribed into text using NLP, restructured to follow sign language grammar, and mapped to corresponding sign gestures for animation.

2. Gesture to Text Conversion: Real-time hand gestures are tracked using MediaPipe’s 21 hand landmarks, with a CNN model classifying these gestures to translate them into text displayed on screen.

The methodology involves speech recognition, text processing, gesture detection, CNN-based classification, and text generation to enable effective, real-time communication between spoken language and sign language.

Conclusion

This project has shown promising results,in converting spoken language to sign and sign to text.By leveraging NLP techniques such as POS tagging,lemmati-zation etc ,the system can preprocess the input audio and extract the meaningful features.The resulting text is then mapped to sign language gesture which can viewed by deaf people. This project successfully integrates several complex technologies, including natural language processing (NLP), speech-to-text conversion, gesture recognition using Media Pipe, movements of the hands and fingers.CNN is used for gesture classification and dynamic sign language animation. The result is a versatile and user-friendly platform that facilitates real-time communication between individuals who use sign language and those who do not, bridging a critical gap in interpersonal interactions.

References

[1] Koller, O., Ney, H., & Bowden, R. (2016). \"Deep Learning of Mouth Shapes for Sign Language.\" In Proceedings of the IEEE International Conference on Computer Vision Workshops. [2] Bragg, D., et al. (2019). \"Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective.\" The 21st International ACM SIGACCESS Conference on Computers and Accessibility. [3] McNeil, M. R. (Ed.). (2016). \"Clinical Management of Sensorimotor Speech Disorders.\" Thieme. [4] A. S. M. Miah, J. Shin, M. Al M. Hasan, M. A. Rahim, andY. Okuyama, ‘‘Rotation, translation and scale invariant sign wordrecognition using deep learning,’’Comput. Syst. Sci. Eng., vol. 44, no. 3,pp. 2521–2536, 2023. [5] A. S. M. Miah, J. Shin, M. A.M. Hasan, and M.A.Rahim, BenSignNet:Bengali sign language alphabet recognition using concatenated segmenta-tion and convolutional neural network,’’Appl. Sci., vol. 12, no. 8, p. 3933,Apr. 2022. [6] S. S. Rautaray and A. Agrawal, ‘‘Vision based hand gesture recognition forhuman computer interaction: A survey,’’Artif. Intell. Rev., vol. 43, no. 1,pp.1–54, Jan. 2015. [7] Parton, B. S. \"Sign Language Recognition and Translation: A Multidisciplined Approach from the Field of Artificial Intelligence.\" Journal of Deaf Studies and Deaf Education.

Copyright

Copyright © 2025 Mrs. D. Srivalli, Komma Mounika, Odugu Hemanth, Gompa Lokesh , Merugu Ratnakumar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.