Indian Sign Language (ISL) Translator: AI-Powered Bidirectional Translation System

Abstract

This work presents an advanced AI-based translation system designed to bridge communication barriers for the Deaf and Hard-of-Hearing (DHH) community by converting spoken and textual language into Indian Sign Language (ISL) and vice versa. The system leverages deep learning techniques, including computer vision and natural language processing (NLP), to interpret hand gestures and facial expressions accurately. Integrated with real-time processing capabilities, the model enables seamless interaction between ISL users and non-signing individuals. By utilizing a custom-trained Transformer-based NLP model and a Convolutional Neural Network (CNN) for visual recognition, the system ensures accurate and efficient translation. The prototype has been developed using VS Code, with datasets managed in local storage to optimize performance. This work aims to enhance accessibility, promote inclusivity, and facilitate effortless communication through a robust and scalable ISL translation model. The importance of an efficient ISL translation system extends beyond accessibility—it fosters independence, enhances social inclusion, and bridges the gap between the DHH community and the hearing population. Many Deaf individuals struggle with traditional text-based communication due to differences in sentence structures and grammar between ISL and spoken languages. By incorporating deep learning models for gesture recognition and NLP-based translation, our system provides a user-friendly solution for effective communication. Additionally, this system has the potential to be implemented in educational institutions, workplaces, and public services, ensuring better integration of the Deaf community into society. By addressing existing gaps and leveraging AI, our translator serves as a critical step toward an inclusive digital ecosystem.

Introduction

The Indian Deaf community, with over 18 million members, faces significant communication barriers due to the lack of standardized Indian Sign Language (ISL) resources and technology. Existing sign language translation systems mainly support American or other regional sign languages, leaving ISL users underserved. To address this, an AI-driven ISL Translator is proposed for real-time, bidirectional translation between ISL and English/Hindi using computer vision (CNN, YOLO) and natural language processing (NLP).

Key points include:

• Literature survey: Current gesture recognition uses vision-based (cameras with CNNs, Transformers) and sensor-based systems (gloves, motion sensors). Vision-based methods are non-intrusive but challenged by variations in signing styles, lighting, and occlusions. Sensor-based systems are precise but costly and less accessible. There is a lack of comprehensive ISL datasets and difficulty in recognizing dynamic gestures, real-time processing delays, and subtle facial expressions.

• Proposed methodology: The system has three modules—Sign-to-Text (gesture recognition with CNN), Text-to-Sign (Transformer-based NLP for grammar and context), and a user-friendly web app interface for real-time interaction. The architecture processes inputs from cameras, microphones, and text, applying preprocessing, gesture recognition, translation, and output generation in text or animated signs.

• Dataset: A diverse, labeled dataset of 42,000 ISL images (alphabets and numbers) is used, enhanced through preprocessing (resizing, noise reduction, augmentation) for robust model training.

• Models: CNNs classify static signs, YOLO handles real-time dynamic gesture detection, and rule-based NLP methods translate text to ISL grammar. Transfer learning and data augmentation improve accuracy.

• Real-time processing: Optimizations like model quantization, efficient frame sampling, GPU acceleration, and multi-threading ensure minimal latency, achieving smooth, real-time translation.

• Web application: A Flask-based system interfaces with users through webcam and microphone inputs, processing gestures and speech to provide instant translation outputs on the frontend.

The goal is to bridge the communication gap for ISL users, promoting inclusivity by offering an accurate, efficient, and accessible translation tool.

Conclusion

The prototype for the real-time ISL translation system was developed using Visual Studio Code (VS Code), with model weights stored locally to optimize performance and minimize latency during real-time translation tasks. This approach ensures faster processing speeds, enhancing system responsiveness and making it suitable for practical use. Initial testing of the prototype demonstrated promising results, achieving an accuracy rate of 85% for single-word translations, indicating the system’s strong ability to recognize and interpret isolated gestures accurately. However, when dealing with more complex inputs, such as continuous sentence-level translations, the system\'s accuracy dropped to 78%, highlighting the challenges of interpreting dynamic gesture sequences and capturing contextual nuances inherent in ISL. Future iterations of the system will focus on expanding the dataset to incorporate a broader range of gestures, facial expressions, and contextual variations. Additionally, refining real-time tracking capabilities will help improve recognition of dynamic gestures, enhance processing speed, and boost overall translation accuracy. These improvements aim to make the system more robust and adaptable, ensuring effective communication for ISL users across various real-world scenarios.

References

[1] World Federation of the Deaf, \"Indian Sign Language Overview,\" 2024. [2] National Institute of Speech & Hearing, \"Challenges in ISL Adoption,\" 2023. [3] OpenAI Research, \"Transformer Models in NLP Translation,\" 2022. [4] Google AI, \"Computer Vision for Sign Language Recognition,\" 2024. [5] Sharma, R., & Kumar, A., \"Deep Learning Approaches for Gesture Recognition in Sign Language,\" Journal of Artificial Intelligence Research, 2023. [6] Patel, S., & Singh, N., \"Real-Time Sign Language Translation Using CNN-LSTM Networks,\" International Journal of Computer Vision and Pattern Recognition, 2022. [7] Rao, M., & Desai, P., \"Challenges in Dynamic Gesture Recognition for ISL,\" Indian Journal of Speech and Language Research, 2023. [8] International Conference on Assistive Technologies, \"Advancements in Real-Time Sign Language Translation Systems,\" 2024. [9] Kaur, J., & Verma, R., \"Facial Expression Recognition for Enhanced Sign Language Translation,\" IEEE Transactions on Affective Computing, 2023. [10] World Health Organization, \"Accessibility and Communication for the Deaf and Hard of Hearing,\" 2022. [11] Papatsimouli, M., & Kailas, A., \"Real-Time Sign Language Translation Systems: A Review Study,\" ResearchGate, 2022. https://www.researchgate.net/publication/362331604_Real_Time_Sign_Language_Translation_Systems_A_review_study [12] Kaur, M., & Kaur, A., \"Indian Sign Language Interpreter Using Image Processing and Machine Learning,\" ResearchGate, 2015. https://www.researchgate.net/publication/342502247_Indian_sign_language_interpreter_using_image_processing_and_machine_learning [13] Signapse AI, \"Sign Language Technology,\" 2024. https://www.signapse.ai/ [14] UCLA Newsroom, \"Wearable-Tech Glove Translates Sign Language into Speech in Real Time,\" 2020. https://newsroom.ucla.edu/releases/glove-translates-sign-language-to-speech [15] Sign Language Processing, \"Sign Language Processing Research,\" 2024. https://research.sign.mt/ [16] Awasthi, S., & Sahu, S., \"A Proposed Framework for Indian Sign Language Recognition,\" ResearchGate, 2014. https://www.researchgate.net/publication/263506784_A_proposed_framework_for_Indian_Sign_Language_Recognition [17] Closing the Gap, \"Real-Time Sign Language Translation,\" 2024. https://www.closingthegap.com/real-time-sign-language-translation/ [18] Scitepress, \"Indian Sign Language Recognition Using Fine-Tuned Deep Learning Models,\" 2021. https://www.scitepress.org/Papers/2021/107903/107903.pdf [19] MDPI, \"Translating Speech to Indian Sign Language Using Natural Language Processing,\" 2022. https://www.mdpi.com/1999-5903/14/9/253 [20] MDPI, \"A Survey of Advancements in Real-Time Sign Language Translators: Integration with IoT Technology,\" 2023. https://www.mdpi.com/2227-7080/11/4/83

Copyright

Copyright © 2025 Adithyaraaj R. P., Mariyammal N., Mohammed Furkhan S., Rathika , Prof. K. Vijayalakshmi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.