Intelligent System for Audio-to-Text and Text-to-Sign Conversion across Indian and American Sign Languages: Implementation and Evaluation

Abstract

Translating sign languages poses real hurdles from regional variations and the push for instant processing, particularly bridging Indian Sign Language (ISL) and American Sign Language (ASL). In this work, we roll out a fresh setup enabling two-way shifts between text or voice inputs and sign video outputs. Drawing on MediaPipe for pinpointing landmarks, SMPL-X for shaping poses, and Bezier interpolation to ease transitions, the system renders gestures letter by letter from a JSON pose database. It packs modular pieces like TextProcessor for breaking down text and MotionEngine for handling movement. Voice handling comes via Whisper transcription and TTS output. Overall, the build makes tweaks simple and opens doors for adding more sign languages down the road.

Introduction

The system describes a bidirectional sign language translation platform designed to bridge communication between Indian Sign Language (ISL) and American Sign Language (ASL). It aims to overcome regional differences in sign languages by enabling real-time conversion between text, speech, and sign language videos.

The proposed solution uses technologies like MediaPipe for real-time hand and body tracking, SMPL-X for 3D avatar-based pose generation, and a CNN model for recognizing sign gestures and converting them into text. It supports multiple input types, including typed text, spoken audio (processed using Whisper), and webcam-based gesture input, and outputs sign language animations, speech, or text.

The system is trained on datasets such as WLASL (ASL) and INCLUDE (ISL), achieving around 92% accuracy for sign recognition and improved performance with air-handwriting features. Motion smoothness is enhanced using Bezier curve interpolation, while Blender is used to render realistic avatar-based sign videos. The platform is built as a Streamlit web application using Python and a set of AI/ML tools for processing, modeling, and rendering.

The literature survey highlights progress in sign language recognition, including deep learning, pose estimation, datasets, and multimodal AI systems. It also identifies key technologies like MediaPipe, SMPL-X, Whisper, and Text-to-Speech systems that support modern sign translation pipelines, along with studies focusing on ASL–ISL bidirectional translation.

Conclusion

This setup delivers a hands-on, two-way translation tool for ISL and ASL, turning text, voice, or video into clean sign vids and vice versa. Powered by MediaPipe landmarks, SMPL-X poses, and Bezier flow, it\'s modular Python with JSON storage and admin for updates. Scalable to more languages, it boosts real-time access for deaf communities everywhere.

References

[1] M. Kumar, S. Sarvajit Visagan, T. Mahajan, A. Natarajan, and P. S. Sreeja, \"Enhanced Sign Language Translation Between American Sign Language and Indian Sign Language Using LLMs,\" IEEE Access, vol. 13, pp. 156270-156XXX, 2025, doi: 10.1109/ACCESS.2025.3595943. [2] S. Aher, P. Avhad, P. Gaikwad, G. Khandage, P. Pedhekar, “REVIEW PAPER ON AN INTELLIGENT SYSTEM FOR AUDIO-TO-TEXT AND TEXT-TO-SIGN TRANSLATION ACROSS INDIAN AND AMERICAN SIGN LANGUAGES,” Industrial Engineering Journal, ISSN: 0970-2555, Volume : 54, Issue 10, No.1, November, 2025. [3] S. Aher, Pratiksha Avhad, Pradnya Gaikwad , Gitanjali Khandage, Prajakta Pedhekar, “Intelligent System for Audio-to-Text and Text-to-Sign Translation Across Indian and American Sign Languages: Implementation and Evaluation,” International Journal for Research in Applied Science & Engineering Technology (IJRASET), vol. 14, no. III, pp. 4140–4144, Mar. 2026, doi: 10.22214/ijraset.2026.78038. [4] S. Patil, H. Chaudhari, H. Chaudhari, K. Kapase, S. Shinde, \"Air Handwriting using AI and MI,\" International Journal of Scientific Research in Engineering and Management (IJSREM), vol. 07, no. 11, pp. 1-4, Nov. 2023. [5] S. Patil, H. Chaudhari, H. Chaudhari, K. Kapase, S. Shinde, \"Air Handwriting using AI and ML,\" International Journal of Scientific Research in Engineering and Management (IJSREM), vol. 08, no. 05, pp. 1-5, May 2024. [6] S. Sharma, R. Kumar, and A. Singh, \"Real-Time Indian Sign Language Recognition Using MediaPipe and Deep Learning,\" IEEE International Conference on Computer Vision and Machine Learning, vol. 12, pp. 234-241, 2024. [7] A. Kumar and P. Singh, \"Bidirectional ASL-ISL Gesture Translation Using Pose Mapping and Sequence Modeling,\" Journal of Visual Communication and Image Representation, vol. 91, p. 103752, 2024. [8] D. Bragg, O. Koller, M. Bellard et al., \"The WLASL Dataset: A Large-Scale Word-Level American Sign Language Video Dataset,\" arXiv preprint arXiv:1910.11006, 2019. [9] ISLRTC, \"INCLUDE Dataset: Annotated ISL Video Corpus for Research,\" Ministry of Social Justice and Empowerment, Government of India, 2023. [10] G. Pavlakos, V. Choutas, N. Ghorbani et al., \"Expressive Body Capture: 3D Hands, Face, and Body from a Single Image,\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10929-10939, 2019. [11] Google Research, \"MediaPipe: Open Source Framework for Multimodal ML Pipelines,\" Google GitHub Repository, 2023. [12] Max Planck Institute, \"SMPL-X: A Unified Body Model for Humans,\" GitHub Repository, 2022. [13] OpenAI, \"Whisper: Robust Speech Recognition via Large-Scale Weak Supervision,\" arXiv preprint arXiv:2212.04356, 2022. [14] Coqui AI, \"Coqui TTS: Deep Learning Toolkit for Text-to-Speech,\" GitHub Repository, 2023 [15] R. Rastgoo, K. Nouri, and S. Escalera, \"Sign Language Recognition: A Deep Survey,\" Expert Systems with Applications, vol. 169, p. 114426, 2021. [16] S. Sharma, R. Kumar, and A. Singh, \"Real-Time Indian Sign Language Recognition Using MediaPipe and Deep Learning,\" IEEE International Conference on Computer Vision and Machine Learning, vol. 12, pp. 234-241, 2024 [17] A. Kumar and P. Singh, \"Bidirectional ASL-ISL Gesture Translation Using Pose Mapping and Sequence Modeling,\" Journal of Visual Communication and Image Representation, vol. 91, p. 103752, 2024. [18] S. Zhang, C. Zhu, J. K. O. Sin, and P. K. T. Mok, “A novel ultrathin elevated channel low-temperature poly-Si TFT,” IEEE Electron Device Lett., vol. 20, pp. 569–571, Nov. 1999.

Copyright

Copyright © 2026 Shital Aher, Pratiksha Avhad, Pradnya Gaikwad, Gitanjali Khandage, Prajakta Pedhekar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.