Real-Time Bidirectional Indian Sign Language (ISL) Translation System for Deaf and Speech-Impaired Individuals Using IoT

Abstract

Communication barriers among hearing impaired and non-signing individuals remain a major challenge in India due to limited use of Indian Sign Language (ISL). Although many gesture recognition systems exist, most are unidirectional, rely on expensive hardware such as sensor-equipped gloves, or rely on cloud-based processing that limits use in rural areas. This paper proposes a real-time, fully offline, two-way ISL translation system using a low-cost Raspberry Pi 5. The solution integrates MediaPipe-based hand landmark tracking, CNN-based motion recognition, and offline speech-to-text, using the VOSK engine to convert ISL characters to text and spoken audio to text. The 7-inch LED screen provides immediate feedback to both hard-of-hearing and hearing-impaired users. The system achieved 92% accuracy for the static ISL alphabet, 95% accuracy for numeric gestures, and less than 1 second latency, demonstrating its feasibility as a portable and accessible communication aid.

Introduction

More than five million people in India depend on Indian Sign Language (ISL) for communication, yet most hearing individuals cannot understand it, creating major communication barriers in education, healthcare, and public services. Existing sign language translation systems have several limitations, including one-way communication, reliance on expensive wearable devices such as sensor gloves, dependence on cloud-based processing, and limited support for ISL due to the focus on American Sign Language (ASL) and British Sign Language (BSL). To address these challenges, the proposed project develops a portable, low-cost, camera-based bidirectional ISL translation system that operates entirely offline. The system combines MediaPipe hand landmark detection with a lightweight Convolutional Neural Network (CNN) for ISL-to-text translation and uses the VOSK offline speech recognition engine for speech-to-text conversion, making it suitable for deployment on resource-constrained devices such as the Raspberry Pi 5.

The literature review highlights previous research in sign language recognition and translation. Earlier CNN-based ISL recognition systems achieved good accuracy for static alphabet recognition but lacked support for dynamic gestures and required high computational resources. Other systems converted ISL to speech but only enabled one-way communication and depended on controlled environments. Deep learning approaches primarily focused on recognizing individual letters rather than complete words or sentences. MediaPipe has emerged as an efficient solution for real-time hand landmark detection on embedded devices by extracting twenty-one hand landmarks with low computational cost. In contrast, traditional OpenCV-based methods relying on contour detection and color segmentation perform poorly in cluttered or low-light conditions. Overall, existing systems are limited by hardware requirements, cloud dependency, or lack of bidirectional communication, establishing the novelty of the proposed offline two-way ISL communication system.

The proposed system consists of two primary modules. The ISL-to-Text/Speech Module captures live video using a camera connected to the Raspberry Pi. MediaPipe detects twenty-one hand landmarks representing fingertip positions, joints, and hand orientation. These landmarks are processed by a CNN trained on ISL alphabets, numeric signs, and commonly used gestures. Once a gesture is recognized, the corresponding text is displayed instantly on the screen, with optional speech output generated using the pyttsx3 text-to-speech engine. The system prioritizes fast and reliable text display to minimize delays and improve accessibility for both hearing and hearing-impaired users.

The Speech-to-ISL Module enables hearing individuals to communicate with ISL users by converting spoken language into text. Speech is captured through a standard USB microphone and processed entirely offline using the VOSK speech recognition engine. The recognized speech is displayed as text on the screen instead of using animated sign-language avatars, which are computationally expensive and often inaccurate. This design choice ensures faster processing, lower computational requirements, and greater communication clarity.

The overall system architecture is designed for efficient offline operation. Video and audio inputs from the camera and microphone are processed by MediaPipe, the CNN classifier, and the VOSK engine. The processed outputs are coordinated within the application layer and displayed in real time on a 7-inch LED screen. This lightweight architecture eliminates the need for internet connectivity, making the system suitable for rural and low-resource environments.

The system emphasizes accessibility, affordability, and practicality. By relying solely on a standard camera, it avoids expensive wearable sensors while maintaining real-time performance on the Raspberry Pi 5 without dedicated GPU hardware. It supports recognition of ISL alphabets and numeric gestures while consuming minimal power. Its portability and offline operation make it particularly useful in schools, hospitals, government offices, and rural communities where reliable communication assistance is required.

The implementation integrates both hardware and software components. The Raspberry Pi 5 serves as the central processing unit, connected to a camera module, USB microphone, and display. The software is developed in Python using OpenCV for video capture, MediaPipe for hand landmark extraction, TensorFlow or PyTorch for CNN-based gesture classification, VOSK for offline speech recognition, and pyttsx3 for speech synthesis. The CNN is trained on a combination of publicly available ISL datasets and custom-collected images covering alphabets (A–Z), numbers (0–9), and common gestures such as "hello," "thank you," "yes," "no," and "help." Data preprocessing, including landmark normalization, improves robustness against variations in hand shape, orientation, and lighting conditions.

Experimental evaluation on the Raspberry Pi 5 demonstrates promising performance. The system achieves 92% accuracy for ISL alphabet recognition with an average latency of 0.21 seconds, 95% accuracy for numeric gesture recognition with 0.18-second latency, and 90% accuracy for offline speech-to-text conversion with a response time between 0.4 and 1 second. MediaPipe performs consistently across different skin tones and background environments, while the Raspberry Pi maintains stable operation during extended use without overheating. However, recognition accuracy decreases by approximately 6% under poor lighting conditions, and the current system is limited to recognizing one gesture at a time rather than continuous signing or complete ISL sentences. Speech recognition accuracy also declines in noisy environments, indicating opportunities for future improvements through enhanced lighting adaptation, noise reduction techniques, and continuous sign language recognition. Overall, the proposed system provides an effective, low-cost, offline, and portable bidirectional communication solution that significantly improves accessibility for ISL users in real-world settings.

Conclusion

This paper presents a real-time, offline, two-way Indian Sign Language (ISL) translation system built on a Raspberry Pi 5, designed to bridge the communication gap between deaf and hearing people. Leveraging MediaPipe for hand and gesture tracking and VOSK for offline speech recognition, the system allows for seamless sign-to-text and speech-to-text translation without relying on sensors or an Internet connection. Future improvements aim to extend the capabilities of dynamic motion detection using LSTM or Transformer models, support for continuous signal sequences, rich vocabulary training, more robust noise-resistant speech-to-text performance, and deployment as an Android mobile app. Overall, the system provides an affordable, scalable solution suitable for classrooms, hospitals, government offices and rural communities across India.

References

[1] P. Jadhav, R. Dhok, “Real-Time ISL Recognition Using CNN,” IEEE, 2021. [2] V. Priyadharshini et al., “ISL to Speech Using CNN and TTS,” IRJET, 2020. [3] P. Bhardwaj et al., “Static ISL Gesture Recognition Using Deep Learning,” Springer, 2020. [4] R. Nandhini, T. S. Kumar, “Hand Gesture Recognition Using MediaPipe,” IJSREM, 2022. [5] Sneha R. et al., “Real-Time Translator for ISL Alphabets,” IJCA, 2021. [6] F. Zhang, V. Bazarevsky, I. Grishchenko, Y. Raveh, T. Vakunov, M. Grundmann and S. J. K. Arora, “MediaPipe Hands: On-Device Real-Time Hand Tracking,” arXiv preprint arXiv:2006.10214, 2020. [7] A. Graves, Supervised Sequence Labelling with Recurrent Neural Networks, Studies in Computational Intelligence, vol. 385, Springer, 2012. [8] A. G. Howard et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv preprint arXiv:1704.04861, 2017.

Copyright

Copyright © 2026 Nishitaa Raghuanth, Ritu M Shetty, Dr. Smitha Shekar B, Dr Harish G. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.