AI-Driven Lip-Reading System

Abstract

AI Based Visualization System for Displaying Text- to-Speech and Sign Language to Individuals with Hearing Impairment is an AI/Machine Learning and Computer Vision based system that recognises silent moving lips and translates them into readable text so that people with hearing impairment can communicate with people verbally, without needing to rely on auditory signals. The system uses video from a webcam or cellphone camera to capture images and analyse those images by first identifying sub-regions of interest around the mouth and then applying a predefined series of processes to bring all frames into a common standard form with regard to lighting, frame size, and frame frequency (e.g. frame rate, etc.). Second, it uses a Hybrid Architecture made up of Convolutional Neural Networks (CNNs) to extract spatial features of the videos as part of the input to a Sequential Model (Long Short-Term Memory network [LSTM] and/or Transformer) to model the temporal evolution of the visual representations of the speaking person’s lip movements and then generate temporal-level transcriptions of those movements (in character and/or word format). The training of this model has been based on benchmark datasets (GRID and LRW) designed to capture multiple people’s speaking patterns under different conditions. Evaluation metrics include—Word Accuracy, Character Accuracy, and Word Error Rate—allowing for quantifying performance of the lip reading model. To provide real time access for lip reading support, a web-based user interface will be established, allowing streaming of live video along with displaying recognised text and confidence scores, plus optional visualisation of the attention of the model over key frames to improve overall interpretability and user confidence in the output.

Introduction

The text presents an AI-driven lip-reading system designed to reduce communication barriers for individuals who are hard of hearing, especially in situations where audio is unavailable, unreliable, or obstructed by noise. Traditional alternatives such as text, video, audio, or manual lip-reading require training and are often ineffective in fast-paced, real-world settings. Advances in computer vision and deep learning now enable accurate visual-only speech recognition by interpreting lip movements from video.

The proposed system uses a modular, web-based architecture combining CNNs, LSTMs, and Transformer models to convert sequences of mouth movements into text in real time. A user-friendly front end supports live webcam streaming or video uploads, while a Python-based backend performs face detection, mouth localization, visual preprocessing, and deep-learning inference. Secure cloud storage enables session tracking, analytics, and continuous model improvement.

The system is trained and evaluated using benchmark datasets such as GRID and LRW, along with custom datasets collected from real-world environments. Extensive preprocessing—including frame extraction, normalization, temporal alignment, and data augmentation—ensures robustness across speakers, lighting conditions, and recording devices.

Results show high character- and word-level recognition accuracy (over 90%), low latency, and strong performance across diverse conditions, with most errors arising from visually similar lip movements. The system outputs live captions, session transcripts, visual attention feedback, and supports user correction, enabling continuous learning and improved accessibility.

Conclusion

AI-Powered Lip Reading to bridge communication for peo- ple with hearing loss is an application of AI and Machine Learning (ML) and Speech-to-Text using computer vision to convert non-verbal gestures into written text. As an AI- powered system that integrates computer vision and machine learning (ML), the AI-Powered Lip Reader uses facial land- marks to detect the mouth, calculates the 3D coordinates of the facial landmark, does preprocessed videos, trains convolutional neural networks and long short-term memory (CNN-LSTM) to recognize visual speech (lip movements) and converts that visual speech to text via a web-based user-friendly interface through the web-based platform of the system. The results from testing the system on several public datasets demon- strated reasonable levels of accuracy and latency, particularly when it is used with a highly constrained vocabulary and formatted sentences. Therefore, the AI-Powered Lip Reading system is expected to improve the accessibility of people with hearing loss in educational institutions, workplaces, and government entities. In addition to providing assistive capability for its in- tended audience, this framework will also provide insight into developing more advanced visual speech interfaces to support different modalities of silent communication, develop capabilities in a noisy environment of industrial settings, and facilitate the interaction between humans and computers in the work environment or everyday life. With its modular design, interpretability of functionality, and the ability to work within existing standardized software applications and toolchains, the AI-Powered Lip Reading system provides an exciting and informative resource for students, researchers, and industry professionals to learn about the technologies available to create AI-Enabled Solutions for Accessibility.

References

[1] Geetha C. et al., 2024 (CONIT Conference). AI Lip Reader Detecting Speech Visual Data with Deep Learning. [2] Wang et al., 2024. Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert. [3] Yue Cao & Wei Qi Yan. (2023). The Lips Reading Using Deep Learning Model. [4] Sahed et al., 2025 – Data in Brief. LipBengal: Pioneering Bengali Lip- Reading Dataset. [5] Varshney, S., & Kapoor, R. (2022). Deep Learning in Image Classifica- tion using VGG-19 and Residual Networks. [6] Liu, W., She, T., Liu, J., Li, B., Yao, D., Liang, Z., & Wang, R. (2024). [7] Lips Are Lying: Spotting the Temporal Inconsistency in DeepFake Lip Syncing. NeurIPS 2024. [8] Exarchos, T., Dimitrakopoulos, G. N., Vrahatis, A. G., Chrysovitsiotis, G., Zachou, Z., & Kyrodimos, E. (2024). Lip-Reading Advancements: A 3D CNN/LSTM Fusion for Word Recognition. [9] Kumar, A., Nair, R., & Mehta, S. (2025). Real-Time Lip Reading Using Lightweight CNN and Temporal Attention. Journal of Intelligent Systems and Applications. [10] Nair, R., & Mehta, S. (2025). Cross-Lingual Visual Speech Recognition with Transformer-Based Encoders. Pattern Recognition Letters. [11] Cao, Y., & Yan, W. Q. (2024). LipReader++: 3D CNN and Transformer Fusion for Robust Lip Reading. In Lips Reading Using Deep Learning Architecture.

Copyright

Copyright © 2026 Mrs. Chetana K N, Mahesh , Prajwal A R , Poorvith R. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.