LIP Reading Using Neural Network and Deep Learning

Abstract

Lip reading, the task of inferring spoken words purely from visual observation of lip movements, remains a challenging problem even for trained human lip readers, who are typically able to correctly identify only about every second word. This paper presents an automated lip reading system built using Convolutional Neural Networks (CNNs) to classify spoken words from video sequences of a speaker\'s mouth region. The system uses a Haar Cascade classifier to localize the face and mouth region in each video frame, followed by a dlib-based facial landmark detector to extract a precise lip Region of Interest (ROI). The cropped and normalized lip images are then passed to a trained CNN, which was benchmarked against a Long Short-Term Memory (LSTM) network and a Temporal Convolutional Network (TCN) to model the sequential nature of lip movement, with the CNN architecture selected for deployment based on superior classification accuracy on the Lip Reading in the Wild (LRW) dataset. The trained model was integrated into a real-time application capable of capturing live webcam video, isolating the speaker\'s lip region frame-by-frame, and predicting the spoken word with an associated confidence score. Experimental evaluation on a ten-class subset of the LRW dataset shows that the proposed CNN model is able to reliably identify isolated words such as “Hello”, “Start”, “Stop”, and “Previous” in real time. The system demonstrates the practical feasibility of vision-only speech recognition and its potential applications in assistive hearing technology, security and surveillance, and human–computer interaction in audio-degraded environments.

Introduction

This study presents a real-time automated lip reading system based on deep learning, designed to recognize spoken words by analyzing lip movements from video input. Lip reading is a challenging perceptual task even for skilled human experts, who typically identify only about half of spoken words correctly. Advances in machine learning and deep neural networks have made it possible to automate this process with greater consistency and accuracy. Automated lip reading has numerous practical applications, including improving speech recognition in noisy environments, assisting people with hearing impairments, and extracting speech information from video recordings where audio is unavailable or corrupted.

The primary challenge in visual speech recognition arises from the variability in languages, accents, speaking styles, and individual lip movements, all of which affect the appearance of spoken words. To address these challenges, the study employs Convolutional Neural Networks (CNNs) for learning discriminative visual features from cropped lip images and compares their performance with temporal sequence models, including Long Short-Term Memory (LSTM) networks and Temporal Convolutional Networks (TCNs). Following comparative evaluation, the CNN architecture demonstrated superior performance for the selected task and was integrated into a real-time web application capable of recognizing spoken words directly from a webcam.

The literature review highlights recent advances in lip reading research. Transformer-based attention mechanisms have improved the modeling of long-range dependencies in lip movements, while TCNs have demonstrated better temporal modeling than conventional CNN-only approaches. Transfer learning using large benchmark datasets such as LRW-1000 and LRS3 has enhanced model performance by reducing training data requirements. Researchers have also explored multimodal systems that combine visual lip information with audio and contextual cues to improve robustness in noisy environments.

Other studies have focused on practical deployment and robustness. Model compression techniques such as quantization and pruning enable real-time inference on mobile and edge devices. Adversarial training and synthetic data augmentation have been used to improve resistance to occlusions, articulation variations, and limited training data. Three-dimensional CNNs have enhanced joint spatial-temporal feature extraction, while cross-language adaptation has improved recognition across different languages and accents. Interactive lip reading systems providing real-time user feedback have also been proposed. In addition, researchers have emphasized the ethical implications of lip reading technology, highlighting the importance of informed consent, privacy protection, and safeguards against misuse in surveillance applications.

Despite these advances, the study identifies an important research gap. Most existing lip reading systems are evaluated offline using curated benchmark datasets and are rarely deployed as responsive real-time applications. This project addresses that limitation by integrating a lightweight CNN classifier with OpenCV and dlib-based lip detection in a live webcam system optimized for responsiveness and practical usability.

The proposed system follows a sequential processing pipeline. Video frames are first captured from a webcam or prerecorded video. A Haar Cascade face detector identifies the face within each frame, after which a dlib 68-point facial landmark detector isolates the mouth region using landmarks corresponding to the lips. A MedianFlow tracker maintains continuous tracking of the mouth across consecutive frames, reducing computational overhead and ensuring stable localization. The extracted lip images are resized to 64 × 64 pixels, normalized, and prepared for classification.

The model was trained using a subset of the Lip Reading in the Wild (LRW) dataset containing ten target word classes: Begin, Choose, Connection, Navigation, Next, Previous, Start, Stop, Hello, and Web. Images were resized to a uniform 64 × 64 × 3 resolution, converted into NumPy arrays, normalized to pixel values between 0 and 1, randomly shuffled, and encoded using one-hot class labels before training.

Three deep learning architectures were implemented and compared. The Convolutional Neural Network (CNN) consisted of multiple convolutional and max-pooling layers followed by fully connected layers using ReLU and softmax activation functions. The Long Short-Term Memory (LSTM) model employed stacked recurrent layers with dropout regularization to capture temporal dependencies across sequences of lip movements. The Temporal Convolutional Network (TCN) used one-dimensional dilated convolutions with varying dilation rates to model long-range temporal relationships within sequential visual features. Among these models, the CNN achieved the best balance between recognition accuracy and computational efficiency, making it the preferred architecture for deployment.

Training was performed using the Adam optimizer, categorical cross-entropy loss, a batch size of 16, and 250 training epochs. The CNN accepted input images of size 64 × 64 × 3, with classification accuracy serving as the primary evaluation metric. After training, the model architecture and learned weights were saved, allowing efficient reuse during real-time inference without retraining.

For deployment, the trained CNN was integrated into a live webcam application developed using OpenCV. During operation, each video frame undergoes face detection, lip localization, image preprocessing, and classification. Predicted class probabilities are compared against a confidence threshold of 0.98. Predictions exceeding this threshold are displayed as recognized words overlaid on the live video, while lower-confidence predictions are labeled as unidentified, reducing false-positive classifications and improving system reliability.

Experimental results demonstrate the effectiveness of the proposed approach. The CNN achieved 100% training accuracy on the selected ten-word LRW dataset and successfully recognized words such as "Web," "Previous," "Hello," and "Start" during real-time webcam testing. Correct predictions generally produced confidence scores greater than 0.98, while ambiguous or partially occluded lip images were appropriately rejected rather than incorrectly classified. This confidence-based filtering significantly reduced false predictions and enhanced practical usability.

The study also observed that recognition performance depends heavily on dataset size and diversity. Because the model was trained on a relatively limited subset of the LRW dataset, occasional misclassifications occurred, particularly among visually similar words with comparable lip movements. These findings align with previous research demonstrating that larger and more diverse training datasets substantially improve model generalization and recognition accuracy.

Conclusion

This paper presented the design, implementation, and evaluation of an automated lip reading system based on Convolutional Neural Networks. By combining Haar Cascade-based face detection, dlib facial-landmark-based lip localization, and a trained CNN classifier, the system is able to identify spoken words from visual lip movement alone and deliver predictions in real time through a webcam-based application. Comparative evaluation against LSTM and TCN architectures confirmed the CNN\'s suitability for this task in terms of both accuracy and inference speed. The results affirm the practical viability of vision-only speech recognition for applications in assistive hearing technology, human–computer interaction, and security systems where audio signals are unreliable or unavailable. Future work will focus on expanding the training dataset to cover a substantially larger vocabulary and a more diverse population of speakers, languages, and accents, integrating temporal architectures such as LSTM, TCN, or Transformer-based attention mechanisms to model full phrases and sentences rather than isolated words, and optimizing the trained model for deployment on resource-constrained edge devices to broaden real-world accessibility.

References

[1] Chen, X. et al. “Lip Reading with Transformers: Exploring the Potential of Attention Mechanisms.” 2023. [2] Zhu, Y. et al. “Temporal Convolutional Networks for Enhanced Lip Reading Performance.” 2023. [3] Bae, S. et al. “Transfer Learning for Lip Reading: Leveraging Large-Scale Datasets.” 2023. [4] Hsu, W. et al. “Multi-Modal Lip Reading: Combining Visual and Audio Data for Robust Predictions.” 2023. [5] Kumar, R. et al. “Real-Time Lip Reading on Edge Devices: Optimization Techniques.” 2023. [6] Shao, L. et al. “Ethical Considerations in Lip Reading Technology: Ensuring Privacy and Responsible Use.” 2023. [7] Yang, F. et al. “Adversarial Training for Robust Lip Reading Models.” 2023. [8] Li, H. et al. “Synthetic Data Augmentation for Lip Reading: Generating Diverse Lip Movements.” 2023. [9] Zhang, Q. et al. “3D-CNNs for Spatio-Temporal Lip Reading: Enhancing Feature Extraction.” 2022. [10] Wang, Y. et al. “Cross-Language Lip Reading: Adapting Models to Diverse Languages and Accents.” 2022. [11] Lee, J. et al. “Lip Reading in Noisy Environments: Enhancing Model Resilience.” 2022. [12] Cheng, Z. et al. “Interactive Lip Reading Systems: Real-Time Feedback and User Interaction.” 2022. [13] Goodfellow, I.; Bengio, Y.; Courville, A. “Deep Learning.” MIT Press, 2016.

Copyright

Copyright © 2026 Bandari Arjun, Mohammed Abdul Saad, Aaron Biju, G Udaya Sree. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.