Novel CNN Based Model Using LSTM for Driver Drowsiness Detection

Abstract

Drowsiness affects human cognitive functioning, response time, and decision-making capacity, which is a major risk factor in road safety. Drowsiness-caused accidents are increasingly common, which indicates the need for real-timedrowsinessdetectionsystems.Thepurposeofthisstudyistodetectsignsof fatigue through facial characteristics such as eye closure, yawning, and tilting of the head in binary and multi-class classification functions. The study proposes a hybrid model, which integratesConvolutionNeural Networks (CNN) and Long Short-Term Memory (LSTM), which uses a camera-based real -time detection system with DLIB for face tracking. CNN extracts spatial functions, while LSTM preserves temporal dependencies and improves the accuracy of the detection.Theexperimentalresultmodelshowsa significant improvement in performance, receiving an accuracy of 96.3%, precision rate of 96.45%, recall rate of 96.33%, and F-measure of 96.32%. The proposed model optimizes feature selection, reduces false alarms, and increases reliability,makingthisastrongandeffectivesolutiontothedriver\'ssafetymonitoring systems.

Introduction

Drowsiness significantly impairs cognitive functions, attention, and productivity, contributing heavily to accidents, especially in driving. Traditional drowsiness detection methods (self-assessment, vital signs, or wearables) have limitations such as cost, discomfort, or inaccuracy. Visual cues like eye closure, blinking, yawning, and head movements offer non-invasive indicators of drowsiness. Advances in deep learning, especially combining Convolutional Neural Networks (CNNs) for spatial feature extraction and Long Short-Term Memory (LSTM) networks for temporal pattern recognition, enable accurate, automated, and real-time drowsiness detection.

The study proposes a hybrid CNN-LSTM model that leverages CNNs to analyze static facial features and LSTMs to track changes over time, improving early detection and reducing false positives. This model uses transfer learning to perform well with less labeled data and adapt to diverse real-world conditions, addressing challenges like lighting variations, face angles, and obstructions (e.g., masks or glasses).

A thorough literature review highlights various machine learning techniques, datasets, and their limitations. Many studies achieve high accuracy under controlled conditions but struggle with real-world variability. The hybrid CNN-LSTM approach is identified as particularly effective for capturing both spatial and temporal drowsiness indicators.

The experimental framework uses several datasets (Driver Drowsiness Dataset, MRLDataset, YAWDD) covering diverse conditions. The model pipeline includes face and landmark detection, CNN-based spatial feature extraction, followed by LSTM temporal analysis. Various classifiers were tested, with CNN-LSTM showing the best balance of accuracy and adaptability, supporting real-time monitoring and alert systems.

Performance metrics such as accuracy, precision, recall, and F1-score demonstrate the hybrid model’s superiority over standalone models (CNN, LSTM, SVM, etc.) across multiple datasets, emphasizing its suitability for practical, real-world applications to improve safety and productivity.

Conclusion

This study describes an efficient real-time driver drowsiness detection system with the help of a hybridCNN-LSTM model. CNNhelpedinextractingspatialfeaturesandLSTMwasusedtocapture.Thesystemis capable of successfully detecting signs for drowsiness, such as eye closure and yawning. This model produces results with high accuracy and has real-life applications for driver safety. To that end, the system includes a time camera to providealertstotheriskofdrivinginadrowsystate.Thisresearchhasopenedthe door for the outcomes presented here, with theaimofmakingroadssafer,andfuturedevelopmentsindriver monitoring systems. In the future, other datasets fromdifferentsourcescanbeusedtomakethemodelmore accurate. This system can also be applied to real-life scenarios by using real-life platforms like (Microsoft Teams and Netflix). Additional features will be used like haptic feedback whichwillhelpinenhancinguser safety and comfort. This system aims to provide solutions for practical use and adaptability to user\'s preferences and environment.

References

[1] NHTSA,“DrowsyDrivingandAutomobileCrashes,”NationalHighwayTrafficSafetyAdministration,Tech.Rep.,2017.[Online].Available:https://www.nhtsa.gov [2] T.A.Dingusetal.,“Drivercrashriskfactorsandprevalenceevaluationusingnaturalisticdrivingdata,”*ProceedingsoftheNationalAcademyofSciences*,vol.113,no.10,pp.2636–2641,2016 [3] M. Sahayadhas, K. Sundaraj, and M. Murugappan, “Detecting Driver Drowsiness Based on Sensors: A Review,” Sensors, vol. 12, no. 12, pp. 16937–16953, 2012. [4] Y. Jap, S. Lal, P. Fischer, and E. Bekiaris, “Using EEG Spectral Components to Assess Algorithms for Detecting Fatigue,” Expert Systems with Applications, vol. 36, no. 2, pp. 2352–2359, Mar. 2009. [5] M.Zhao,G.Zheng,X.Liu,andJ.Li,“DriverDrowsinessDetectionUsingElectroencephalogram Signals:AReview,”IEEETransactionsonNeural Systems and Rehabilitation Engineering, vol. 26, no. 4,pp.727–736,Apr.2018 [6] S.Abtahi,B.Hariri,andL.Shirmohammadi,“DriverDrowsinessMonitoringBasedonYawning Detection,” in Proc. IEEE Instrumentation and Measurement Technology Conference, May 2011, pp. 1–4. [7] P. Viola and M. Jones, “Robust Real-Time Face Detection,” International Journal of Computer Vision, vol. 57, no. 2, pp. 137–154, May 2004 [8] Y. Li, S. Zheng, J. Wang, and C. Chen, “Driver Drowsiness DetectionUsingFusionofCNNandLSTM Networks,” Multimedia Tools and Applications, vol. 78, no. 11, pp. 15057–15070, Jun. 2019 [9] Y. Li, S. Zheng, J. Wang, and C. Chen, “Driver Drowsiness DetectionUsingFusionofCNNandLSTM Networks,” Multimedia Tools and Applications, vol. 78, no. 11, pp. 15057–15070, Jun. 2019 [10] A.Krizhevsky,I.Sutskever,andG.E.Hinton,“ImageNetClassificationwithDeepConvolutional Neural Networks,” in *Advances in Neural Information Processing Systems*, vol. 25, 2012. [11] S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, Oct. 2010 [12] R. Daza, I. C. Daza, and R. T. Villar, “Challenges of Drowsiness Detection in Real Environments: A Review,” in *Proc. International Conference on Machine Vision and Applications*, 2019 [13] B. Wang, H. Wu, and Y. Li, “Cross-Dataset Driver Drowsiness Detection UsingTransferLearningand Spatio-Temporal Features,” IEEE Access, vol. 9, pp. 15078–15089, 2021 [14] M. Zhao et al., “A Deep Learning Framework for Real-Time Drowsiness Detection in Driver Monitoring Systems,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 2, pp. 1040–1052, Feb. 2021. [15] M. Sahayadhas, K. Sundaraj, and M. Murugappan, “Detecting DriverDrowsinessBasedonSensors:A Review,” Sensors, vol. 12, no. 12, pp. 16937–16953, Dec. 2012. [16] P. Singh, R. Gupta, and S.Sharma,“Real-TimeDriverDrowsinessDetectionSystemUsingEyeAspect Ratio and Facial Landmarks,” in Proc. IEEE ICCCNT, Jul. 2020, pp. 1–6 [17] R. Kumar, A. Verma, and P. Choudhury, “CNN-Based Driver Drowsiness Detection Using Facial Features,” in *International Journal of Intelligent Transportation Systems Research*, vol. 19, no. 3, pp. 415–424, Sep. 2021. [18] Liu,M.-Z.,Xu,X.,Hu,J.,Jiang,Q.-N.:Real-time detection of driver fatigue based on CNN-LSTM. IET Image Process. 16, 576–595 (2022). [19] Dipender,S.,Avtar,R.:Driverfatiguedetectionusingdeeplearningandimagepreprocessingtechniques. Neural Comput. Appl. 34, 453–467 (2023)[2]. [20] Swapnil,S.,Mansi,R.,Yash,P.:DetectionofdriverdrowsinessusingeyeaspectratioandHaar Cascade. Int. J. Comput. Sci. Appl. 15(2), 101-115 (2021)[3]. [21] John, J., Vijay, R., Nishanth, P.: Real-time detection of driver fatigue using deep learning and imageprocessing. Int. J. Comput. Vis. 33, 440–450 (2022)[4]. [22] Srinivasan, R., Mahesh, K., Suresh, P.: Detection of driver drowsiness using support vector machines and machine learning techniques. Int. J. Comput. Eng. 13(1), 25-35 (2021)[5] . [23] Chen,F.,Zhang,J.,Wang,X.:Ahybridmodelfordriverdrowsinessdetectionusingsensorfusion. IEEE Access 10, 134-142 (2023)[6] [24] Gupta, P., Sharma, N., Mehta, R.: Real-time fatigue detection system using CNNandfacialexpression analysis. Computer Vis. Image Underst. 191, 102900 (2020)[7]. [25] Kumar,A.,Sharma,V.,Kapoor,S.:Real-timedrowsinessdetectionusingaCNN-basedframework.Int.J.Artif.Intell.27,509-520(2023)[8]. [26] Mei, L., Zhou, Y., Zhang, Z.:Drowsinessdetectionbasedoneyemovementtrackinganddeeplearning. Computer Vis. Media 6(3), 410-417 (2020)[9] [27] Hassan, M., Shahnaz, M., Mahmud, R.: DetectingdriverdrowsinessusingLSTMnetworksandvehicle data. Autom. Sci. Eng. 8(2), 302-315 (2021)[10]. [28] Patel, D., Soni,A.:DriverdrowsinessdetectionusinghybridCNN-RNNmodels.ExpertSyst.Appl.72, 152-163 (2023)[11]. [29] Tan,H.,Wang,L.,Zhang,Y.:DriverdrowsinessdetectionbasedonHOGandSVM.J.Real-Time Image Process. 20(6), 1461-1470 (2022)[12] [30] Zhao,X.,Zhang,M.,Xie,Y.:Driverfatiguedetectionbasedonmultimodaldatafusionusingdeep learning. J. Sens. 2021, 1001-1010 (2021) [13]. [31] Wang,Z.,Liu,H.,Yu,W.:Gazedetectionanddriverdrowsinesspredictionusingconvolutional networks. J. Electr. Computer Eng. 2021, 119-130 (2021) [14] [32] Patel, P., Soni,S.:Evaluationofdrowsinessdetectionsystemusinganoveldatasetofphysiologicaldata and facial analysis. Int. J. Image Process. 8(4), 204-213 (2023) [15] [33] Zhu, X., Zhang, L., Yang, H.: Deep learning for driver drowsiness detection: A comparative study of CNN and LSTM. Neural Comput. Appl. 22(5), 183-194 (2021) [16] [34] Singh,P.,Sharma,V.:EEGsignal-baseddetectionofdriverdrowsiness.Neurocomputing401,1003-1011 (2022) [17] [35] Li, J., Chen, T., Xie, S.: Real-time driver drowsiness detection using deep convolutional networks. J.Comput. Vis. Image Process. 35(8), 2250-2260 (2023) [18] [36] Bai,J.,Xu,J.,Li,Y.:Facialemotionrecognition-baseddriverdrowsinessdetectionsystem.Int.J. Comput. Vision. 42(4), 457-469 (2020) [19] [37] R. Kumar, A. Verma, and P. Choudhury, “CNN-Based Driver Drowsiness Detection Using Facial Features,” *Int. J. of Intelligent Transportation Systems Research*, vol. 19, no. 3, pp. 415–424, 2021. [38] Y. Li, S. Zheng, J.Wang,andC.Chen,“DriverDrowsinessDetectionUsingFusionofCNNandLSTM Networks,” *Multimedia Tools and Applications*, vol. 78, no. 11, pp. 15057–15070, 2019 [39] M. N. Karthick, K. Gopalakrishnan, and M. Aramudhan, “Real-Time Drowsiness Detection Using CNN-LSTM with Driver Eye Closure and Yawning Analysis,” in *Proc. 2020 Int. Conf. on Signal Processing and Communication (ICSPC)*, pp. 247–252. [40] C. Wang et al., “YawDD: A Multi-modal Dataset for Driver Drowsiness Detection,” in *Proc. IEEE Intelligent Vehicles Symposium (IV)*, 2016, pp. 1–6 [41] M. Sahayadhas, K. Sundaraj, and M. Murugappan, “Detecting DriverDrowsinessBasedonSensors:A Review,” *Sensors*, vol. 12, no. 12, pp. 16937–16953, 2012 [42] M. Majumder, H. Behera, and D. K. Behera, \"An efficient hybrid deep learning model for driver drowsiness detection,\" Procedia Computer Science, vol. 167, pp. 1050–1059, 2020. [43] R.SinghandR.Kumari,\"CNN-LSTM-basedhybridmodelfordetectingdriverfatigueinreal-time,\"InternationalJournalofIntelligentSystemsandApplications,vol.12,no.6,pp.45–54,2020. [44] S. Singh and P. P. Roy, “Driver drowsiness detection based on spatio-temporal representation ofdriver video using deep CNN and LSTM,” Pattern Recognition Letters, vol. 138, pp. 494–501, Sep. 2020. doi: 10.1016/j.patrec.2020.07.022 [45] M. K. Hasan, M. M. Islam, M. T. Rahman, and M. A. Al Mahmud, “Real-time driver drowsiness detection using facial features based on deep learning,” Computers, vol. 10, no. 8, p. 92, Aug. 2021. doi: 10.3390/computers1008009 [46] V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 2014, pp. 1867–1874. doi: 10.1109/CVPR.2014.24 [47] A. Krizhevsky, I. Sutskever, and G. E.Hinton,“ImageNetclassificationwithdeepconvolutionalneural networks,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 25, 2012, pp. 1097–1105. [48] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997. doi: 10.1162/neco.1997.9.8.1735 [49] A. Bosman, \"Evaluation Metrics for Classification Models in Machine Learning,\" Journal of Data Science and Analytics, vol. 7, no. 2, pp. 123–130, 2020.

Copyright

Copyright © 2025 Tulip Gautam, Suhani Sharma, Snigdha Mishra, Jassica Rajora, Khyati Ahlawat. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.