Real-Time Interview Analysis using Deep Learning

Abstract

Traditional interview processes often suffer from subjectivity and lack real-time feedback, limiting effective candidate evaluation. Existing automated systems typically focus on either textual or behavioral analysis without integrating both modalities in real time. This paper presents a multimodal deep learning-based framework for real-time interview analysis that combines facial emotion recognition with adaptive question generation. The system uses computer vision to detect facial emotions from live video and a natural language processing module to generate context-aware interview questions. A decision layer integrates emotional cues with conversational flow to dynamically adjust question difficulty and relevance, while automated evaluation is performed through emotion trend and response analysis. Results indicate improved interactivity and more meaningful feedback, making the system suitable for recruitment, mock interviews, and skill assessment applications.

Introduction

This paper presents a multimodal deep learning-based framework for real-time interview analysis that combines facial emotion recognition and natural language processing (NLP) to create an intelligent, adaptive interview environment. Traditional interviews often suffer from subjectivity, inconsistency, and lack of structured feedback, leading to biased assessments and limited opportunities for candidate improvement. Existing AI-based interview systems typically focus on either textual responses or behavioral analysis separately, making comprehensive real-time evaluation difficult.

To address these limitations, the proposed system integrates facial emotion recognition, adaptive question generation, sentiment analysis, and automated performance evaluation into a unified framework. A CNN-based facial emotion recognition model analyzes live video streams to detect emotions such as confidence, stress, happiness, surprise, and nervousness, while NLP techniques evaluate candidate responses and generate context-aware interview questions. The system dynamically adjusts question difficulty based on both emotional state and response quality, creating a personalized interview experience.

The framework performs multimodal analysis by combining visual cues from facial expressions with textual information from candidate responses. It evaluates communication skills, emotional stability, response relevance, and behavioral consistency to generate objective performance assessments and real-time feedback. Candidates receive insights about confidence levels, communication effectiveness, strengths, and areas for improvement.

The methodology incorporates several AI technologies, including CNNs for facial analysis, transformer-based language models for question generation, supervised learning using labeled interview datasets, sentiment analysis for response evaluation, and computer vision techniques for continuous behavioral monitoring. A web-based interface developed using HTML, CSS, and JavaScript enables smooth interaction between users and the system.

The literature review highlights previous research in personality prediction, sentiment analysis, multimodal interview assessment, adaptive questioning, and fairness-aware AI systems. While earlier studies demonstrated the benefits of visual analysis, NLP, or multimodal evaluation individually, most lacked real-time integration of emotional feedback with adaptive interviewing. The proposed framework fills this gap by combining both aspects into a single real-time system.

The study emphasizes several advantages of AI-powered interview analysis, including reduced human bias, improved consistency, scalable recruitment processes, enhanced candidate preparation, personalized interview experiences, and intelligent real-time decision-making. Compared to traditional interviews that rely on manual evaluation and fixed questioning, the proposed system offers automated assessment, adaptive interaction, and instant feedback.

Future enhancements include integrating voice and speech emotion analysis, advanced audio-video behavioral monitoring, and customizable interview domains such as technical, HR, and aptitude interviews. Overall, the proposed framework demonstrates how AI, deep learning, computer vision, and NLP can be combined to create a more objective, interactive, and efficient interview evaluation system for recruitment, mock interviews, and skill assessment applications.

Conclusion

The proposed Real-time Interview Analysis using Deep Learning system successfully combines facial emotion recognition, natural language processing, and adaptive question generation to create an intelligent and interactive interview environment. The system is capable of analyzing candidate emotions in real time through computer vision techniques while simultaneously generating context-aware interview questions using NLP models. By integrating both visual and textual analysis, the framework provides a more comprehensive and objective evaluation compared to traditional interview methods. The developed model improves interview assessment by reducing human bias, increasing consistency, and delivering instant performance feedback. The adaptive questioning mechanism enhances user engagement by adjusting question difficulty based on candidate responses and emotional behavior. Experimental implementation demonstrates that the system performs efficiently in real-time conditions and can support applications such as recruitment, mock interviews, skill assessment, and training platforms. Overall, the project highlights the potential of deep learning and multimodal AI systems in transforming conventional interview processes into intelligent, automated, and data-driven evaluation systems. Future enhancements such as voice emotion analysis, multilingual support, and advanced behavioral analytics can further improve the effectiveness and scalability of the system.

References

[1] Yi-Chi Chou, Felicia R. Wongso, Chun-Yen Chao, and Han-Yen Yu, “Mock-Interview Platform (MIP): An AI-Assisted Virtual Interview System” Digital Education Institute, Institute for Information Industry, Taipei, Taiwan, 2023. https://ieeexplore.ieee.org/document/9778999 [2] P. Salehi, S. Z. Hassan, G. A. Baugerud, M. Powell, M. S. Johnson, D. Johansen, S. S. Sabet, M. A. Riegler, and P. Halvorsen, “A Theoretical and Empirical Analysis of 2D and 3D Virtual Environments in Training for Child Interview Skills” SimulaMet, UiTThe Arctic University of Norway, OsloMet, and Griffith University, 2024 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10634162 [3] F. Nagasawa, S. Okada, T. Ishihara, and K. Nitta, “Adaptive Interview Strategy Based on Interviewees’ Speaking Willingness Recognition for Interview Robots” IEEE Trans. Human-Mach. Syst., 2024.. https://ieeexplore.ieee.org/document/10234087 [4] G. ShivajiRao, M. Jaiganesh, M. A. K., M. A. A. A., N. R., and P. K. Parida “AI Powered Virtual Job Interview Simulator Using Natural Language Processing” Karpagam College of Engineering, Coimbatore, India, 2024 https://ieeexplore.ieee.org/document/11013504 [5] M. K. Mythili, K. M. Krithika, J. A. Jamuna, and I. S. Indhulega, “AI-Based Mock Interview Application” Department of Information Technology, Sri Krishna College of Technology, Coimbatore, India, 2024. https://ieeexplore.ieee.org/document/11012461 [6] M. Fathima, D. P. Dhinakaran, T. Thirumalaikumari, S. R. Devi, B. M. R. B. Putra, K. Azizah, C. O. Mawalim, I. A. Hanif, S. Sakti, C. W. Leong, and S. Okada, “MAG-BERT-ARL for Fair Automated Video Interview Assessment” Universitas Indonesia; Japan Advanced Institute of Science and Technology; Nara Institute of Science and Technology; Educational Testing Service, 2024 https://ieeexplore.ieee.org/document/10704666 [7] S. Artiran, R. Ravisankar, S. Luo, L. Chukoskie, and P. Cosman, “Measuring Social Modulation of Gaze in Autism Spectrum Condition With Virtual Reality Interviews” IEEE Trans. Affective Comput., 2023 https://www.researchgate.net/publication/362704971 [8] M. Jayaratne and B. Jayatilleke, “Predicting Personality Using Answers to Open-Ended Interview Questions” PredictiveHire Pty Ltd. and La Trobe University, 2023 https://ieeexplore.ieee.org/document/9121971 [9] S. Sivaramakrishnan, F. Z. Minni, A. Anand, A. Sahoo, and B. Hemang, “Real Time Mock Interview Evaluation using CNN” New Horizon College of Engineering, Bengaluru, India, 2024 https://ieeexplore.ieee.org/document/10503311/ [10] S. M. A. Shah, D. Sundmark, B. Lindström, and S. F. Andler, “Robustness Testing of Embedded Software Systems: An Industrial Interview Study” Swedish Institute of Computer Science and Mälardalen University, 2023. https://ieeexplore.ieee.org/document/7438745 [11] H.-Y. Suen, K.-E. Hung, and C.-L. Lin, “TensorFlow-Based Automatic Personality Recognition Used in Asynchronous Video Interviews,” National Taiwan Normal University and National Chengchi University, 2023.Video Interviews\" National Taiwan Normal University & National Chengchi University, 2023. https://ieeexplore.ieee.org/document/8660507 [12] C. Kim, J. Choi, J. Yoon, D. Yoo, and W. Lee, “Fairness-Aware Multimodal Learning in Automatic Video Interview Assessment” Dongguk University, South Korea, 2023 https://ieeexplore.ieee.org/document/10287972 [13] S. P., “Effectual Contract Management and Analysis with AI-Powered Technology: Reducing Errors and Saving Time in Legal Document” in Proc. 2024 9th Int. Conf. Science Technology Engineering and Mathematics (ICONSTEM), Chennai, India, 2024, pp. 1–6 [14] https://ieeexplore.ieee.org/document/10568733S. Khan, M. Zakir, S. Bashir, and R. Ali, “Artificial Intelligence and Machine Learning in Legal Research: A Comprehensive Analysis” Qlantic J. Social Sciences, vol. 5, pp. 307–317, 2024 https://www.researchgate.net/publication/378690075 [15] R. S. Rana, S. Singh, M. Aggarwal, and M. Badoni, “Unveiling the Future: Exploring AI Applications in the Indian Judicial System” in Proc. 2023 11th Int. Conf. Intelligent Systems and Embedded Design (ISED), Dehradun, India, 2023, pp. 1–5. https://ieeexplore.ieee.org/document/10444600 [16] S. Sharma, S. Srivastava, P. Verma, A. Verma, and S. Chaurasia, “A Comprehensive Analysis of Indian Legal Documents Summarization Techniques” SN Comput. Sci., vol. 4, 2023 https://www.researchgate.net/publication/373074828 [17] M. S. Kabir and M. N. Alam, “The Role of AI Technology for Legal Research and Decision Making” Int. Res. J. Eng. Technol. (IRJET), vol. 10, 2023 https://www.researchgate.net/publication/372790308 [18] R. Agarwal, A. Hariharan, O. Atkari, S. Ghogare, and S. Ghorpade, “Real-Time Interview Analysis Using Deep Learning” Marathwada Mitra Mandal’s Institute of Technology, Pune, India, 2026. https://www.ijfmr.com/research-paper.php?id=67359

Copyright

Copyright © 2026 Rucha Agrawal, Mr. Arvind Hariharan, Mr. Omkar D. Atkari, Mr. Sujal K. Ghogare, Mr. Suyash R. Ghorpade. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.