Facial Emotion Detection and Music Recommendation using Deep Learning

Abstract

Customary music recommendation systems depend on past tuning in history and course slants to recommend unused music. In any case, this will lead to clients being proposed music that\'s comparable to what they have as of presently tuned in. This paper proposes an unused music proposal framework that livelihoods multimodal feeling affirmation to endorse music that\'s custom-fitted to the user\'s current personality. The system businesses significant learning illustrates to distinguish the user\'s sentiments from their facial expressions and other multimodal signals. Once the user\'s sentiments have been recognized, the system endorses music that\'s likely to facilitate those sentiments. The proposed system is more exact than single-modal or other procedures that have been utilized in the past. More often than not since the system takes into thought various sources of information nearly the user\'s sentiments. The makers acknowledge that their exploration has the potential to revolutionize the way that people tune in to music. By endorsing music that\'s custom-fitted to the user\'s current disposition, the system can offer help to clients to discover unused music that they appreciate and to have a more personalized music tuning-in experience.

Introduction

Overview:

The text describes a novel music recommendation system that uses facial expression recognition and deep learning to identify a user’s current mood and suggest music tailored to that emotional state. By analyzing facial cues, the system aims to provide a highly personalized listening experience that can enhance mood, reduce stress, improve sleep quality, increase focus, and overall promote well-being.

System Functionality:

• Uses deep learning models to extract features from facial expressions.

• Classifies user emotions in real-time.

• Recommends music that aligns with or enhances the detected mood.

• Potentially transforms how people discover and interact with music.

Literature Review Highlights:

• Various studies demonstrate the effectiveness of using facial expressions, voice, and body language combined with machine learning for emotion recognition.

• Multimodal systems (combining multiple data sources) improve accuracy.

• Deep learning architectures like Convolutional Neural Networks (CNNs) and transformers are commonly used.

• Several emotion recognition systems classify emotions into categories such as happy, sad, angry, neutral, surprise, disgust, and fear.

• Challenges include small datasets, need for real-time processing, and personalization.

• Explainability and bias detection in AI systems are important ongoing research areas.

• Applications extend beyond music recommendation to healthcare, education, and entertainment.

Future Directions and Considerations:

• Multimodal fusion of facial, audio, and physiological signals can improve emotion detection.

• Context-aware models that understand the situation behind expressions for better accuracy.

• Systems need to work efficiently on resource-constrained devices.

• Personalization to individual user differences and cultural context is critical.

• Large, diverse datasets are required to build robust models.

Key Research Examples:

• M3ER model achieves state-of-the-art emotion recognition by weighting multiple modalities differently.

• Emoticon model incorporates context using attention mechanisms to interpret facial and behavioral cues more accurately.

Summary:

The proposed system leverages advanced machine learning and facial emotion detection to deliver music recommendations uniquely suited to a user’s emotional state, promising a personalized, mood-enhancing music experience. Current research supports the feasibility and benefits of such systems, while highlighting the need for better data, real-time adaptability, and contextual understanding for broader and more effective applications.

Conclusion

In this review paper, we have evaluated the potential of utilizing multimodal examination for music proposition. We have showed up a music recommendation system that businesses facial feeling area to back music to clients. Our system has been showed up to be more commonsense than single-modal examination or other strategies utilized until clearly. The multimodal approach highlights a humble bunch of centers of captivated over single-modal examination. To start with, it licenses us to capture more information around the client, which can lead to more rectify recommendations. Scaled down, it is more extraordinary to clamor and desire. Third, it is more generalizable to particular clients and particular circumstances. Our system is still underneath modify, but we recognize that it has the potential to revolutionize the way that music is gotten a handle on to clients. We are particularly inquisitive generally analyzingthe utilize of multimodal examination for music suggestion in personalized learning and healthcare applications. We recognize that this think approximately paper has enacted other specialists to examine the utilize of multimodal examination for music proposal. We recognize that as a run the appear up a promising increase of ask around with the potential to have a central impact on the way that people appreciate music.

References

[1] Weng, Yabin, and Feiting Lin. \"Multimodal emotion recognition algorithm for artificial intelligence information system.\" Wireless Communications and Mobile Computing 2022 (2022): 1-9. [2] Wang, Yan, et al. \"A systematic review on affective computing: Emotion models, databases, and recent advances.\" Information Fusion 83 (2022): 19-52. [3] Roy, Dharmendra, et al. \"Music Recommendation Based on Current Mood Using AI & ML.\" (2023). [4] Abdullah, Sharmeen M. Saleem Abdullah, et al. \"Multimodal emotion recognition using deep learning.\" Journal of Applied Science and Technology Trends 2.02 (2021): 52-58. [5] Florence, S. Metilda, and M. Uma. \"Emotional detection and music recommendation system based on user facial expression.\" IOP conference series: Materials science and engineering. Vol. 912. No. 6. IOP Publishing, 2020. [6] Hussain, ShaikAsif, and AhlamSalimAbdallah Al Balushi. \"A real time face emotion classification and recognition using deep learning model.\" Journal of physics: Conference series. Vol. 1432. No. 1. IOP Publishing, 2020. [7] Raut, Nitisha. \"Facial emotion recognition using machine learning.\" (2018). [8] Mahadik, Ankita, et al. \"Mood based music recommendation system.\" INTERNATIONAL JOURNAL OF ENGINEERING RESEARCH & TECHNOLOGY (IJERT) Volume 10 (2021). [9] Rahmad, Cahya, et al. \"Comparison of Viola-Jones Haar Cascade classifier and histogram of oriented gradients (HOG) for face detection.\" IOP conference series: materials science and engineering. Vol. 732. No. 1. IOP Publishing, 2020. [10] Huang, Jian, et al. \"Multimodal transformer fusion for continuous emotion recognition.\" ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. [11] Florence, S. Metilda, and M. Uma. \"Emotional detection and music recommendation system based on user facial expression.\" IOP conference series: Materials science and engineering. Vol. 912. No. 6. IOP Publishing, 2020. [12] Kumar, Ashu, AmandeepKaur, and Munish Kumar. \"Face detection techniques: a review.\" Artificial Intelligence Review 52 (2019): 927-948. [13] Dalal, Navneet, and Bill Triggs. \"Histograms of oriented gradients for human detection.\" 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR\'05). Vol. 1. Ieee, 2005. [14] Mukhopadhyay, Moutan, et al. \"Facial emotion detection to assess Learner\'s State of mind in an online learning system.\" Proceedings of the 2020 5th international conference on intelligent information technology. 2020. [15] Pathar, Rohit, et al. \"Human emotion recognition using convolutional neural network in real time.\" 2019 1st International Conference on Innovations in Information and Communication Technology (ICIICT). IEEE, 2019. [16] Hizlisoy, Serhat, SerdarYildirim, and ZekeriyaTufekci. \"Music emotion recognition using convolutional long short term memory deep neural networks.\" Engineering Science and Technology, an International Journal 24.3 (2021): 760-767. [17] Lopes, André Teixeira, et al. \"Facial expression recognition with convolutional neural networks: coping with few data and the training sample order.\" Pattern recognition 61 (2017): 610-628. [18] Poria, Soujanya, et al. \"A review of affective computing: From unimodal analysis to multimodal fusion.\" Information fusion 37 (2017): 98-125. [19] Maheshwari, Shikhar C., Amit H. Choksi, and Kaiwalya J. Patil. \"Emotion based Ambiance and Music Regulation using Deep Learning.\" 2020 International Conference on Communication and Signal Processing (ICCSP). IEEE, 2020. [20] Anggo, Mustamin, and La Arapu. \"Face recognition using fisherface method.\" Journal of Physics: Conference Series. Vol. 1028. No. 1. IOP Publishing, 2018. [21] Viola, Paul, and Michael J. Jones. \"Robust real-time face detection.\" International journal of computer vision 57 (2004): 137-154. [22] JavedMehediShamrat, F. M., et al. \"Human face recognition applying haar cascade classifier.\" Pervasive Computing and Social Networking: Proceedings of ICPCSN 2021. Springer Singapore. [23] Mittal, Trisha, et al. \"M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 02. 2020. [24] Mittal, Trisha, et al. \"Emoticon: Context-aware multimodal emotion recognition using frege\'s principle.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [25] Zhang, Jianhua, et al. \"Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review.\" Information Fusion 59 (2020): 103-126. [26] Dzedzickis, Andrius, Art?rasKaklauskas, and VytautasBucinskas. \"Human emotion recognition: Review of sensors and methods.\" Sensors 20.3 (2020): 592. [27] Lim, JiaZheng, James Mountstephens, and Jason Teo. \"Emotion recognition using eye-tracking: taxonomy, review and current challenges.\" Sensors 20.8 (2020): 2384. [28] Mellouk, Wafa, and WahidaHandouzi. \"Facial emotion recognition using deep learning: review and insights.\" Procedia Computer Science 175 (2020): 689-694. [29] Da’u, Aminu, and NaomieSalim. \"Recommendation system based on deep learning methods: a systematic review and new directions.\" Artificial Intelligence Review 53.4 (2020): 2709-2748. [30] Zepf, Sebastian, et al. \"Driver emotion recognition for intelligent vehicles: A survey.\" ACM Computing Surveys (CSUR) 53.3 (2020): 1-30. [31] Kortli, Yassin, et al. \"Face recognition systems: A survey.\" Sensors 20.2 (2020): 342. [32] Bah, SerignModou, and Fang Ming. \"An improved face recognition algorithm and its application in attendance management system.\" Array 5 (2020): 100014. [33] Song, Yading, Simon Dixon, and Marcus Pearce. \"A survey of music recommendation systems and future perspectives.\" 9th international symposium on computer music modeling and retrieval. Vol. 4. 2012. [34] Bartlett, Marian Stewart, et al. \"Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction.\" 2003 Conference on computer vision and pattern recognition workshop. Vol. 5. IEEE, 2003. [35] Michel, Philipp, and Rana El Kaliouby. \"Real time facial expression recognition in video using support vector machines.\" Proceedings of the 5th international conference on Multimodal interfaces. 2003. [36] Jiang, Ning, et al. \"A cascade detector for rapid face detection.\" 2011 IEEE 7th International Colloquium on Signal Processing and its Applications. IEEE, 2011. [37] [38] Athavle, Madhuri, et al. \"Music recommendation based on face emotion recognition.\" Journal of Informatics Electrical and Electronics Engineering (JIEEE) 2.2 (2021): 1-11. [38] James, H. Immanuel, et al. \"Emotion based music recommendation system.\" Emotion 6.3 (2019). [39] Iyer, Aurobind V., et al. \"Emotion based mood enhancing music recommendation.\" 2017 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT). IEEE, 2017. [40] Zhou, Hailing, et al. \"Recent advances on singlemodal and multimodal face recognition: a survey.\" IEEE Transactions on Human-Machine Systems 44.6 (2014): 701-716. [41] Kakadiaris, Ioannis A., et al. \"Multimodal face recognition: Combination of geometry with physiological information.\" 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\'05). Vol. 2. IEEE, 2005. [42] Ding, Changxing, and Dacheng Tao. \"Robust face recognition via multimodal deep face representation.\" IEEE transactions on Multimedia 17.11 (2015): 2049-2058. [43] Gupta, Alpika, and RajdevTiwari. \"Face detection using modified Viola jones algorithm.\" International Journal of Recent Research in Mathematics Computer Science and Information Technology 1.2 (2015): 59-66. [44] JavedMehediShamrat, F. M., et al. \"Human face recognition applying haar cascade classifier.\" Pervasive Computing and Social Networking: Proceedings of ICPCSN 2021. Springer Singapore, 202 [45] Li, Haoxiang, et al. \"A convolutional neural network cascade for face detection.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

Copyright

Copyright © 2025 Ghanshyam Bagadi, Harshada Mhaske, Chaitanya Asole, Pranay Ambade, Piyush Agawane. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.