Music Recommendation System based on Facial Emotion Detection using Spotify API

Abstract

In the world of entertainment, music holds considerable importance, especially for those who find joy in rhythmic experiences. Despite the abundance of streaming platforms that enable access to favorite songs, they often fall short in capturing the intricate emotional nuances of users. This research recognizes a spectrum of emotions, including fear, happiness, sadness, anger, and neutrality. Its goal is to enrich the user experience by developing a recommendation system that proposes songs based on the user\'s current emotionalstate.Theemotion-driven recommendationengine has seamlessly integrated into Spotify, a well-known music streaming service, providing users with a smooth and individualizedjourneyinexploringmusic.Thesystemaimsto simplify the user experience by eliminating the necessity for manualsongsearchesand,instead,intuitivelysuggeststracks thatresonatewiththeuser\'semotions.TheSpotifyAPIserves as a crucial tool for accessing curated playlists, enabling the retrieval of desired music from thoughtfully organized collections centered around specific themes or titles.

Introduction

1. Introduction

• Music is a central part of daily life and varies by culture, location, and personal taste.

• Traditional recommendation systems (e.g., Spotify) use hybrid models based on content and collaborative filtering.

• The proposed system enhances this by adding emotion recognition using facial expressions, aiming to recommend songs that align with the user’s current mood, regardless of a song’s age or popularity.

2. Related Work

• Prior studies explored genre classification using CNNs and LSTMs on audio features (e.g., MFCCs).

• Emotion recognition from facial expressions has been widely studied using deep learning.

• Some works integrated psychological traits for personalized music recommendations, showing improved performance over genre-only models.

• A few systems experimented with context-aware suggestions (e.g., user’s activity), and some developed emoji-based emotion mapping tools.

3. Proposed Method

The system consists of three main modules:

A. Emotion Detection Module

• Uses CNN on the FER-2013 dataset to classify user facial expressions into one of seven emotions: anger, disgust, fear, happiness, sadness, surprise, neutrality.

• CNN architecture includes layers such as Conv2D, MaxPooling, Batch Normalization, Dropout, and Dense layers.

B. Face Detection Module

• Implements the Viola-Jones algorithm using OpenCV to detect faces.

• The cropped face is passed to the CNN model for emotion prediction.

C. Song Recommendation Module

• Users log in via Spotify Web API to retrieve their most-played tracks.

• A heatmap identifies key song features (e.g., valence, danceability, energy, tempo) for recommendation.

• Based on the detected emotion, songs are recommended from an emotion-matched playlist.

4. Visual Insights

• Heatmaps reveal patterns like:

  • Acousticness ↔ Valence (positive correlation: calming acoustic music evokes positive emotions).

  • Danceability ↔ Energy (upbeat tracks are more danceable).

  • Speechiness ↔ Instrumentalness (negative correlation).

  • Loudness ↔ Valence (slight negative: louder songs may relate to anger or aggression).

• Flow diagrams show how facial features are captured, emotion is identified, and matching music is played.

5. Results & Discussion

• The system recommends songs based on the user’s real-time facial emotion, demonstrated through test cases (e.g., sad face → sad playlist).

• Emotion-to-song mapping improves personalization compared to static playlist recommendations.

Conclusion

Insummary,theamalgamationoffacialemotiondetection with the Spotify API in our music recommender system represents a unique and captivating approach to enriching user interactions. Employing the FER13 dataset ensures precise identification of emotions, allowing our system to not only leverage advanced technology but also interpret users\' nuanced facial expressions, providing music recommendations that are not just personalized but emotionally resonant. Our recommender system strengthens its bond with users by interpreting facial cues and linking them to emotional states,aligningtheirmusicalpreferenceswiththeircurrent moods.Thereal-timematchingofsongstousers\'evolving emotional states, coupled with the seamless integration with the extensive Spotify API, ensures a diverse and expansive music selection. This innovative methodology goes beyond traditional recommendation systems, offering a responsive and dynamicmusicdiscovery experience.Asusersconveytheir emotions through facial expressions, our system adapts, curating playlists and suggesting songs that mirror the shifting emotional landscape. The harmonious interplay between Spotify\'s vast music library and facial emotion detection transforms the system into more than just a recommendation tool—it becomes a companion in the user\'s emotional journey. Positioned at the intersection of emotion, technology, and music within the dynamic realm of personalized technology, this music recommender system promises a comprehensive and immersive user experience. As the work progress, continual enhancements and the incorporation of state-of-the-art technologies will ensure thatthissystemremainsafrontrunnerindeliveringtailored musicalexperiences,redefininghowusersengagewithand appreciate the impact of music in their lives.

References

[1] Hassen,AlanKai,etal.\"Classifyingmusicgenresusing image classification neural networks.\"Archives of Data Science, Series A (Online First) 5.1 (2018): 20. [2] Gessle, Gabriel, and Simon Åkesson. \"A comparative analysisofCNNandLSTMformusicgenreclassification.\" (2019). [3] Mellouk,Wafa,andWahidaHandouzi.\"Facialemotion recognition using deep learning: review and694. [4] Erdal, Bar??, et al. \"The magic of frequencies-432 Hz vs. 440 Hz: Do cheerful and sad music tuned to different frequencies cause different effects on human psychophysiology?Aneuropsychologystudyonmusicand emotions: Frekanslar?nsihri–432 Hz 440 Hz’ekar??: Ayr?frekanslaragöreakortlanm??ne?elivehüzünlümüziklerinsanpsikofizyolojisiüzerindefarkl?etkileryarat?rm?? Müzikveduygularüzerinebirnöropsikolojiara?t?rmas?.\"JournalofHumanSciences18.1(2021):12-33. [5] M.J.Awan,A.Raza, A.Yasin,H.M. F.Shehzad,andI.Butt,“TheCustomizedConvolutionalNeuralNetworkof Face Emotion Expression Classification,” Annals of the Romanian Society for Cell Biology, vol. 25, no. 6, pp. 5296-5304, 2021. [6] MadhuriAthavle,etal.\"MusicRecommendationBased on Face Emotion Recognition.\" Journal of unformatics Electrical and Electronics Engineering, Vol 2. No.2, 2021 [7] Akrati Gupta, Saurabh Kumar, Rachit Kumar, Vikash Kumar Mishra “Emojify –Create your own emoji with DeepLearning”IRJMETS,Volume:04/Issue:07/July-2022 [8] Chaturvedi, V., Kaur, A.B., Varshney, V. et al.Music mood and human emotion recognition based on physiological signals: a systematic review. Multimedia Systems28,21–44(2022).https://doi.org/10.1007/s00530-021-00786-6. [9] Ghosh, Oindrella, et al. \"Music Recommendation System based on Emotion Detection using Image Processing and Deep Networks.\"2022 2nd International Conference on Intelligent Technologies (CONIT). IEEE, 2022. [10] Sana, S. K., et al. \"Facial emotion recognition based musicsystemusingconvolutionalneuralnetworks.\"MaterialsToday:Proceedings62(2022):4699-4706. [11] Phaneendra, A., et al. \"EMUSE–An emotion based music recommendation system.\"International Research Journal of Modernization in Engineering Technology and Science 4.5 (2022): 4159-4163. [12] Shanthakumari, R., et al. \"Spotify Genre Recommendation Based On User Emotion Using Deep Learning.\"2022 Fifth International Conference on Computational Intelligence and Communication Technologies (CCICT). IEEE, 2022. [13] Bhowmick, Anusha, et al. \"Song Recommendation System based on Mood Detection using Spotify\'s Web API.\"2022 International Interdisciplinary Humanitarian Conference for Sustainability (IIHC). IEEE, 2022. [14] Dubey, Arnav, et al. \"Digital Content Recommendation System through Facial Emotion Recognition.\"Int. J. Res. Appl. Sci. Eng. Technol11 (2023): 1272-1276. [15] Bokhare, Anuja, and Tripti Kothari. \"Emotion Detection-Based Video Recommendation System Using Machine Learning and Deep Learning Framework.\" SN Computer Science 4.3 (2023): 215. [16] Sharath, P., G. Senthil Kumar, and Boj KS Vishnu. \"Music Recommendation System Using FacialEmotions.\"Advances in Science and Technology 124 (2023): 44-52

Copyright

Copyright © 2025 Prasanjit Singh, Machanuru Suresh Babu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.