Activity Recommendation System Based on Emotion Recognition

Abstract

Activity Recommendation System for Emotion Recognition is a mobile app designed to identify people\'s emotion and suggest them suitable activities based on those emotions. The emotion analysis takes place based on three types of inputs: text; voice and facial expression, via mobile device. Emotion identification occurs through the use of machine learning models and natural language processing with the following classifiers included in the application: happy; sad; angry; frustrated; and stressed. Once the emotion has been identified, various independent activities will be recommended to the user, such as: listening to music, meditating, exercising, or socializing. The user\'s age is also taken into consideration, thereby allowing a specific and personalized recommendation for each user. The application is built using TensorFlow, OpenCV, and Android Studio. The goal of this system is to help promote a positive mental state in the user by providing support in their understanding of emotional states and providing the user with recommendations of suitable activities to help them improve their emotional state. Experimentation indicates that emotion-based recommendation systems can provide substantial support in stress management and mood enhancement through identifying appropriate activities for users.

Introduction

AI-based multimodal emotion recognition and activity recommendation system designed to improve users’ mental well-being.

It explains that human emotions strongly influence decision-making and mental health, and modern stress levels have increased the need for intelligent emotion-aware systems. The proposed system uses text, voice, and facial expressions as inputs, analyzed through NLP, speech emotion recognition, and computer vision techniques. Machine learning models classify emotions into categories such as happiness, sadness, anger, frustration, stress, and neutral.

Based on the detected emotion, the system recommends personalized activities (like meditation, music, exercise, or social interaction) to improve the user’s mood. The system is implemented as an Android application with a backend database (Firebase/SQLite) that stores user data, emotional history, and recommendations.

The literature survey highlights that emotion recognition has evolved from traditional methods to deep learning approaches (CNNs, RNNs, transformers), and multimodal systems are more accurate than single-input systems.

The methodology includes data collection, preprocessing, emotion detection, classification, recommendation generation, and result display. Experimental results show that combining multiple modalities improves emotion detection accuracy and enables effective personalized recommendations.

Conclusion

The Emotion Recognition Activity Recommendation System (ERARS) is a smart system that uses different ways to find out how you are feeling and to suggest activities that might help you feel better. The system has different ways of detecting how you feel: through text, voice, and facial expressions. Some of the emotional states the system can detect include: happy, sad, angry, frustrated and stressed. With the help of machine learning, natural language processing, speech analysis, and facial expression recognition techniques, the system accurately interprets your feelings. Once the system identifies your emotional state and checks your user profile, it will suggest personalized activities to improve your emotional state. The suggested activities may include: listening to music, meditating, doing physical activity, or having some form of social interaction. By utilizing a database to store your user history, the system can monitor your patterns of emotion and improve on future suggestions. The research provides empirical evidence to demonstrate the benefit of emotion-aware recommendation systems in supporting an awareness of mental health as well as providing assistance to users in managing their emotional states using intelligent mobile applications.

References

[1] S. Li and W. Deng, “Deep Facial Expression Recognition: A Survey,” IEEE Transactions on Affective Computing, vol. 13, no. 3, pp. 1195–1215, 2022. [2] W. Mellouk and W. Handouzi, “Facial Emotion Recognition Using Deep Learning: A Review,” Procedia Computer Science, vol. 170, pp. 689–694, 2020. [3] J. Chen, Z. Wang, and Y. Liu,”Context-aware emotion recognition using multimodal fusion”,IEEE Access,vol. 9, pp. 112345-112356, 2021. [4] X. Zeeng, H. Hu, and L. Chen, “Emotion recognition-based activity recommendation,” IEEE Access, vol. 10,pp. 45678-45689, 2022 [5] Z. Zhang, P. Luo, C. Loy, and X. Tang, “Facial Expression Recognition Using Deep Learning: A Review,” IEEE Conference On Computer Vision and Pattern Recognition Workshops, 2020 [6] D. Duki? et al., “Real-Time Facial Expression Recognition Using Deep Learning,” Electronics, vol. 11, no. 8, 2022. [7] M. Sajjad et al., “A Comprehensive Survey on Deep Facial Expression Recognition,” Alexandria Engineering Journal, vol. 63, pp. 147–168, 2023. [8] S. Tripathi, S. Acharya, and S. Sharma, “Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on Speech,” arXiv preprint, 2019. [9] J. Zhao et al., “Speech Emotion Recognition Using Deep 1D & 2D CNN-LSTM Networks,” Biomedical Signal Processing and Control, vol. 47, pp. 312–323, 2019. [10] Y. Badr et al., “Speech Emotion Recognition Using MFCC and Hybrid Neural Networks,” International Conference on Artificial Intelligence, 2021. [11] M. Mohan et al., “Speech Emotion Classification Using Ensemble Learning Models,” Procedia Computer Science, vol. 218, pp. 172–181, 2023. [12] R. Ullah et al., “Speech Emotion Recognition Using Convolutional Neural Networks and Transformer Encoder,” Sensors, vol. 23, no. 13, 2023. [13] S. M. George et al., “A Review on Speech Emotion Recognition Techniques,” Neurocomputing, 2024. [14] H. Lian et al., “A Survey of Deep Learning-Based Multimodal Emotion Recognition,” Entropy, vol. 25, no. 10, 2023. [15] J. Pan et al., “Multimodal Emotion Recognition Based on Facial Expression and Speech Features,” IEEE Access, 2023. [16] T. Kopalidis et al., “Advances in Facial Expression Recognition: A Survey,” Information, vol. 15, no. 3, 2024. [17] M. Alisawi et al., “Real-Time Emotion Recognition Using Deep Learning Techniques,” International Journal of Intelligent Systems, 2023. [18] J. Bhanbhro et al., “Deep Learning Models for Speech Emotion Recognition,” AI, vol. 6, no. 2, 2025. [19] T. Mary Little Flower et al., “Data Augmentation Using 1D CNN Models for Speech Emotion Recognition,” Connection Science, 2024. [20] M. Joshi and S. Agarwal, “Facial Emotion Recognition Using Deep Learning: A Survey,” International Journal of Engineering Research, 2022. [21] M. C. Gursesli, L. Bocchi, “Facial Emotion Recogniotion Through Custom Lightweight CNN Model:Performance Evaluation in Public Datasets;” IEEE Access, 10.1109/ACCESS, 2024. [22] M. Borgaonkar, M. Katta and V. Deshpande, “Emotion Based Personalized Recommendation System,” , International Research Journal of Engineering and Technology, 2020. [23] J. Y. Chen, V. John , Y. Kawanishi, “Cross-modal Emotion-specific Attention model for Multimodal Emotion Recognition,” International Conference on Automatic Face and Gesture Recognition, 2025. [24] B. Prsanth, V. S . Kumar , “AI-Bot Powered Emotion Detection and Mental Wellness System with Facial and Voice Analysis for Personalized Recommendations,” International Conference on Compuational Robotics,Testing and Engineering Evaluation, 2025. [25] R. Pandey, S. Barde, “EEG-Based Emotion Classification and Personal Response Optimization Using SVM,” International Conference on Ambient Intelligence in Health Care, 2025. [26] L. Deng and D. YU,”Deep learning methods for speech emotion recognition” IEEE Signal Processing Magazine, vol. 37, no. 6, pp. 106-116, 2020 [27] P. K. Gaikwad, S. U. Haldipur, J. Ramteke , “Moody.ai: An Adaptive Activity Recommendation System Based on Emotion Detection, IEEE 2023. [28] M. N. Ekanayake, G. C. Kaveendhva, “Emotional State Monitoring with Personalized Activity Recommendations,” 5th International Conference on Advancements in Computing(ICAC), 436-441, 2023. [29] K. R. Gavhale, U. H. Gawande, “An Analysis of Recommendation System Based on Multimodal Emotion Detection Using Facial Expression and Voice Modulation,” International Conference on Emerging Trends in Engineering and Medical Sciences,65-70, 2024. [30] S. M. Saleem, “Multimodal Emotion Recognition Using Deep Learning,” Journal of Applied Science and Technology Trends ,73-79, 2021.

Copyright

Copyright © 2026 Prof. A. D. Dhawale, Dinesh Ambhore, Vaishnavi Nikam, Samyak Ilamkar, Pravin Ingole. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.