Transformer Based Approaches for Sentiment Analysis on Restaurant Domain

Abstract

This project addresses the critical challenge of accurately analysing sentiment in restaurant reviews, where traditional machine learning methods struggle with linguistic nuances. Existing solutions, which are frequently basic ML or older deep learning models, typically lack the capacity to properly address class imbalance and capture deep contextual knowledge. To address these limitations, the study implements and evaluates two advanced transformer-based models: fine-tuning RoBERTa-large with Focal Loss, which focuses on learning from difficult examples and underrepresented classes, and a hybrid DistilBERT-CNN-BiLSTM model with fast gradient method, which combines efficient contextual embedding, local feature extraction, and sequential pattern recognition. These models aim to improve sentiment classification performance by making use of transformer-based architectures and modifying loss functions. Aspect-based sentiment analysis integration and the implementation of models for real-time feedback systems are examples of future development.

Introduction

The study focuses on improving sentiment analysis of restaurant reviews, which is crucial for understanding consumer opinions, social media trends, and forecasting marketing success. Restaurant reviews contain rich, subjective feedback but pose challenges for automated analysis due to nuanced language, sarcasm, domain-specific terms, and class imbalance (unequal distribution of sentiment categories).

Traditional machine learning methods require manual feature engineering and often lack deep contextual understanding, whereas deep learning models, especially transformer-based models like RoBERTa and DistilBERT, excel in capturing complex language patterns with less manual effort.

The research proposes two deep learning approaches to enhance sentiment classification accuracy and handle class imbalance effectively:

1. Fine-tuning RoBERTa-large with Focal Loss:

   • RoBERTa, a robust transformer model, is fine-tuned on restaurant review data.

   • Focal Loss is used to mitigate class imbalance by focusing training on harder-to-classify examples, improving minority class prediction.

2. Hybrid DistilBERT-CNN-BiLSTM model with Weighted Cross-Entropy Loss and Fast Gradient Method (FGM):

   • DistilBERT provides efficient embeddings.

   • CNN extracts local phrase patterns.

   • BiLSTM captures sequential context bidirectionally.

   • Focal Loss addresses class imbalance; FGM adds adversarial training for robustness.

Both models are trained on a dataset of nearly 10,000 restaurant reviews, classified into Negative, Neutral, and Positive sentiments. Data preprocessing includes mapping ratings to sentiment classes, tokenization, and stratified splitting.

Results:

• RoBERTa with Focal Loss achieved higher accuracy (~87.9%) and better performance on minority classes compared to the hybrid model (~85.9% accuracy).

• Both models showed strong precision, recall, and F1 scores for Positive and Negative sentiments, with Neutral class being more challenging.

• The hybrid model offers a computationally efficient alternative with competitive accuracy.

Contributions:

• Application of advanced transformer-based models fine-tuned specifically for restaurant review sentiment analysis.

• Use of Focal Loss and adversarial training techniques to address class imbalance and improve model robustness.

• A hybrid model architecture combining DistilBERT, CNN, and BiLSTM for effective feature extraction and sequence modeling.

The study demonstrates how leveraging recent NLP advances and tailored loss functions can significantly enhance the accuracy and reliability of sentiment classification in the hospitality domain.

Conclusion

This work successfully addressed the important issues of linguistic nuance and class imbalance in restaurant review sentiment analysis. The project achieved high validation accuracies (0.8754 and 0.8594, respectively) and robust F1-scores by implementing and evaluating two sophisticated transformer-based approaches: a hybrid DistilBERT-CNN-BiLSTM architecture and a fine-tuned RoBERTa-large model, both of which were enhanced with specialized loss functions (Focal Loss) and adversarial training. This is a significant advance over traditional machine learning and simpler deep learning algorithms, which frequently fail under such complications. The created models show a better ability to acquire profound contextual awareness and efficiently address data imbalance, especially when it comes to categorizing positive and negative attitudes. Future research could focus on integrating aspect-based sentiment analysis to provide more nuanced insights and enhancing algorithms for more accurate neutral class classification.Additionally, investigating the potential of even larger language models (LLMs) offers a viable path forward.

References

[1] Putta Durga And Deepthi Godavarth. “Deep-Sentiment: An Effective Deep SentimentAnalysis Using a Decision-Based RecurrentNeural Network (D-RNN),” IEEE Access (2023). [2] Kanwal Zahoor, Narmeen Zakaria Bawany, Soomaiya Hamid. “Sentiment Analysis and Classification of Restaurant Reviews using Machine Learning,” IEEEAccess(2020) [3] Chang Liu, Lei Li, Chen Shan, Xinyu Hu, Zhifeng Diao, Mao-en He. “Exploring Neighborhood Service and Development Strategies by Multi-dimensional Sentiment Analysis of Online Restaurant Review,” IEEE Access (2021). [4] Krosuri Lakshmi Revathi, Aravapalli Rama Satish, Popuri Srinivasa Rao. “Sentiment Analysis Using Deep Learning Techniques: A Review,” (IJACSA) International Journal of Advanced Computer Science and Applications, (2017). [5] Yaya Heryadi, Bambang Dwi Wijanarko; Dina Fitria Murad; Cuk Tho; Kiyota Hashimoto “Aspect-based Sentiment Analysis using Long Short-term Memory Model for Leveraging Restaurant Service Management,” IEEE Access(2023) [6] Kanwal Zahoor; Narmeen Zakaria Bawany; Soomaiya Hamid, “Sentiment Analysis and Classification of Restaurant Reviews using Machine Learning,” IEEE Access(2020) [7] Qurat Tul Ain, Mubashir Ali, Amna Riaz, Amna Noureen, Muhammad Kamran, Babar Hayat and A. Rehman, “Sentiment Analysis Using Deep Learning Techniques: A Review,” (IJACSA) International Journal of Advanced Computer Science and Applications(2017) [8] Ramadhani, A. M., and Goo, H. S. (2017). “Twitter Sentiment Analysis Using Deep Learning Methods,” 2017 7th International Annual Engineering Seminar (InAES). [9] Alfreihat, MAlmousa, OSTashtoush, Y AlSobeh, A Mansour, andMigdady, H,“Emo-SL Framework: Emoji Sentiment Lexicon Using Text-Based Features and Machine Learning for Sentiment Analysis,” IEEE Access (2024) [10] Polisena, J Andellini, M Salerno, P Borsci, S Pecchia, L and Iadanza, E,“Case Studies on the Use of Sentiment Analysis to Assess the Effectiveness and Safety of Health Technologies: A Scoping Review.” IEEE Access (2021). [11] Wang, L, Niu, J, and Yu,S, “SentiDiff: Combining Textual Information and Sentiment Diffusion Patterns for Twitter Sentiment Analysis,” IEEE Transactions on Knowledge and Data Engineering (2020). [12] Nazir, A, Rao, Y, Wu, L, and Sun, L. “Issues and Challenges of Aspect-Based Sentiment Analysis: A Comprehensive Survey,” IEEE Transactions on Affective Computing (2022). [13] Kim, R. Y. “Using Online Reviews for Customer Sentiment Analysis,” IEEE Engineering Management Review (2021). [14] Kastrati, Z., Imran, A. S., and Kurti, A,“Weakly Supervised Framework for Aspect-Based Sentiment Analysis on Students’ Reviews of MOOCs,” IEEE Access (2020). [15] Lin, N., Fu, Y., Lin, X., Zhou, D., Yang, A., and Jiang, S. “CL-XABSA: Contrastive Learning for Cross-Lingual Aspect-Based Sentiment Analysis,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023). [16] Xu, J., Li, Z., Huang, F., Li, C., and Yu, P. S. “Visual Sentiment Analysis with Social Relations-Guided Multiattention Networks.” IEEE Transactions on Cybernetics, (2022). [17] Li, N., Chow, C., & Zhang, J. “SEML: A Semi-Supervised Multi-Task Learning Framework for Aspect-Based Sentiment Analysis,” IEEE Access (2022). [18] Zhou, J., Jin, S., and Huang, X. “ADeCNN: An Improved Model for Aspect-Level Sentiment Analysis Based on Deformable CNN and Attention,” IEEE Access (2020). [19] Zhang, T., Gong, X., and Chen, C. L. P. “BMT-Net: Broad Multitask Transformer Network for Sentiment Analysis,” IEEE Transactions on Cybernetic (2022). [20] Parandham G, Mr. Raghavendra R, “Sentiment Analysis on Restaurant Reviews,” IRJETS (2021).

Copyright

Copyright © 2025 P Sai Lakhsmi, K V Ramana, K Ravi Kiran. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.