Sentiment Analysis using BERT, CNN and BI-LSTM

Abstract

Sentiment Analysis (SA) has emerged as a critical task in Natural Language Processing (NLP), especially with the exponential growth of user-generated content. Although deep learning models like Bi-directional Long Short-TermMemory(Bi-LSTM),Convolutional Neural Networks (CNN), and Bidirectional Encoder Representations from Transformers (BERT) have greatly improved sentiment classification accuracy, they often fail to capture the meaning of idiomatic expressions. In this paper, we propose a hybrid sentiment analysis framework that integrates BERT embeddings with CNN for feature extraction, Bi-LSTM for sequence modeling, and an attention mechanism for enhanced sentiment prediction. Additionally, we introduce a dynamic idiom detection and sentiment inference module that identifies both known and unknown idioms. For unknown idioms, contextual information is retrieved via web search to lexically infer sentiment. The model is evaluated on a benchmark English hotel review dataset, along with an extended idiom-labeled dataset. Our approach achieves a state-of-the-art accuracy of 92.24%, outperforming traditional Bi-LSTM and RNN models. Results show that our hybrid system significantly enhances sentiment classification performance, particularly in the presence of idiomatic expressions.

Introduction

Sentiment analysis (SA) is crucial in NLP for classifying text into positive, negative, or neutral sentiments, widely applied in reviews, social media, and customer feedback. Traditional machine learning struggles with idiomatic expressions and deep context, but advances in deep learning—especially transformer models like BERT, Bi-LSTM, and CNNs—have improved accuracy by capturing contextual, sequential, and local text patterns.

However, idioms (e.g., “kick the bucket”) remain challenging due to their non-literal meanings, often causing misclassification. To address this, the paper proposes a hybrid deep learning model combining BERT embeddings, CNN, and Bi-LSTM with an attention mechanism, improving handling of nuanced sentiment, including idioms.

A novel idiom detection module uses a two-stage approach: known idioms are detected via cosine similarity with a fine-tuned idiom dataset, while unknown idioms are identified and their sentiment inferred using real-time web search and lexical analysis. This module preprocesses text before sentiment classification.

The model is evaluated on a hotel review dataset augmented with idiomatic phrases, achieving state-of-the-art accuracy (92.24%) and outperforming traditional models. The work’s contributions include a hybrid architecture with idiom-aware processing, real-time idiom sentiment inference, and improved interpretability and accuracy.

Related work discusses limitations of existing deep learning and lexicon-based approaches in handling idioms, highlighting the benefit of hybrid models with attention and dynamic idiom recognition.

Overall, the hybrid approach effectively integrates context understanding, local pattern extraction, sequence modeling, and idiom processing for robust sentiment analysis across literal and figurative language.

Conclusion

Based on the comparative analysis conducted before and after fine-tuning, it can be concluded that the integration of idiom detection and sentiment inference significantly enhanced the overall performance of the sentiment analysis model. Prior to fine-tuning, the hybrid model (BERT + CNN + Bi-LSTM) exhibited poor generalization on idiomatic expressions, with an overall accuracy of only 51.43% and a positive class recall of just 0.06, indicating its inability to correctly identify sentiment when idioms were present. This weakness highlighted the model’s lack of exposure to figurative language during training. After incorporating unknown idioms through a detection and inference pipeline—where sentiments were inferred via contextual web scraping and lexicon-based analysis—the model was fine-tuned on the extended dataset. Post-fine-tuning evaluation showed a remarkable improvement, with the model achieving an accuracy of 95.45%, a perfect recall for positive sentiments (1.00), and an overall F1-score of 0.95. These results demonstrate that: 1) Idiom detection is essential for real-world sentiment analysis tasks, particularly in domains like hotel reviews where figurative language is prevalent. 2) Sentiment inference for unknown idioms can be reliably performed using contextual search and lexicon-based methods. 3) Fine-tuning with the newly inferred idioms transforms the model from being overly literal to effectively understanding figurative speech. Thus, the final hybrid model proves to be robust, context-aware, and capable of accurately handling both literal and idiomatic reviews, establishing it as a reliable approach for sentiment analysis in real-world applications.

References

[1] L. Khan, A. Qazi, H.-T. Chang, M. Alhajlah, and A. Mahmood, \"Empowering Urdu sentiment analysis: an attention-based stacked CNN-Bi-LSTM DNN with multilingual BERT,\" Complex & Intelligent Systems, vol. 11, pp. 10, 2025. [Online]. Available: https://doi.org/10.1007/s40747-024-01631-9. [2] Wouter van Atteveldt, Mariken A. C. G. van der Velden & Mark Boukes,”The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms”,Communication Methods and Measures, 28 Jan 2021. [3] Nhan Cach Dang, María N. Moreno-García, Fernando De la Prieta, “Sentiment Analysis Based on Deep Learning: A Comparative Study”, Electronics, Volume 9, Issue 4, April 2020, ISSN 2079-9292. [4] Santwana Sagnika, Anshuman Pattanaik, Bhabani Shankar Prasad Mishra, Saroj K. Meher, “A Review on Multi-Lingual Sentiment Analysis by Machine Learning Methods”, Journal Articles, Indian Statistical Institute, 2020. [5] George Manias, Argyro Mavrogiorgou, Athanasios Kiourtis, Chrysostomos Symvoulidis, Dimosthenis Kyriazis, “Multilingual Text Categorization and Sentiment Analysis: A Comparative Analysis of the Utilization of Multilingual Approaches for Classifying Twitter Data”, Neural Computing and Applications, Volume 35, Issue 21415–21431, May 2023, DOI: 10.1007/s00521-023-08629-3. [6] Adwan OY, Al-Tawil M, Huneiti AM, Shahin RA, Abu Zayed AA, Al-Dibsi RH (2020) Twitter sentiment analysis approaches: a survey. Int J Emerg Technol Learn. https://doi.org/10.3991/ijet.v15i15.14467

Copyright

Copyright © 2025 S. Saraswathi, Athul K, S. Abhinav, S. Sivasankar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.