Enhanced Sentiment Analysis Using RoBERTa and BiLSTM: A Context-Aware Hybrid Deep Learning Approach

Abstract

With the widespread growth of digital platforms, online interaction has become an essential part of everyday life. Users frequently express their opinions, feedback, and emotions through reviews and comments on various platforms. Analyzing such textual data plays a crucial role in understanding user sentiment and supporting effective decision-making. However, sentiment analysis faces several challenges, including long-range dependencies within text and the presence of unknown words and symbols. Traditional sentiment analysis approaches mainly rely on sequential models, which process text step by step and often require higher computational time. In contrast, Transformer-based models offer improved efficiency through parallel processing. To address these challenges, this paper presents a context-aware hybrid deep learning approach by integrating the Robustly Optimized BERT Pretraining Approach (RoBERTa) with Bidirectional Long Short-Term Memory (BiLSTM) networks. RoBERTa is employed to generate rich contextual word embeddings, while BiLSTM captures long-term semantic dependencies by processing text in both forward and backward directions. The proposed model is trained and evaluated on the Twitter US Airline Sentiment dataset comprising 14,299 samples across three sentiment classes. Experimental analysis demonstrates that the hybrid approach achieves an accuracy of 85.14% and an F1-score of 0.8487, highlighting its effectiveness for sentiment analysis tasks compared to baseline models.

Introduction

This study proposes a hybrid deep learning model for sentiment analysis by combining RoBERTa and Bidirectional Long Short-Term Memory (BiLSTM) to improve sentiment classification of social media text. With the rapid growth of user-generated content on digital platforms, sentiment analysis has become essential for understanding public opinions in areas such as business, marketing, customer service, and social media monitoring. Traditional machine learning and sequential deep learning models often struggle with sarcasm, contextual meaning, long-range dependencies, and noisy social media language.

The proposed approach addresses these challenges by using RoBERTa to generate contextual word embeddings and BiLSTM to capture bidirectional sequential dependencies in text. This combination leverages the strengths of Transformer-based contextual understanding and recurrent sequence learning to improve classification accuracy for positive, negative, and neutral sentiments.

The literature review highlights previous work demonstrating that hybrid models combining Transformer architectures with recurrent networks outperform standalone machine learning and deep learning models. Prior studies also emphasize the effectiveness of attention mechanisms and hybrid architectures for handling noisy and complex social media text.

The proposed methodology consists of several stages: data collection, preprocessing, contextual feature extraction, sequence learning, and classification. The model is trained using the Twitter US Airline Sentiment dataset from Kaggle, which contains 14,299 labelled tweets. During preprocessing, Twitter handles, URLs, hashtags, punctuation, and other noise are removed, text is converted to lowercase, and tokenization is performed using the RoBERTa tokenizer. The data is then split into 80% training and 20% testing sets.

The architecture uses the pretrained RoBERTa-base model to generate 768-dimensional contextual embeddings, followed by a two-layer BiLSTM with 256 hidden units per direction to capture sequential information. A dropout layer reduces overfitting, and a fully connected layer with a softmax classifier predicts one of the three sentiment classes. The model is trained using the AdamW optimiser, CrossEntropyLoss, and evaluated using accuracy, precision, recall, and F1-score.

Experimental results show that the proposed hybrid model achieves an accuracy of 85.14%, weighted precision of 84.78%, weighted recall of 85.14%, and a weighted F1-score of 84.87%. The model performs best on the negative sentiment class due to its larger number of training samples, while the neutral class remains the most difficult to classify because of class imbalance. Overall, the results demonstrate that combining RoBERTa with BiLSTM provides an effective and robust solution for sentiment analysis on real-world social media data.

Conclusion

This paper presented an enhanced sentiment analysis system using a hybrid deep learning approach combining RoBERTa and BiLSTM. The proposed model leverages the contextual understanding capability of RoBERTa through its Transformer-based self-attention mechanism and the sequential learning capability of BiLSTM to capture bidirectional dependencies in text. The model was trained and evaluated on the Twitter US Airline Sentiment dataset containing 14,299 samples across three sentiment categories. The experimental results demonstrate that the proposed hybrid model achieves an accuracy of 85.14%, a weighted precision of 0.8478, a weighted recall of 0.8514, and a weighted F1-score of 0.8487. The per-class analysis shows strong performance on the negative class (F1-score of 0.9124) with competitive results on the positive class (F1-score of 0.7910). The system was implemented as a Streamlit-based web application supporting both single text and batch prediction with confidence scores and visualizations. Future work includes extending the system to support multilingual sentiment analysis for processing text in multiple languages. The model can be enhanced by integrating real-time data from social media platforms for live sentiment monitoring. Deployment as a full-scale web or mobile application can improve accessibility and usability. Furthermore, incorporating attention mechanisms and explainable AI methods can provide better interpretability of predictions. Domain-specific sentiment analysis for healthcare, finance, and other specialized domains can also be explored. Advanced ensemble techniques and larger pretrained models may further improve classification performance.

References

This paper presented an enhanced sentiment analysis system using a hybrid deep learning approach combining RoBERTa and BiLSTM. The proposed model leverages the contextual understanding capability of RoBERTa through its Transformer-based self-attention mechanism and the sequential learning capability of BiLSTM to capture bidirectional dependencies in text. The model was trained and evaluated on the Twitter US Airline Sentiment dataset containing 14,299 samples across three sentiment categories. The experimental results demonstrate that the proposed hybrid model achieves an accuracy of 85.14%, a weighted precision of 0.8478, a weighted recall of 0.8514, and a weighted F1-score of 0.8487. The per-class analysis shows strong performance on the negative class (F1-score of 0.9124) with competitive results on the positive class (F1-score of 0.7910). The system was implemented as a Streamlit-based web application supporting both single text and batch prediction with confidence scores and visualizations. Future work includes extending the system to support multilingual sentiment analysis for processing text in multiple languages. The model can be enhanced by integrating real-time data from social media platforms for live sentiment monitoring. Deployment as a full-scale web or mobile application can improve accessibility and usability. Furthermore, incorporating attention mechanisms and explainable AI methods can provide better interpretability of predictions. Domain-specific sentiment analysis for healthcare, finance, and other specialized domains can also be explored. Advanced ensemble techniques and larger pretrained models may further improve classification performance.

Copyright

Copyright © 2026 Dr. Veguru Gayatri, Dr. Rajani Rajalingam . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.