Explainable and Feature-Engineered Machine Learning for Stock Market Forecasting: A Survey of Lexicon-Based, Statistical, and Sentiment-Driven Approaches

Abstract

Forecasting the direction and magnitude of stock price movements remains one of the most persistently difficult problems in applied machine learning, owing to the noisy, non-stationary, and weak-form-efficient character of financial time series. This survey examines three recent studies that approach the problem through complementary lenses: an explainable Decision Tree that builds dynamic, industry-specific lexicons from financial news to classify next-day price-change magnitude [1]; a feature-engineering and portfolio-construction pipeline that derives new price-based predictors via Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and a proposed Statistical Average Method (SAM) before applying eight regression architectures, including Bi-LSTM [2]; and an LSTM-based trend classifier that fuses FinBERT-scored Twitter sentiment with price data to forecast North American and European bank-stock movements during the macroeconomically turbulent year of 2022 [3]. Beyond summarising each study\'s objectives, methodology, data, and results, this survey situates the three works within the broader and rapidly evolving literature on deep sequence models (LSTM, Bi-LSTM, GRU, and transformer architectures), sentiment analysis with pretrained financial language models such as FinBERT, and explainable artificial intelligence (XAI) in finance. A structured comparative analysis contrasts the three studies along ten dimensions, including data sources, feature engineering, model architecture, explainability, validation design, and reported performance, while explicitly cautioning against directly ranking studies whose prediction tasks and evaluation metrics are not commensurable. A cross-domain case study of an attention-based legislative-vote predictor is retained to examine whether a learned similarity mechanism could improve on the fixed similarity rules used throughout the finance literature reviewed here. The survey\'s contribution lies in bringing explainable lexicon-based methods, statistically engineered price features, and sentiment-driven deep learning together within a single comparative framework, and in using that synthesis to identify concrete, evidence-grounded gaps around auxiliary-signal fusion, the explainability of high-accuracy sequence models, and cross-market generalisation.

Introduction

The text reviews the application of machine learning (ML) and deep learning (DL) techniques to stock market forecasting, focusing on three studies that aim to improve prediction accuracy and interpretability by combining stock price data with additional features or sentiment information.

The first study, by Carta et al., uses financial news to create sector-specific lexicons and extract meaningful features for a Decision Tree model. Its main advantage is explainability, as it provides understandable rules for predicting whether stock price changes will be high or low. However, its reliance on word matching limits its ability to understand context and semantics.

The second study, by Sami et al., enhances traditional stock price attributes using Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and a Statistical Average Method (SAM). Various machine learning and deep learning models are evaluated, with a Bi-LSTM model using SAM-enhanced features achieving the best reported results. The predicted prices are then used for portfolio optimization, demonstrating the potential of feature engineering to improve forecasting and investment performance.

The third study, by Bacco et al., combines Long Short-Term Memory (LSTM) networks with financial sentiment extracted from Twitter using FinBERT. It examines bank-stock trends in the United States and Europe during 2022. The results show better prediction performance for US banks than European banks, while adding sentiment information provides mixed benefits depending on the market and company.

The literature review also discusses recent developments in stock forecasting, including transformer-based architectures, attention mechanisms, hybrid sentiment models, feature engineering, and explainable artificial intelligence (XAI). A cross-domain case study on legislative voting prediction further highlights the potential of attention-based similarity mechanisms for improving predictive models.

Overall, the three studies demonstrate that incorporating financial news, engineered price features, and social media sentiment can enhance stock market prediction. However, their effectiveness varies according to the model, dataset, market conditions, and feature integration method. Key limitations include restricted datasets, inconsistent sentiment benefits, limited interpretability in deep learning models, and challenges in generalizing results across markets.

Conclusion

This survey has brought together three recent and complementary approaches to machine-learning-based stock market forecasting: an explainable, lexicon-based Decision Tree method [1]; a feature-engineering and portfolio-optimisation pipeline built around PCA-, LDA-, and statistical-average-derived price attributes combined with Bi-LSTM forecasting [2]; and an LSTM model enriched with FinBERT-derived Twitter sentiment, evaluated across US and EU banking markets under conditions of high macroeconomic uncertainty [3]. Together, these studies show that auxiliary information — textual, statistical, or sentiment-based — can meaningfully improve on price-only forecasting, though its value is neither universal nor guaranteed: engineered features can fail statistical significance tests [2], lexicon-based signals can transfer unevenly across sectors [1], and sentiment can help in one market while offering little in another [3]. Reading the three papers against the broader and more recent literature reviewed in Section 3 sharpens this picture in two ways. First, the field has largely moved, or is moving, toward transformer-based and more tightly fused hybrid architectures that none of the three focal papers evaluates, suggesting that some of their reported limitations — particularly the uneven benefit of sentiment reported by Bacco et al. [3] — may be as much a function of architectural and fusion choices as of the underlying value of the auxiliary signal itself. Second, the systematic XAI-in-finance literature confirms that the trade-off between explainability and predictive accuracy visible across these three papers is not idiosyncratic to them but reflects a broader pattern in the field, where post-hoc explanation of high-accuracy deep models remains comparatively underdeveloped [5]. A short cross-domain case study of an attention-based legislative-prediction model [4] further suggests that replacing the fixed, hand-specified similarity and aggregation formulas used throughout this finance literature with a learned, data-driven similarity mechanism is a promising direction that remains largely unexplored here. Future work that combines richer, explainable, and adaptively weighted auxiliary information — rather than treating lexicon-, feature-, and sentiment-based approaches as mutually exclusive choices — appears to be the most promising route toward stock market forecasting systems that are more accurate, more generalisable, and more trustworthy. This survey does not claim to resolve that trade-off, nor to guarantee that any of the directions proposed in Section 14 will succeed; it aims only to state the evidence for these gaps as precisely as the surveyed literature allows.

References

[1] S. M. Carta, S. Consoli, L. Piras, A. S. Podda, and D. Reforgiato Recupero, “Explainable machine learning exploiting news and domain-specific lexicon for stock market forecasting,” IEEE Access, vol. 9, pp. 30193–30205, 2021. [2] H. M. Sami, M. S. Jalal, M. F. Kabir, and M. N. Huda, “IntPort: An intelligent portfolio construction technique based on financial forecasting by statistical average method,” IEEE Access, vol. 13, pp. 35355–35375, 2025. [3] L. Bacco, L. Petrosino, D. Arganese, L. Vollero, M. Papi, and M. Merone, “Investigating stock prediction using LSTM networks and sentiment analysis of tweets under high uncertainty: A case study of North American and European banks,” IEEE Access, vol. 12, pp. 122239–122248, 2024. [4] B. Wang, Y. Li, and C. Xu, “Exploring bill similarity with attention mechanism for enhanced legislative prediction,” Journal of Social Computing, vol. 6, no. 2, pp. 112–125, 2025. [5] J. ?ernevi?ien? and A. Kabašinskas, “Explainable artificial intelligence (XAI) in finance: a systematic literature review,” Artificial Intelligence Review, vol. 57, no. 8, article 216, 2024. [6] J. Zou, Q. Zhao, Y. Jiao, H. Cao, Y. Liu, Q. Yan, E. Abbasnejad, L. Liu, and J. Q. Shi, “Stock market prediction via deep learning techniques: A survey,” arXiv preprint arXiv:2212.12717, 2022 (revised 2023). [7] D. Araci, “FinBERT: Financial sentiment analysis with pre-trained language models,” arXiv preprint arXiv:1908.10063, 2019. [8] W. Lu, J. Li, J. Wang, R. Zhang, and Y. Qin, “A CNN-BiLSTM-AM method for stock price prediction,” Neural Computing and Applications, vol. 33, pp. 4741–4753, 2021. [9] Y. Nejatbakhsh and M. Aliasgari, “Enhancing stock market prediction with hybrid deep learning: Integrating LSTM, transformer attention, federated learning, and sentiment analysis,” IEEE Access, vol. 14, pp. 3926–3942, 2025. [10] P. Sonkiya, V. Bajpai, and A. Bansal, “Stock price prediction using BERT and GAN,” arXiv preprint arXiv:2107.09055, 2021. [11] W. Jiang, “Applications of deep learning in stock market prediction: Recent progress,” Expert Systems with Applications, vol. 184, article 115537, 2021. [12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2019, pp. 4171–4186.

Copyright

Copyright © 2026 Sidharth S, Harichandana H S, Albin Tom Thomas, Akhilesh P A, Mrs. Sagara M R. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.