Explainable Malware Detection Framework Using Memory-Based Machine Learning, Windows Live Monitoring, and Containment

Abstract

The advancement of malware in areas such as obfuscation and fileless is among the important factors affecting the efficiency of static analysis. The memory analysis technique has the potential to provide insights into how processes operate and detect attributes of possible malicious software; something which can only be achieved partially by static analysis. Many machine learning methods utilizing memory analysis characteristics have been tested in offline experiments, thus creating a gap between Windows endpoint online analysis and offline malware detection. Furthermore, lack of interpretability in many machine learning techniques makes the use of memory analysis features hard in malware detection. This research work offers a transparent algorithm for the detection of malware that consists of adaptability, process containment, process monitoring on Windows OS, interpretation using SHAP, and a memory-centric machine learning framework. Malware detection framework suggested by this study is trained using ObfuscatedMalMem2022 dataset, which includes 58,596 data instances along with 52 memory features, collected using Volatility software. In order to perform the classification, a stacked ensemble classifier is developed using Random Forest, XGBoost, and LightGBM as base classifiers, while Logistic Regression is used as the metaclassifier. The proposed ensemble achieves 98.17% accuracy, 98.43% precision, 97.92% recall, 98.17% F1-score, and 0.9822 AUC-ROC on the held-out test set. SHAP-based analysis is employed to interpret the model predictions and determine which memory-derived properties have the greatest influence, with process-handle and service-related attributes being the most important contributors. To implement the offline method for real-time endpoint monitoring, a Windows process monitoring module based on psutil gathers run-time process telemetry, leveraging feature adaptation to generate the feature representation necessary for the classifier. As some Volatility-based features cannot be monitored via user-space monitoring, the missing features are represented by benign baselines from training data. The generated risk score is then used to classify the process as being one of three types: safe, suspicious, and malware-risk. The Framework provides mechanisms for process suspension and executable isolation for high-risk processes. Thus, the viability of the integration of an explainable memory-based malware classification system with Windows live process monitoring and containment is demonstrated, yet the mismatch between training features based on memory and available telemetry is the most significant limitation to live operation.

Introduction

The text presents an explainable, machine-learning-based malware detection and response framework for Windows. Its main goal is to overcome the limitations of traditional signature-based antivirus systems by detecting modern malware—including obfuscated, fileless, and memory-resident threats—using memory-derived features, ensemble machine learning, explainable AI, live process monitoring, and automated containment.

Key points

• Problem: Modern malware uses techniques such as polymorphism, metamorphism, encryption, and fileless execution, making traditional signature-based detection less effective.

• Memory-based detection: Malware leaves traces in RAM while executing. Tools such as Volatility can extract these memory artifacts, which can be useful for identifying malicious processes.

• Machine learning: The proposed system uses a stacking ensemble consisting of:

  • Random Forest

  • XGBoost

  • LightGBM

  • Logistic Regression as the meta-learner

• Dataset: The model is trained on the Obfuscated-MalMem2022 dataset, containing 58,596 samples—29,298 benign and 29,298 malware—with 52 memory-derived features.

• Explainable AI: SHAP is used to explain why the model classifies a sample as malware by identifying the features that contribute most strongly to each prediction.

• Live Windows monitoring: Since offline memory datasets contain features that are not directly available during live monitoring, the system uses psutil to collect runtime information such as process IDs, CPU usage, memory consumption, threads, and handles.

• Feature adaptation: A feature-adaptation mechanism converts available live process information into the 52-feature format expected by the trained model. Missing memory-specific features are supplemented using benign baseline values.

• Risk classification and response: The model produces a malware probability that is converted into a risk category. High-risk processes can trigger automated actions such as suspending or terminating the process, isolating artifacts, and checking system registry settings.

• Malware family identification: After detecting malware, process names, executable paths, and command-line arguments are examined to classify threats into categories such as Ransomware, Spyware, and Trojans.

• Research gap: Existing studies often address memory analysis, machine learning, explainability, live monitoring, and automated response separately. The proposed work combines these components into one framework, particularly addressing the difficult mismatch between offline memory features and live Windows telemetry.

Overall workflow

Offline stage:

Obfuscated-MalMem2022 → Preprocessing → RF + XGBoost + LightGBM → Logistic Regression → SHAP Explainability

Online stage:

Windows Processes → Live Monitoring → Feature Adapter → Scaling → Stacking Model → Risk Assessment → Malware Classification → Automated Containment

Conclusion

This research presented an explainable malware detection framework that integrates containment, Windows live process monitoring, SHAP-based interpretability, and memory-based machine learning into a single architecture. On the examined unseen test dataset, the suggested Stacking Ensemble achieved 98.17% accuracy, 98.43% precision, 97.92% recall, 98.17% F1-score, and 0.9822 ROC-AUC by integrating Random Forest, XGBoost, and LightGBM via a Logistic Regression meta-learner. A Windows live-monitoring Framework was created to showcase the integration of runtime process telemetry, feature adaption, risk assessment, and containment in addition to the offline detection and explainability components. The main goal of future work will be to close the feature-space gap between live Windows telemetry and the memory-based training environment. To lessen reliance on baseline representations for features that are currently unavailable through psutil, more thorough Windows telemetry and deeper memory-level feature acquisition will be explored. In order to quantify real-time detection accuracy, false-positive behavior, response latency, and containment dependability, the live-monitoring and containment components will also need to be systematically evaluated in an isolated environment using controlled, confirmed malware and benign-process datasets. To assess how well the suggested approach generalizes outside of the current experimental environment, other research can look into larger datasets and different malware types. Additionally, future iterations of this framework could replace the heuristic threat-typing engine with a dedicated multiclass machine learning model, allowing the system to dynamically categorize entirely novel malware families without relying on predefined execution keywords.

References

[1] J. Ferdous, R. Islam, A. Mahboubi, and M. Z. Islam, “A Review of [2] State-of-the-Art Malware Attack Trends and Defense Mechanisms,” IEEE Access, vol. 11, pp. 121118–121141, 2023. [3] S. Chandran, S. R. Syam, S. Sankaran, T. Pandey, and K. Achuthan, “From Static to AI-Driven Detection: A Comprehensive Review of Obfuscated Malware Techniques,” IEEE Access, vol. 13, 2025. [4] R. Sihwail, K. Omar, and K. A. Z. Ariffin, “An Effective Memory Analysis for Malware Detection and Classification,” Computers, Materials & Continua, vol. 67, no. 2, pp. 2301–2320, 2021. [5] D. Vekariya, M. Patidar, M. Amin, and C. Padhiyar, “Memory Forensics [6] Using the Volatility Framework: A Structured Approach for Detecting [7] Fileless Malware,” in 2025 IEEE 5th International Conference on ICTBIG, 2025. [8] A. Almuqren, M. Frikha, and A. Albuali, “Automated Malware Detection Based on a Machine Learning Algorithm,” in 2023 IEEE 10th Int. Conf. on ComNet, 2023. [9] A. Brown, M. Gupta, and M. Abdelsalam, “Automated Machine [10] Learning for Deep Learning Based Malware Detection,” Computers & Security, vol. 137, 103582, 2024. [11] A. Galli, V. La Gatta, V. Moscato, M. Postiglione, and G. Sperlì, [12] “Explainability in AI-based behavioral malware detection systems,” Computers & Security, vol. 141, 103842, 2024. [13] M. Rhode, P. Burnap, and A. Wedgbury, “Real-Time Malware Process Detection and Automated Process Killing,” Security and Communication Networks, vol. 2021, 8933681, 2021. [14] B. Yu, Y. Fang, Q. Yang, Y. Tang, and L. Liu, “A Survey of Malware Behavior Description and Analysis,” FITEE, vol. 19, no. 5, pp. 583–603, 2018. [15] T. Vyšniunas, D. ?eponis, N. Goranin, and A. ?enys, “Risk-Based System-Call Sequence Grouping Method for Malware Intrusion [16] Detection,” Electronics, vol. 13, no. 1, 206, 2024. [17] S. Chidambaram and S. M. P., “MemLOL: Memory-Based LOLBins Dataset for Fileless Malware Detection,” in 2025 13th ISDFS, 2025. [18] I. Kara and K. Y?lmaz, “HAP-Analyzer: A Hybrid Static, Dynamic, and Memory Analysis Approach for Fileless Malware,” IEEE Access, vol. 14, 2026. [19] B. Ç. Sar? and ?. F. K?l?nçer, “Hybrid Deep Learning Model for Memory-Based Malware Detection,” in 9th Int. IDAP Symp., 2025. [20] Y. Imamverdiyev, E. Baghirov, and J. C. Ikechukwu, “Detecting [21] Obfuscated Malware Infections on Windows Using Ensemble Learning Techniques,” Informatics and Automation, vol. 24, no. 1, pp. 99–124, 2025. [22] M. Torres, R. Álvarez, and M. Cazorla, “A Malware Detection [23] Approach Based on Feature Engineering and Behavior Analysis,” IEEE Access, vol. 11, pp. 105355–105367, 2023. [24] B. Yu et al., “Explainable AI for Malware Analysis: A Systematic [25] Review of Benchmark Datasets, Traffic-Oriented Detection, and Interpretability Methods,” PeerJ Computer Science, vol. 12, e3902, 2026. [26] S. R. M. Zeebaree, “A New Approach for Process Monitoring,” [27] Polytechnic Journal, vol. 1, no. 1, 2011. [28] H. Bandara et al., “Stealth Eye: Behavioral Analysis for Fileless Malware Detection,” in 13th ISDFS, 2025. [29] B. Pal, G. H. Reddy, S. Saurav, and A. R. Vasudevan, “Detection of Fileless Malware,” in 2025 5th ICSC, 2025. [30] R. Sun et al., “Learning Fast and Slow: PROPEDEUTICA for Real-time Malware Detection,” arXiv preprint arXiv:1712.01145, 2021.

Copyright

Copyright © 2026 Cheekurumelli Rupa Sri, Dr. Kuda Nageswara Rao. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.