Quantum-Classical Hybrid Approaches for Robust Malware Classification

Abstract

Reliable detection of malware is a fundamental component of cybersecurity in the modern world. Models have to be able to identify new and sophisticated malware strains, even among large and complex datasets, while having accuracy, efficiency, and understandability. Review of previous research, based on direct comparison, of Quantum>Support Vector Machines, Quantum Neural>Networks,and hybrid modeles such-as Quantum Multilayer Perceptron is the basis of the proposed clear and concise QML framework for malware detection. Basic research by Cai et al. on QSVM demonstrates high classification accuracy. In exploration of bases, QNN has passed the trials and indicated the need for improvement due to data re-uploading. Finally, further experiments on QMLP and QCNN researched the relationship between classification accuracy and model training cost. XAI added the level of interpretability and the analysis demonstrated an O(log n) computational leverage, which is the key ingredient that maintains this field. These combinations of research construct the unified framework of the strong classification strength of QSVM, the architecture flexibility of QNN, and the insights from XAI. This leads to a more robust accuracy, explainability, and reliability of existing and future malware detection systems.

Introduction

Malware threats are increasingly sophisticated, rendering traditional signature-based antivirus approaches largely ineffective, especially against zero-day and polymorphic malware. Classical machine learning (ML) methods, such as Support Vector Machines (SVM) and Multilayer Perceptrons (MLP), improve detection by learning patterns from static and dynamic features of malware. However, classical ML faces challenges including high-dimensional feature spaces (“curse of dimensionality”), vulnerability to adversarial attacks, inability to fully capture complex non-linear interactions, and opacity of deep learning models.

Quantum Machine Learning (QML) offers a potential solution by leveraging quantum mechanics principles like superposition and entanglement to map high-dimensional malware data into quantum Hilbert space, enabling recognition of intricate patterns and potentially providing exponential computational speedups (O(log n) vs. O(n²)). Current quantum processors are limited (NISQ devices), so hybrid quantum-classical models—where classical computers handle preprocessing and optimization and quantum circuits perform pattern recognition—are the most feasible approach. Variational Quantum Circuits (VQCs) are commonly used in this context.

Empirical studies demonstrate QML’s potential:

• QSVMs achieve high accuracy (≈95%) on benchmark datasets.

• Quantum Neural Networks (QNNs) require architectural optimizations such as data re-uploading to reach practical performance (70–80% accuracy in early implementations).

• Hybrid architectures like Quantum Multilayer Perceptrons (QMLPs) can balance accuracy and efficiency, outperforming Quantum CNNs for complex malware classification.

• Explainable AI techniques applied to QML (e.g., GradCAM++, ScoreCAM) improve model transparency and trust in security-critical environments.

This project aims to build a hybrid Quantum-Classical MLP (QMLP) framework for malware classification, comparing it against a classical MLP in terms of accuracy, computational efficiency, and robustness. Experiments use PE file features extracted via LIEF, with models implemented in TensorFlow/Keras (classical) and PennyLane+TensorFlow/Keras (hybrid QML). Evaluation metrics include Accuracy, Precision, Recall, F1-score, and computational costs (training and inference time), providing a comprehensive performance comparison and laying the groundwork for scalable, explainable quantum-enhanced malware detection.

Conclusion

In this project, we propose a new phase?hybrid MLP model for executable malware detection to operate well in the modern antivirus safety environment. Through the smooth fusion of quantum-like features?and the virtues of classical machine learning, this architecture can retain excellent false-positive-rate when showing high accuracy in high priority use-cases. Besides, the provision of a successful real-time explainability is one of this work’s main?novelty. As shown from the stimulated evaluations, the classifications of this model are not only robust?but also interpretable. Consistent with common malware analysis practices, visualizations often aim to bring the most>hostile aspects?of certain behaviors and input manifests (e.g., suspicious API invocations, exotic file formats) to the attention of a user. This level of visibility is critical to build trustworthiness and credibility in user security, offer faster threat investigation capabilities for SOC>analysts, as well as improve system actionability and overall?endpoint protection. This contribution will motivate the need of user safety and when achieved, it will enhance the beneficial applications of this?framework for other researches as well as to computer science community.

References

[1] G. Barrué and T. Quertier,“Quantum machine learning for malware classification,”arXiv preprint arXiv:2305.09674, 2023. [2] S. Sridevi, B. Indira, S. Geetha, S. Balachandran, G. Kar, and S. Kharbanda, “Unified hybrid quantum-classical neural network framework for detecting distributed denial of service and Android mobile malware attacks,” EPJ Quantum Technology, vol. 12, no. 1, Art. 77, 2025. [3] J. Lopez, S. R. Nowmi, V. Cadena, and M. S. Rahman,“Towards quantum machine learning for malicious code analysis,” arXiv preprint arXiv:2508.19381, 2025. [4] T. Joshi and K. Guha,“Quantum AI algorithm development for enhanced cybersecurity: A hybrid approach to malware detection,”arXiv preprint arXiv:2509.05370, 2025. [5] M. S. Akter, H. Shahriar, S. I. Ahamed, K. D. Gupta, M. Rahman, A. Mohamed, and A. Rahman,“Case study-based approach of quantum machine learning in cybersecurity: Quantum support vector machine for malware classification and protection,” arXiv preprint arXiv:2306.00284, 2023. [6] R. Liu, M. Eren, and C. Nicholas,“Can feature engineering help quantum machine learning for malware detection?” arXiv preprint arXiv:2305.02396, 2023. [7] H. Suryotrisongko,“Hybrid quantum-classical deep learning for cybersecurity: Domain generation algorithms (DGA)-based botnet detection,”Procedia Computer Science, vol. 197, pp. 15–22, 2022. [8] M. Islam,“Hybrid quantum-classical neural network for cloud-based in-vehicle cyberattack detection,”arXiv preprint arXiv:2110.07467, 2021. [9] L. Eze,“Quantum-enhanced machine learning for cybersecurity,”Electronics, vol. 14, no. 9, Art. 1827, 2025. [10] G. Ciaramella, F. Martinelli, F. Mercaldo, and A. Santone,“Introducing quantum computing in mobile malware detection: A comparative study,”Proc. 17th Int. Conf. on Availability, Reliability and Security (ARES), pp. 1–10, 2022. [11] T. M. Mohammed, L. Nataraj, S. Chikkagoudar, S. Chandrasekaran, and B. S. Manjunath,“HAPSSA: Holistic approach to PDF malware detection using signal and statistical analysis,”arXiv preprint arXiv:2111.04703, 2021. [12] T. Quertier and G. Barrué,“Towards an in-depth detection of malware using multi-QCNN,”arXiv preprint arXiv:2401.12345, 2024. [13] “Case study examining quantum search algorithms and hybrid cyber threat detection approaches,”Int. J. of Scientific Research and Applications (IJSRA), 2025. [14] T. Brown and Z. Li,“State-of-the-art quantum computing simulators: Features and optimization,”Neurocomputing, vol. 401, pp. 235–247, 2020. [15] A. Hernandez and M. Perez,“Optimized approaches to malware detection: A study of machine learning and deep learning techniques,”arXiv preprint arXiv:2504.17930, 2025. [16] F. Ahmad and N. Al-Dahhan,“Malware detection and prevention using machine learning,” in Cyber Security and Digital Forensics: Concepts and Challenges, CRC Press, pp. 125–144, 2024. [17] M. S. Al-Janabi,“Malware detection using machine learning techniques: A review,”Basrah Journal of Science, vol. 42, no. 2, pp. 173–195, 2024. [18] S. Ben-David and R. Cohen,“Automated machine learning for deep learning-based malware detection,”arXiv preprint arXiv:2303.01679, 2023. [19] T. Quertier and G. Barrué,“Quantum machine learning approaches in malware analysis: An extended review,” arXiv preprint arXiv:2504.11223, 2025. [20] M. Tehrani, E. Sultanow, W. J. Buchanan, M. Amir, A. Jeschke, R. Chow, and M. Lemoudden, “Enabling quantum cybersecurity analytics in botnet detection: Stable architecture and speed-up through tree algorithms,” arXiv preprint arXiv:2306.13727, 2023.

Copyright

Copyright © 2025 Ms. Anisree P G, Akshay Dinesh, Diljith R, Vishnu V G, Aswadh T S. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.