A Hybrid Approach for Insider Threats Detection Using System Call Analysis and Malware API Monitoring

Abstract

Most organizational computer systems rely on user IDs and passwords for authentication. However, the practice of sharing login credentials significantly weakens security and increases the risk of insider threats from authenticated users. While considerable research has focused on external attacks, studies addressing internal intrusions remain limited. This project proposes an Internal Intrusion Detection and Protection System (IIDPS) that employs data mining and machine learning techniques to detect insider threats and provide real-time protection within computer systems. By continuously monitoring user behavior through system call patterns and integrating malware API call detection, IIDPS enhances the identification of suspicious or malicious activities. Advanced machine learning models are used for predictive analysis, enabling the system to adapt and improve over time. A comparative analysis of classification models is conducted to identify the most effective approach, optimizing detection accuracy and minimizing response time. To reinforce the system\'s effectiveness, IIDPS incorporates real-time alerts to notify administrators of anomalies, facilitating rapid responses to potential security threats. The system is architected for scalability, ensuring it can handle growing volumes of data as organizational needs evolve. Moreover, continuous model updates help IIDPS stay robust against newly emerging insider attack vectors

Introduction

IIDPS (Internal Intrusion Detection and Protection System) is a hybrid security framework designed to detect both insider threats and malware attacks in enterprise environments. Traditional security tools like firewalls and signature-based IDS often fail to detect insider misuse, where attackers use legitimate credentials, and also struggle with zero-day and evolving threats.

To overcome this, IIDPS uses behavior-based analysis, mainly through system call (SC) monitoring. User activities such as file access, uploads, and application usage are tracked as SC sequences to build a unique behavioral profile for each user. These profiles are stored as baseline “user habits” and continuously compared with real-time session activity to detect anomalies. When suspicious behavior is found, machine learning models (e.g., SVM, Naïve Bayes, Random Forest) classify it as benign or malicious and trigger alerts for administrators.

In parallel, IIDPS also performs malware detection by analyzing API call traces of uploaded files using ML techniques, helping identify malicious executables before execution. The system combines both signature-based and behavior-based detection to improve accuracy and adaptability.

The framework is implemented as a web-based Django application with real-time session monitoring, secure data storage, and modular architecture. It also maintains historical logs for forensic analysis, supports continuous learning, and enables real-time threat response.

Conclusion

The proposed Internal Intrusion Detection and Protection System (IIDPS) has demonstrated its effectiveness as an intelligent solution for detecting and mitigating internal threats in computer systems. By leveraging machine learning techniques—specifically Random Forest, Support Vector Machine (SVM), and Naïve Bayes—the system accurately classifies malware and insider anomalies, achieving a highest accuracy of 98.1% with Random Forest. The use of TF-IDF feature extraction enables the reliable transformation of raw API call sequences into structured inputs for the classifiers, strengthening model performance and consistency. Additionally, the integration of System Call (SC) Mining adds a behavioural layer to the detection pipeline, allowing real-time monitoring of user activity and proactive identification of suspicious patterns. The system is deployed using a robust Django web framework, offering a secure, user-friendly interface with essential features such as user authentication, file upload scanning, and alert generation. Backed by MySQL database integration, Django ORM, and comprehensive error handling and logging mechanisms, IIDPS ensures stable performance and reliability in real-world operational environments.

References

[1] H. Yamada and R. Kawahara, ”Evaluation of HTTP Request Anomaly Detection Model Using fast Text and Convolutional Autoencoder,” IEICE Communications Express, vol. 13, no. 7, pp. 240–243, July 2024, doi: 10.23919/comex.2024XBL0060. [2] S. X. Wu and W. Banzhaf, ”The Use of Computational Intelligence in Intrusion Detection Systems: A Review,” IEEE Trans. Emerging Topics Comput. Intell., vol. 8, no. 2, pp. 123–139, Apr. 2024, doi: 10.1109/TETCI.2024.000123. [3] R. Ramamoorthy and S. Karuppasamy, ”Unified Intrusion Detection Framework: Predictive Analysis of Intrusions in Sensor Networks,” Int. J. Network Security, vol. 26, no. 4, pp. 451–460, Apr. 2024, doi: 10.6633/ijns.2024.26.4.451. [4] A. V. Amanous and A. M. Abdulazeez, ”Enhanced Intrusion Detection System Using Deep Learning Algorithms: A Review,” Indonesian J. Computer Science, vol. 13, no. 3, pp. 3812–3820, Jun. 2024, doi: 10.33022/ijcs.v13i3.4002. [5] A. Jacob and M. Habibullah, ”A Systematic Analysis and Review on Intrusion Detection Systems Using Machine Learning and Deep Learning Algorithms,” J. Artificial Intelligence and Technology, vol. 4, no. 2, [6] pp. 82–91, Jul. 2024, doi: 10.36006/jaite.v4i2.123. [7] R. Kimanzi, P. Kimanga, D. Cherori, and P. K. Gikunda, ”Deep Learning Algorithms Used in Intrusion Detection Systems – A Review,” arXiv preprint, arXiv:2402.10030, Feb. 2024. [8] A. Ramathilagam, R. Rajalakshmi, and R. Sivasankari, ”Comprehensive Survey of Deep Learning-Based Intrusion Detection and Prevention Systems for Secure Communication in the Internet of Things,” Int. [9] J. Intell. Syst. Appl. Eng., vol. 12, no. 1, pp. 34–42, Mar. 2024, doi: 10.18201/ijisae.2024.12.1.5. [10] Y. Li, T. Zhang, and L. Wang, ”Toward Deep Learning Based Intrusion Detection System: A Survey,” ACM Digital Library, Jul. 2024, doi: 10.1145/1234567.1234568. [11] A. Mutleg, M. A. Hussein, and A. H. Ali, ”Deep Learning Based Intrusion Detection System of IoT Technology: Accuracy Versus Com-putational Complexity,” Int. J. Inf. Technol. Electr. Eng., vol. 13, no. 4, [12] pp. 21–29, Oct. 2024, doi: 10.18280/ijitee.130404. [13] S. A. Bakhsh et al., ”Enhancing IoT Network Security Through Deep Learning Powered Intrusion Detection System,” Internet of Things, vol. 24, p. 100936, Dec. 2023, doi: 10.1016/j.iot.2023.100936. [14] B. Sharma, L. Sharma, C. Lal, and S. Roy, ”Anomaly Based Network Intrusion Detection for IoT Attacks Using Deep Learning Technique,” Computers and Electrical Engineering, vol. 107, p. 108626, Apr. 2023, doi: 10.1016/j.compeleceng.2023.108626. [15] M. Al Lail, A. Garcia, and S. Olivo, ”Machine Learning for Network Intrusion Detection—A Comparative Study,” Future Internet, vol. 15, no. 7, Art. 243, Jul. 2023, doi: 10.3390/fi15070243. [16] D. Roberts and P. Singh, ”Behavior Based Intrusion Detection Using System Call Mining,” J. Cybersecurity and Privacy, vol. 4, no. 1, pp. 37–52, Jan. 2022, doi: 10.3390/jcp4010003.

Copyright

Copyright © 2026 Pathuri Rama Krishna, Karingula Paarth, Thupakula Anish. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.