InsightLog: A Lightweight Framework for Explainable Incident Detection in Linux System Logs

Abstract

Modern Linux systems generate massive volumesof authentication and system logs that capture critical security events. However, traditional log analysis methods struggle with the high volume, unstructured format and noisy nature of these logs, making timely detection of suspicious activity difficult, especially in small-scale and resource-constrained environments. EnterpriseSecurityInformationandEventManagement(SIEM) solutionsareoftentoocomplexandexpensive,whilebasiclog viewers offer no automated analysis. This paper presents InsightLog, a lightweight, rule-based framework for explainable incident detection in Linux environments. The framework per- forms structured log parsing using regular expressions, rule- based anomaly detection with temporal window analysis,and incident correlation to transform unstructured log data into coherentincidentcontexts.Ahuman-in-the-loopdecisionsupport interface provides explainable summaries and evidence-linked recommendations, requiring explicit operator approval before response execution. Experimental evaluation demonstrates that InsightLog processes 5,200 log entries per second with 99.2% parsing accuracy, achieves a 96.5% detection rate with 2.3%false positives, reduces alert noise by 78% and consumes only 120MB RAM and 8-12% CPU on standard hardware. The framework provides a practical, accessible security monitoring solution ideal for academic institutions, small organizations and security research applications.

Introduction

InsightLog is a lightweight, explainable log analysis framework designed for Linux systems to improve security monitoring and incident detection. It addresses the limitations of manual log inspection, heavyweight SIEM tools, and black-box AI systems by offering an efficient, rule-based, and human-in-the-loop approach that works on standard hardware.

The system continuously monitors Linux authentication and system logs, parses them using regex into structured data, and applies rule-based anomaly detection with temporal analysis to identify threats such as brute-force attacks or unauthorized access. Related events are then grouped by an incident correlation engine to reduce noise and provide full context. A decision interface presents explainable summaries to operators, who approve or reject responses, ensuring transparency and control. All actions are logged for auditing and compliance.

Unlike complex enterprise solutions, InsightLog is designed to be lightweight (low CPU and memory usage), fast (real-time processing), and easy to deploy without cloud dependencies. It achieves high performance, including ~99% parsing accuracy, ~96% attack detection rate, low false positives, and significant reductions in alert fatigue and investigation time.

Conclusion

ThedevelopmentoftheInsightLogframeworksuccessfully establishes a lightweight, explainable and human-centric log analysis solution for Linux environments. Unlike traditional SIEM solutions that require enterprise-grade infrastructureand complex machine learning models, InsightLog provides effective security monitoring through a rule-based approach specifically designed for resource-constrained environments. The framework’s five-layer architecture—comprising Log Ingestion, Regex-based Parsing Engine, Rule-based Anomaly Detector,IncidentCorrelationEngineandHuman-in-the-Loop Decision Interface—proves that structured log processing can beachievedwithminimalcomputationaloverhead.Theregex- basedparsingengineconsistentlyachieves99.2%accuracy in extracting critical fields from unstructured logs, while processingupto5,200entriespersecondonstandardhardware with only 120MB RAM consumption. The incident correlation engine enhances operational ef- ficiency by reducing alert noise by 78% and decreasing investigationtimeby65%,provingthatcontextualaggregation is essential for practical security monitoring. The human-in- the-loop decision interface represents a critical advancementinmaintainingoperatoroversightwhilereducingfalsepositive fatigue, with 92% of test users rating evidence-linked recom- mendations as helpful. Operationally,InsightLogbridgesthegapbetweencomplex enterpriseSIEMsolutionsandbasiclogviewersbyprovidinga practical, accessiblealternative foracademicinstitutions,small organizationsandindividualsystemadministrators.Theframe- work’s ability to operate entirely offline with near-zero de- ployment cost makes it economically viable for environments where commercial security solutions are cost-prohibitive.

References

[1] P. Lohar and T. Baraskar, “Automated ai tool for log file analysis,” pp.1762–1766, 2025. [2] J. Pan, “Ai based log analyser: A practical approach,” arXiv preprintarXiv:2203.10960, 2022. [3] F. A. Bhuiyan and A. Rahman, “Log-related coding patterns to conductpostmortems of attacks in supervised learning-based projects,” ACMTransactions on Privacy and Security, vol. 26, no. 2, pp. 1–24, 2023. [4] T. K. Chan, I. F. B. Kamsin, S. Amin, and N. K. Zainal, “A completelog files security solution using anomaly detection and blockchaintechnology,” pp. 112–117, 2023. [5] J. Andrew, “Using natural language processing for log analysis andautomated alert prioritization,” 2025. [6] K. A. Garcia, R. Monroy, L. A. Trejo, C. Mex-Perera, and E. Aguirre,“Analyzing log files for postmortem intrusion detection,” IEEE Trans-actions on Systems, Man, and Cybernetics, Part C (Applications andReviews), vol. 42, no. 6, pp. 1690–1704, 2012. [7] Q.Fu,J.-G.Lou,Y.Wang,andJ.Li,“Executionanomalydetectionin distributed systems through unstructured log analysis,” in 2009 ninthIEEE international conference on data mining.IEEE, 2009, pp. 149–158. [8] O. Johnphill, A. S. Sadiq, O. Kaiwartya, and M. Aljaidi, “An intelligentapproach to automated operating systems log analysis for enhancedsecurity,” Information, vol. 15, no. 10, p. 657, 2024. [9] B.Debnath,M.Solaimani,M.A.G.Gulzar,N.Arora,C.Lumezanu, [10] J. Xu, B. Zong, H. Zhang, G. Jiang, and L. Khan, “Loglens: A real-timelog analysis system,” pp. 1052–1062, 2018. [11] J. P. Rouillard, “Real-time log file analysis using the simple eventcorrelator (sec).” vol. 4, pp. 133–150, 2004. [12] Z. Zhao, C. Xu, and B. Li, “A lstm-based anomaly detection model forlog analysis,” Journal of Signal Processing Systems, vol. 93, no. 7, pp.745–751, 2021. [13] S. Alspaugh, B. Chen, J. Lin, A. Ganapathi, M. Hearst, and R. Katz,“Analyzing log analysis: An empirical study of user log mining,” pp.62–77, 2014.

Copyright

Copyright © 2026 Hridhay Krishna H , Jaise Joy, Jishnudev P M, Abhiram M P, Ms. Jinsu Anna John. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.