Cloud-Based Security Analytics SIEM Using ELK Stack and Machine Learning

Abstract

The increasing frequency and sophistication of cyberattacks have created significant challenges for organizations in protecting their digital infrastructure. Traditional Security Information and Event Management (SIEM) systems primarily depend on predefined rules and signature-based detection techniques, which are often inadequate for identifying emerging or previously unseen attack patterns. To address these limitations, this paper presents a Cloud-Based Security Analytics SIEM Using ELK Stack and Machine Learning for intelligent cyberattack detection and real-time security monitoring. The proposed framework integrates Elasticsearch, Logstash, and Kibana (ELK Stack) with supervised machine learning algorithms to automate log collection, processing, threat classification, and security visualization. The CICIDS2017 dataset, containing approximately 2.5 million network traffic records, 70 selected features, and 15 attack categories, was used to train and evaluate the intrusion detection models. Random Forest, LightGBM, and XGBoost classifiers were developed and compared using multiple performance metrics. Experimental results indicate that the XGBoost model achieved the best overall performance with an accuracy of 99.86% and a Macro F1-score of 89.57%, demonstrating its effectiveness in multiclass cyberattack classification. The trained model was integrated with the ELK-based analytics platform to support automated attack prediction and centralized visualization through Kibana dashboards. The proposed system provides an efficient, scalable, and intelligent approach for enhancing cybersecurity monitoring by combining cloud-based log analytics with machine learning-driven threat detection. The framework can assist Security Operations Centers (SOCs) in improving incident detection, accelerating security analysis, and supporting proactive cyber defense.

Introduction

This paper proposes a Cloud-Based Security Analytics Security Information and Event Management (SIEM) framework that integrates the ELK Stack (Elasticsearch, Logstash, Kibana) with machine learning (ML) to improve real-time cyberattack detection, centralized log management, and security analytics. The framework addresses the limitations of traditional rule-based SIEM systems by incorporating intelligent attack classification using supervised learning algorithms.

Background

Organizations generate massive volumes of security logs from firewalls, servers, endpoints, cloud services, and network devices. While these logs contain valuable information for identifying cyber threats, manual analysis is impractical. Conventional SIEM systems rely on predefined rules and signatures, making them effective against known attacks but less capable of detecting:

• Zero-day attacks
• Advanced Persistent Threats (APTs)
• Previously unseen attack patterns

These systems also require continuous rule updates and often produce high false-positive rates.

Motivation

The rise of ransomware, DDoS attacks, phishing, and other sophisticated cyber threats has increased the demand for intelligent security monitoring systems. Machine learning can automatically learn attack patterns from historical data, reducing reliance on manually created detection rules. Combining ML with the ELK Stack enables:

• Automated threat detection
• Centralized log management
• Real-time visualization
• Faster incident response

Objectives

The study aims to:

1. Develop a cloud-based SIEM framework integrating the ELK Stack with supervised machine learning.
2. Preprocess the CICIDS2017 intrusion detection dataset.
3. Train and compare Random Forest, XGBoost, and LightGBM classifiers.
4. Integrate the best-performing model into the ELK analytics pipeline.
5. Provide real-time visualization through Kibana dashboards.

Problem Statement

Existing SIEM platforms effectively centralize logs but depend heavily on rule-based detection, limiting their ability to identify evolving cyber threats. Although many ML-based intrusion detection models achieve high accuracy, they are often evaluated only in offline environments without integration into operational SIEM platforms. This research addresses the need for a unified framework combining:

• Centralized log management
• Intelligent attack classification
• Real-time monitoring
• Interactive security analytics

Literature Review

The review highlights four key areas:

• Traditional SIEM Systems: Platforms such as IBM QRadar, Splunk, ArcSight, and Microsoft Sentinel provide centralized monitoring but rely primarily on signature-based detection.
• ELK Stack: Elasticsearch, Logstash, and Kibana offer scalable log collection, storage, search, and visualization but generally lack automated threat detection.
• Machine Learning for Intrusion Detection: Ensemble algorithms such as Random Forest, XGBoost, and LightGBM have demonstrated excellent performance on benchmark datasets but are rarely integrated into operational SIEM systems.
• Cloud-Based Security Analytics: Cloud-native monitoring improves scalability, yet many solutions depend on proprietary software or focus only on infrastructure monitoring rather than intelligent attack classification.

Proposed Framework

The proposed architecture consists of five major components:

1. Data Collection
   • Collects logs from servers, operating systems, network devices, and datasets.
2. Log Processing
   • Logstash ingests, parses, filters, and standardizes logs from multiple sources.
3. Data Storage
   • Elasticsearch stores indexed logs and supports fast searching and retrieval.
4. Machine Learning
   • Uses supervised learning to classify network traffic.
   • Models evaluated:
     • Random Forest
     • XGBoost
     • LightGBM
5. Visualization
   • Kibana dashboards display attack categories, event trends, and security analytics in real time.

Dataset and Preprocessing

The framework uses the CICIDS2017 benchmark dataset, containing:

• Approximately 2.5 million network flow records
• 70 numerical features
• 15 traffic classes (1 benign and 14 attack types)

Attack categories include:

• DDoS
• DoS Hulk
• DoS GoldenEye
• Slowloris
• PortScan
• Bot
• FTP and SSH attacks
• Heartbleed
• SQL Injection
• XSS
• Other web attacks

Preprocessing involved:

• Removing duplicate and incomplete records
• Replacing invalid values
• Eliminating constant-value features
• Feature normalization using StandardScaler
• Encoding categorical labels
• Splitting data into 80% training and 20% testing sets

ELK Stack Integration

The ELK Stack provides centralized security analytics:

• Logstash collects and preprocesses logs using Elastic Common Schema (ECS).
• Elasticsearch stores and indexes structured events for rapid querying.
• Kibana visualizes attack statistics, event timelines, and security dashboards.
• Docker is used for deployment, ensuring portability and scalability.

Machine Learning Model Development

Three ensemble learning models were trained:

• Random Forest
  • Uses bootstrap aggregation and balanced class weighting.
• XGBoost
  • Gradient boosting optimized for high-dimensional intrusion detection.
• LightGBM
  • Leaf-wise tree growth with Gradient-based One-Side Sampling (GOSS) for faster training.

Class imbalance was handled using model-specific balancing techniques.

Models were evaluated using:

• Accuracy
• Precision
• Recall
• Macro F1-score

Experimental Results

The framework was implemented using:

• Python 3.11
• Elasticsearch 8.11
• Logstash 8.11
• Kibana 8.11
• Docker Desktop
• Scikit-learn, XGBoost, and LightGBM libraries

Performance comparison:

Although LightGBM achieved the highest overall accuracy (99.87%), XGBoost achieved the highest Macro F1-score (89.57%), providing better balanced performance across multiple attack categories. Consequently, XGBoost was selected for integration into the SIEM framework.

Conclusion

This paper presented a Cloud-Based Security Analytics SIEM Using ELK Stack and Machine Learning for intelligent cyberattack detection and centralized security monitoring. The proposed framework integrates the ELK Stack with supervised machine learning techniques to automate log collection, processing, attack detection, and visualization. The CICIDS2017 dataset was used to train and evaluate Random Forest, XGBoost, and LightGBM models. Experimental results showed that XGBoost achieved the best overall performance with 99.86% accuracy and a Macro F1-score of 89.57%, making it the most suitable model for deployment in the proposed SIEM framework. The developed system successfully combines cloud-based log analytics with machine learning to provide centralized monitoring, automated threat detection, and real-time visualization through Kibana dashboards. The proposed framework offers a scalable and efficient solution for enhancing cybersecurity monitoring and supporting Security Operations Centers (SOCs). Future enhancements may include integrating live network traffic analysis, deep learning-based detection models, and automated incident response to further improve detection capabilities.

References

[1] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization,” Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP), Madeira, Portugal, pp. 108–116, 2018. [2] A. H. Lashkari, G. D. Gil, M. S. I. Mamun, and A. A. Ghorbani, “Characterization of Tor Traffic Using Time Based Features,” Proceedings of the 3rd International Conference on Information Systems Security and Privacy (ICISSP), Porto, Portugal, pp. 253–262, 2017. [3] T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, pp. 785–794, 2016. [4] G. Ke, Q. Meng, T. Finley et al., “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 3146–3154, 2017. [5] L. Breiman, “Random Forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001. [6] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, Omega, and Kubernetes,” Communications of the ACM, vol. 59, no. 5, pp. 50–57, May 2016. [7] Elastic NV, Elasticsearch: The Official Distributed Search and Analytics Engine, Available: https://www.elastic.co/elasticsearch [8] Elastic NV, Logstash: Data Collection and Processing Pipeline, Available: https://www.elastic.co/logstash [9] Elastic NV, Kibana: Data Visualization and Analytics Platform, Available: https://www.elastic.co/kibana [10] F. Chollet, Deep Learning with Python, 2nd ed., Manning Publications, 2021. [11] S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson Education, 2021. [12] W. Stallings, Network Security Essentials: Applications and Standards, 7th ed., Pearson, 2022. [13] D. E. Denning, “An Intrusion Detection Model,” IEEE Transactions on Software Engineering, vol. SE-13, no. 2, pp. 222–232, Feb. 1987. [14] NIST, Guide to Intrusion Detection and Prevention Systems (IDPS), NIST Special Publication 800-94, National Institute of Standards and Technology, 2007. [15] C. Krügel, F. Valeur, and G. Vigna, Intrusion Detection and Correlation: Challenges and Solutions, Springer, 2005

Copyright

Copyright © 2026 Gugulothu Sai Charan, Dr. P Sammulal. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.