A Review on Deep Learning for Anomaly and Malicious Traffic Detection in Cloud Environments

Abstract

This review paper explores the development of a deep learning-based framework aimed at improving the security of cloud computing environments by detecting anomalous and malicious traffic. The paper emphasises the requirement of real-time detection in view of the growing issues presented by sophisticated threats including distributed denial of service (DDoS) assaults, botnet traffic, and data exfiltration. Combining recurrent neural networks (RNNs) with convolutional neural networks (CNNs), models and detects traffic anomalies rather effectively. Here we derive important characteristics obtained from a large cloud traffic dataset—such as packet size, source IP, and traffic patterns—which the model subsequently employs. The study contrasts the performance of the deep learning model with traditional machine learning techniques such decision trees and support vector machines (SVMs) using evaluation metrics including accuracy, precision, recall, F1-score, and area under the curve (AUC). The deep learning-based model reveals to be superior to more conventional techniques with enhanced accuracy and recall. It is versatile enough to match shifting attack patterns with minimal training and quite sensitive to known and novel anomalies. Moreover, our method excels in identifying sometimes ignored subtle and nuanced negative behaviours by conventional models. At last, the findings suggest that deep learning offers a scalable, adaptable, and effective solution to enhance negative traffic identification in cloud systems, therefore providing a strong means of managing evolving security challenges.

Introduction

Cloud computing offers scalable, flexible, and cost-effective data storage and processing but faces significant security challenges such as illegal access, data breaches, and DDoS attacks. The dynamic, multi-tenant nature of cloud systems complicates security, making traditional detection methods less effective, especially against hidden aberrant and hostile traffic.

Deep learning has emerged as a promising solution for detecting complex and subtle patterns in cloud network traffic. Models like autoencoders, CNNs, and RNNs excel at identifying anomalies and malicious behavior by learning directly from raw data, outperforming traditional rule-based or statistical methods. However, challenges remain, including high computational costs for real-time analysis and the “black-box” nature of deep learning models, which affects interpretability and trust.

Ethical and privacy concerns, especially compliance with regulations like GDPR, are addressed by integrating privacy-preserving techniques such as federated learning and data anonymization into deep learning systems.

The document also reviews related research showing the application of deep learning and machine learning for network intrusion detection, compliance automation, and anomaly detection in cloud environments.

Common types of network threats in cloud environments include DDoS, phishing, malware propagation, Man-in-the-Middle (MitM) attacks, data exfiltration, and SQL injection. Deep learning techniques, especially autoencoders and CNNs, are effective in recognizing complex traffic patterns, extracting relevant features automatically, and detecting anomalies or malicious traffic in real time.

Overall, deep learning-based approaches are crucial for evolving cloud security, enabling improved detection accuracy, scalability, and adaptability against increasingly sophisticated cyberattacks.

Conclusion

In conclusion, Deep learning for anomaly and malicious traffic identification is a significant advance in cloud computing system security. Large volumes of dynamic and changing traffic are part of conventional methods, which sometimes find it challenging to meet the scale and complexity of modern cloud setups. Deep learning models including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and auto encoders have shown incredible capacity in spotting complex patterns of destructive activity including data exfiltration, botnet activity, and distributed denial of service (DDoS) attack[39], [40]. These methods learn and adapt from large datasets without depending on hand-made feature extraction, hence they are really good in spotting heretofore unknown hazards. By learning from fresh data, deep learning methods may adapt with the cloud environment and remain current with developing cyberthreats. This is especially crucial considering the dynamic and often changing character of cloud-based systems, where attack strategies are always changing[41]. Though problems including computational costs, model interpretability, and data privacy still remain, deep learning offers tremendous benefits for cloud security. But explainable artificial intelligence, distributed learning, and hardware acceleration are gradually fixing these issues. Including privacy-preserving methods like federated learning moving forward will assist to increase the scalability and compliance of deep learning-based detection systems even further[42]–[44]. Deep learning offers a robust and adaptable defence mechanism to protect private information and to preserve the integrity of cloud-based systems against ever more advanced cyber threats. It presents a promising future for enhancing anomaly and malicious traffic detection in cloud environments.

References

[1] A. D. Vibhute and V. Nakum, “Deep learning-based network anomaly detection and classification in an imbalanced cloud environment,” Procedia Comput. Sci., vol. 232, no. 2023, pp. 1636–1645, 2024, doi: 10.1016/j.procs.2024.01.161. [2] S. E. H. Hassan and N. Duong-Trung, “Machine Learning in Cybersecurity: Advanced Detection and Classification Techniques for Network Traffic Environments,” EAI Endorsed Trans. Ind. Networks Intell. Syst., vol. 11, no. 3, pp. 1–22, 2024, doi: 10.4108/eetinis.v11i3.5237. [3] A. Abdullah and M. A. Bouke, “Towards Image-Based Network Traffic Pattern Detection for DDoS Attacks in Cloud Computing Environments: A Comparative Study,” Int. Conf. Cloud Comput. Serv. Sci. CLOSER - Proc., no. Closer, pp. 287–294, 2024, doi: 10.5220/0012725600003711. [4] P. Thapa and T. Arjunan, “AI-Enhanced Cybersecurity: Machine Learning for Anomaly Detection in Cloud Computing,” Q. J. Emerg. Technol. Innov., vol. 9, no. 1, pp. 25–37, 2024, [Online]. Available: https://vectoral.org/index.php/QJETI/article/view/64 [5] W. H. Aljuaid and S. S. Alshamrani, “A Deep Learning Approach for Intrusion Detection Systems in Cloud Computing Environments,” Appl. Sci., vol. 14, no. 13, 2024, doi: 10.3390/app14135381. [6] “Anomaly and malicious Traffic Detection in Cloud Computing - - Image Search results.” https://in.images.search.yahoo.com/yhs/search;_ylt=Awr1WSy8sW9nYGEDKFjnHgx.;_ylu=Y29sbwMEcG9zAzEEdnRpZAMEc2VjA3BpdnM-?p=Anomaly+and+malicious+Traffic+Detection+in+Cloud+Computing&vm=r&type=fc_AC934C13286_s58_g_e_d022424_n9998_c999¶m1=7¶m2=eJwtj8lugzAQhl%2FFx0QKMB4bb9wS6ANUPTXKwRCHWKwCKqo%2Bfe20msv3LyPNtP5%2BLW7vJQVABeJ6uo1Ba61VwBgBIkceRPPnB%2FJzQORADdcARilkJlcNM5hTaqytqaklPAyTioV666bQ92PALxtomH5839ssT4Ecdj%2Fep30l40YopFCQYAhekG%2FBj8TOc%2B92V3d%2By3ImUybIoXtuQ38ive8caV3TTUfSPJdpcBllNIU4ZLUPu%2Fj%2FlXjw%2BvoyHrC65cUXgZKdS0hKASyhtKqSM1eR6NtFhFTrKvabWEZAngAmqD5AGCZMrlOU8vMXl3FZkw%3D%3D&hsimp=yhs-2461&hspart=fc&ei=UTF-8&fr=yhs-fc-2461#id=4&iurl=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F340704062%2Ffigure%2Ffig4%2FAS%3A1095915934887936%401638298128182%2FAnomaly-detection-process-of-cloud-computing-network.png&action=click (accessed Dec. 28, 2024). [7] D. Sakthivel and B. Radha, “Network Traffic Analysis of Anomaly Detected Attacks Using Random Forest Algorithm in Cloud Environment,” Nat. Camp., vol. 28, no. 1, pp. 1–11, 2024, [Online]. Available: https://museonaturalistico.it [8] Y. Lin, “Enhanced Detection of Anomalous Network Behavior in Cloud - Driven Big Data Systems Using Deep Learning Models,” vol. 4, no. 8, pp. 1–11, 2024. [9] Mahesh Kumar Bagwani, Anshu Gangwar, Karuna Vishwakarma, and Virendra Kumar Tiwari, “Real-time signature-based detection and prevention of DDOS attacks in cloud environments,” Int. J. Sci. Res. Arch., vol. 12, no. 2, pp. 2929–2935, 2024, doi: 10.30574/ijsra.2024.12.2.1608. [10] F. Zhao, H. Li, K. Niu, J. Shi, and R. Song, “Application of deep learning-based Intrusion Detection System (IDS) in network anomaly traffic detection,” Appl. Comput. Eng., vol. 86, no. 1, pp. 250–256, 2024, doi: 10.54254/2755-2721/86/20241604. [11] S. P. -, J. N. A. M. -, K. T. -, and M. D. -, “Achieving Regulatory Compliance in Cloud Computing through ML,” Adv. Int. J. Multidiscip. Res., vol. 2, no. 2, pp. 1–15, 2024, doi: 10.62127/aijmr.2024.v02i02.1038. [12] W. Elbakri, M. M. Siraj, B. A. S. Al-Rimy, S. N. Qasem, and T. Al-Hadhrami, “Adaptive Cloud Intrusion Detection System Based on Pruned Exact Linear Time Technique,” Comput. Mater. Contin., vol. 79, no. 3, pp. 3725–3756, 2024, doi: 10.32604/cmc.2024.048105. [13] T. Arjunan, “Real-Time Detection of Network Traffic Anomalies in Big Data Environments Using Deep Learning Models,” Int. J. Res. Appl. Sci. Eng. Technol., vol. 12, no. 3, pp. 844–850, 2024, doi: 10.22214/ijraset.2024.58946. [14] K. Mitropoulou, P. Kokkinos, P. Soumplis, and E. Varvarigos, “Anomaly Detection in Cloud Computing using Knowledge Graph Embedding and Machine Learning Mechanisms,” J. Grid Comput., vol. 22, no. 1, 2024, doi: 10.1007/s10723-023-09727-1. [15] A. Thillaivanan, S. R. Wategaonkar, S. Duraisamy, R. Mishra, S. Nagaraj, and K. Singh, “Automated Denial of Service Detection Using Moth Flame Optimization With Machine Learning in Cloud Environment,” 2023 2nd Int. Conf. Smart Technol. Syst. Next Gener. Comput. ICSTSN 2023, no. June, pp. 1–6, 2023, doi: 10.1109/ICSTSN57873.2023.10151478. [16] S. Saleh, B. Cherradi, O. El Gannour, N. Gouiza, and O. Bouattane, “Healthcare monitoring system for automatic database management using mobile application in IoT environment,” Bull. Electr. Eng. Informatics, vol. 12, no. 2, pp. 1055–1068, 2023, doi: 10.11591/eei.v12i2.4282. [17] P. Parameswarappa, T. Shah, and G. R. Lanke, “A Machine Learning-Based Approach for Anomaly Detection for Secure Cloud Computing Environments,” IDCIoT 2023 - Int. Conf. Intell. Data Commun. Technol. Internet Things, Proc., no. January, pp. 931–940, 2023, doi: 10.1109/IDCIoT56793.2023.10053518. [18] K. Wang, Y. Fu, X. Duan, T. Liu, and J. Xu, “Abnormal traffic detection system in SDN based on deep learning hybrid models,” Comput. Commun., vol. 216, pp. 183–194, 2024, doi: 10.1016/j.comcom.2023.12.041. [19] P. Zhong, Y. Liu, H. Zheng, and J. Zhao, “Detection of Urban Flood Inundation from Traffic Images Using Deep Learning Methods,” Water Resour. Manag., vol. 38, no. 1, pp. 287–301, 2024, doi: 10.1007/s11269-023-03669-9. [20] E. Batalov et al., “Ransomware Detection via Network Traffic Analysis Using Isolation Forest and LSTM Neural Networks Ransomware Detection via Network Traffic Analysis Using Isolation Forest and LSTM Neural Networks,” 2024. [21] M. Akibis et al., “Measuring Ransomware Propagation Patterns via Network Tra c Analysis?: An Automated Approach,” 2024. [22] F. Rustam and A. D. Jurcut, “Malicious traffic detection in multi-environment networks using novel S-DATE and PSO-D-SEM approaches,” Comput. Secur., vol. 136, no. August 2023, p. 103564, 2024, doi: 10.1016/j.cose.2023.103564. [23] F. Alzonem et al., “Ransomware Detection Using Convolutional Neural Networks and Isolation Forests in Network Tra c Patterns Network Tra ffi c Patterns,” 2024. [24] U. B. Clinton, N. Hoque, and K. Robindro Singh, “Classification of DDoS attack traffic on SDN network environment using deep learning,” Cybersecurity, vol. 7, no. 1, 2024, doi: 10.1186/s42400-024-00219-7. [25] Y. A. Abid, J. Wu, G. Xu, S. Fu, and M. Waqas, “Multilevel Deep Neural Network Approach for Enhanced Distributed Denial-of-Service Attack Detection and Classification in Software-Defined Internet of Things Networks,” IEEE Internet Things J., vol. 11, no. 14, pp. 24715–24725, 2024, doi: 10.1109/JIOT.2024.3376578. [26] I. Naseer, “The role of artificial intelligence in detecting and preventing cyber and phishing attacks The role of artificial intelligence in detecting and preventing cyber and phishing attacks,” no. October, 2024. [27] A. M. Sayed Ahmed, H. M. Ahmed, T. A. Nofal, A. Darwish, and O. A. M. Omar, “Hilfer-Katugampola fractional epidemic model for malware propagation with optimal control,” Ain Shams Eng. J., vol. 15, no. 10, p. 102945, 2024, doi: 10.1016/j.asej.2024.102945. [28] “Malware Propagation - - Image Search results.” https://in.images.search.yahoo.com/yhs/search;_ylt=Awr1WSwAsW9nWtEBSwTnHgx.;_ylu=Y29sbwMEcG9zAzEEdnRpZAMEc2VjA3BpdnM-?p=Malware+Propagation&vm=r&type=fc_AC934C13286_s58_g_e_d022424_n9998_c999¶m1=7¶m2=eJwtj8lugzAQhl%2FFx0QKMB4bb9wS6ANUPTXKwRCHWKwCKqo%2Bfe20msv3LyPNtP5%2BLW7vJQVABeJ6uo1Ba61VwBgBIkceRPPnB%2FJzQORADdcARilkJlcNM5hTaqytqaklPAyTioV666bQ92PALxtomH5839ssT4Ecdj%2Fep30l40YopFCQYAhekG%2FBj8TOc%2B92V3d%2By3ImUybIoXtuQ38ive8caV3TTUfSPJdpcBllNIU4ZLUPu%2Fj%2FlXjw%2BvoyHrC65cUXgZKdS0hKASyhtKqSM1eR6NtFhFTrKvabWEZAngAmqD5AGCZMrlOU8vMXl3FZkw%3D%3D&hsimp=yhs-2461&hspart=fc&ei=UTF-8&fr=yhs-fc-2461#id=0&iurl=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FKaiming-Xiao%2Fpublication%2F343693498%2Ffigure%2Ffig2%2FAS%3A926251116150784%401597846883066%2FAn-example-of-malware-propagation-in-cyber-physical-systems-CPS.ppm&action=click (accessed Dec. 28, 2024). [29] M. Thankappan, H. Rifà-Pous, and C. Garrigues, “A distributed and cooperative signature-based intrusion detection system framework for multi-channel man-in-the-middle attacks against protected Wi-Fi networks,” Int. J. Inf. Secur., vol. 12, no. February, 2024, doi: 10.1007/s10207-024-00899-9. [30] S. E. Prasetyo, H. Haeruddin, and K. Ariesryo, “Website Security System from Denial of Service attacks, SQL Injection, Cross Site Scripting using Web Application Firewall,” Antivirus J. Ilm. Tek. Inform., vol. 18, no. 1, pp. 27–36, 2024, doi: 10.35457/antivirus.v18i1.3339. [31] F. Zhao, M. Zhang, S. Zhou, and Q. Lou, “Detection of Network Security Traffic Anomalies Based on Machine Learning KNN Method,” J. Artif. Intell. Gen. Sci. ISSN3006-4023, vol. 1, no. 1, pp. 209–218, 2024, doi: 10.60087/jaigs.v1i1.213. [32] G. Almahadin et al., “VANET Network Traffic Anomaly Detection Using GRU-Based Deep Learning Model,” IEEE Trans. Consum. Electron., vol. 70, no. 1, pp. 4548–4555, 2024, doi: 10.1109/TCE.2023.3326384. [33] T. Ali and P. Kostakos, “HuntGPT: Integrating Machine Learning-Based Anomaly Detection and Explainable AI with Large Language Models (LLMs),” 2023, [Online]. Available: http://arxiv.org/abs/2309.16021 [34] M. Vishwakarma and N. Kesswani, “A new two-phase intrusion detection system with Naïve Bayes machine learning for data classification and elliptic envelop method for anomaly detection,” Decis. Anal. J., vol. 7, no. January, p. 100233, 2023, doi: 10.1016/j.dajour.2023.100233. [35] H. Torabi, S. L. Mirtaheri, and S. Greco, “Practical autoencoder based anomaly detection by using vector reconstruction error,” Cybersecurity, vol. 6, no. 1, pp. 1–13, 2023, doi: 10.1186/s42400-022-00134-9. [36] H. Liu and H. Wang, “Real-Time Anomaly Detection of Network Traffic Based on CNN,” Symmetry (Basel)., vol. 15, no. 6, 2023, doi: 10.3390/sym15061205. [37] Yan Lei, “Smart Network Forensics with Generative Adversarial Networks Leveraging Blockchain for Anomaly Detection and Immutable Audit Trails,” Power Syst. Technol., vol. 48, no. 1, pp. 1625–1642, 2024, doi: 10.52783/pst.432. [38] R. Ghiasi, M. A. Khan, D. Sorrentino, C. Diaine, and A. Malekjafarian, “An unsupervised anomaly detection framework for onboard monitoring of railway track geometrical defects using one-class support vector machine,” Eng. Appl. Artif. Intell., vol. 133, no. PB, p. 108167, 2024, doi: 10.1016/j.engappai.2024.108167. [39] S. Chakraborty, S. K. Pandey, S. Maity, and L. Dey, “Detection and Classification of Novel Attacks and Anomaly in IoT Network using Rule based Deep Learning Model,” vol. 2, pp. 1–11, 2022. [40] A. Alshammari and A. Aldribi, “Apply machine learning techniques to detect malicious network traffic in cloud computing,” J. Big Data, vol. 8, no. 1, pp. 1–24, 2021, doi: 10.1186/s40537-021-00475-1. [41] R. H. Hwang, M. C. Peng, C. W. Huang, P. C. Lin, and V. L. Nguyen, “An Unsupervised Deep Learning Model for Early Network Traffic Anomaly Detection,” IEEE Access, vol. 8, pp. 30387–30399, 2020, doi: 10.1109/ACCESS.2020.2973023. [42] S. Yaqoob, A. Hussain, F. Subhan, G. Pappalardo, and M. Awais, “Deep Learning Based Anomaly Detection for Fog-Assisted IoVs Network,” IEEE Access, vol. 11, no. January, pp. 19024–19038, 2023, doi: 10.1109/ACCESS.2023.3246660. [43] J. P. Singh, “Mitigating Challenges in Cloud Anomaly Detection Using an Integrated Deep Neural Network-SVM Classifier Model,” vol. 5, no. 1, pp. 39–49, 2022. [44] S. I. Imtiaz et al., “Efficient Approach for Anomaly Detection in Internet of Things Traffic Using Deep Learning,” Wirel. Commun. Mob. Comput., vol. 2022, 2022, doi: 10.1155/2022/8266347.

Copyright

Copyright © 2025 Tilak Sharma, Unmukh Datta. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.