AI Self-Healing DevOps: A Closed-Loop Framework for Anomaly Detection, Contextual Root-Cause Analysis and Policy-Governed Automated Recovery

Abstract

Modern cloud-native applications generate large volumes of telemetry, yet incident response still depends heavily on engineers who must notice a fault, diagnose it and apply a fix by hand. This paper presents AI Self-Healing DevOps, a working prototype that closes this loop through six stages: Monitor, Detect, Decide, Heal, Verify and Learn. Telemetry (metrics, logs, traces and events) flows through a topic-based event router into an anomaly detector that combines an Isolation Forest with a z-score guard. A Random Forest classifier labels the failure type, a dependency-graph and correlation analysis localises the root cause, and a retrieval step over a growing incident knowledge base supplies similar past cases for an explanation. A policy and risk engine then validates the remediation plan against an allowlist, playbooks, a flapping guard and a human-approval rule for high-risk actions, and dispatches it to Ansible, Argo CD and GitHub Actions executors. A verifier confirms recovery, retries, or escalates to an engineer, and the outcome updates the knowledge base and the approval policy. The prototype runs in a simulation mode and in a live mode against a real Express application, and is evaluated on six fault scenarios. We describe the design, state clearly which components are real and which are stand-ins, and discuss limitations and directions for production use.

Introduction

AI Self-Healing DevOps is an open-source prototype designed to detect, diagnose and automatically resolve software failures with minimal human intervention. Modern microservices and containerised applications are complex, making it difficult to identify the root cause of failures and reducing the effectiveness of manual troubleshooting. The proposed system addresses these challenges through an AI-driven, closed-loop framework that aims to reduce mean time to recovery (MTTR) while ensuring that automated actions remain transparent, controlled and verifiable.

The framework follows a six-stage self-healing cycle: Monitor, Detect, Decide, Heal, Verify and Learn. It collects metrics, logs, traces and events, routes them through a topic-based event system, analyses anomalies, selects suitable corrective actions and verifies whether the service has recovered. Information from resolved incidents is stored in a knowledge base to improve future diagnosis and update remediation policies.

The system combines several AI and automation techniques:

• Anomaly detection: An Isolation Forest identifies unusual system behaviour, supported by a z-score guard that flags metric values far outside the normal training range.

• Failure classification: A Random Forest classifies failures into categories such as service crashes, memory leaks, CPU overload, faulty deployments and dependency failures.

• Root-cause analysis: A dependency graph correlates affected services and prioritises upstream failures that may be responsible for downstream symptoms.

• Knowledge retrieval: Cosine-similarity matching retrieves similar historical incidents and their resolutions to support diagnosis and explanations.

• Policy and risk management: An allowlist, remediation playbooks, risk levels, human approval, repeated-action protection and historical success rates determine whether a proposed action can proceed.

• Verification and escalation: The system checks service health after remediation, retries unsuccessful actions and escalates unresolved incidents to a DevOps engineer.

The prototype is implemented in Python using scikit-learn, with a web dashboard for monitoring, incident management, architecture visualisation, AI model inspection, policy approvals, audit logs and fault injection. It supports a simulated environment containing four services across two nodes and a live mode targeting a Node.js Express application. Its architecture is designed to integrate with Prometheus, Kafka, Kubernetes, Ansible, Argo CD and GitHub Actions.

Six fault scenarios demonstrate the intended behaviour: service crashes, memory leaks, CPU overload, faulty deployments, database failures and failures that cannot be repaired automatically. Low-risk actions can proceed automatically, while high-risk actions require human approval. If recovery fails after repeated attempts, the incident is escalated rather than silently treated as resolved.

An important limitation is that the prototype does not yet demonstrate measured improvements in recovery time or diagnostic accuracy. The failure classifier uses synthetic training data, several infrastructure integrations operate in dry-run mode, and the default event router runs in-process rather than through a fault-tolerant distributed broker. In addition, the system updates its knowledge base and remediation policies but does not retrain its machine-learning models online. Quantitative MTTR comparisons are illustrative rather than based on controlled measurements.

Conclusion

We presented AI Self-Healing DevOps, a prototype that integrates anomaly detection, contextual root-cause analysis, policy-governed remediation, verification and learning into one observable loop. Its main design choices are a hybrid detector with a range guard, retrieval over a self-growing incident knowledge base, and a policy layer that keeps high-risk actions under human control. The prototype is explicit about its scope: the intelligence components are implemented and runnable, whereas infrastructure executors remain dry-run. Future work includes replacing the dry-run executors with real kubectl, Ansible and Argo CD calls on a managed cluster, training the classifier on labelled production incidents, retraining detectors on drift, evaluating retrieval and explanation quality, and measuring MTTR and MTTD on real workloads against a manual baseline.

References

[1] B. Beyer, C. Jones, J. Petoff and N. R. Murphy, Eds., Site Reliability Engineering: How Google Runs Production Systems. Sebastopol, CA, USA: O\'Reilly Media, 2016. [2] J. O. Kephart and D. M. Chess, “The vision of autonomic computing,” Computer, vol. 36, no. 1, pp. 41–50, Jan. 2003. [3] V. Chandola, A. Banerjee and V. Kumar, “Anomaly detection: A survey,” ACM Comput. Surv., vol. 41, no. 3, pp. 1–58, Jul. 2009. [4] F. T. Liu, K. M. Ting and Z.-H. Zhou, “Isolation forest,” in Proc. 8th IEEE Int. Conf. Data Mining (ICDM), Pisa, Italy, 2008, pp. 413–422. [5] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001. [6] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 9459–9474. [7] Argo Project, “Argo CD: Declarative GitOps continuous delivery for Kubernetes.” [Online]. Available: https://argo-cd.readthedocs.io [8] Red Hat, “Ansible documentation.” [Online]. Available: https://docs.ansible.com [9] M. Khadilkar, “AI-Self-Healing-DevOps,” GitHub repository. [Online]. Available: https://github.com/maheshkhadilkar07/AI-Self-Healing-DevOps [10] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011. [11] B. Burns, B. Grant, D. Oppenheimer, E. Brewer and J. Wilkes, “Borg, Omega, and Kubernetes,” ACM Queue, vol. 14, no. 1, pp. 70–93, 2016. [12] J. Kreps, N. Narkhede and J. Rao, “Kafka: A distributed messaging system for log processing,” in Proc. NetDB Workshop, 2011. [13] Prometheus Authors, “Prometheus: Monitoring system and time series database.” [Online]. Available: https://prometheus.io [14] B. H. Sigelman et al., “Dapper, a large-scale distributed systems tracing infrastructure,” Google, Mountain View, CA, USA, Tech. Rep., 2010.

Copyright

Copyright © 2026 Rajshri Pote, Shreya Prasad, Atharva Lutade, Swati Chaudhari, Krunal Vighre, Mahesh Khadilkar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.