A Comparative Empirical Analysis of LLM-Assisted versus Traditional Software Debugging Methodologies: A Controlled Study of Fifty Mid-Level Software Engineers

Abstract

The integration of Large Language Models (LLMs) into the software development lifecycle has introduced a substantive shift in how software defects are diagnosed and remediated. Whereas traditional debugging methodologies rely upon deterministic state inspection and backward reasoning, contemporary AI-assisted workflows leverage statistical pattern matching to accelerate hypothesis generation. This paper reports the findings of a controlled empirical study in which fifty mid-level software engineers were stratified into two cohorts and tasked with resolving complex, multi-file architectural defects spanning C++ memory management, Python multi-hop logical errors, and Java concurrency faults. Group A (n = 25) was restricted to traditional instrumentation—IDE-integrated breakpoints, print statements, and static analysis tools—while Group B (n = 25) was granted access to LLM-assisted workflows including GitHub Copilot and a GPT-4-class chat interface. Mean Time to Resolution (MTTR), bug detection accuracy, false-positive remediation rates, and NASA-TLX cognitive-load scores were collected. The findings demonstrate that the LLM-assisted cohort achieved a 24.6% reduction in aggregate MTTR but exhibited a 7.8 percentage-point decrement in overall detection accuracy, a 2.4-fold elevation in false-positive remediation attempts, and a statistically significant reversal in C++ memory-management scenarios. While subjective cognitive load was substantially reduced, the results are consistent with the emerging “Comprehension Debt” hypothesis, suggesting that velocity gains are partially offset by the introduction of undetected architectural flaws.

Introduction

This study examines how Large Language Models (LLMs) affect software debugging within the Software Development Life Cycle (SDLC), where developers traditionally spend a large share of time (35–75%) identifying and fixing defects, costing the industry billions annually. The paper investigates whether tools like GPT-4, Copilot, and Claude improve debugging efficiency compared to conventional methods.

LLM-assisted development shifts debugging from manual, backward reasoning toward AI-supported diagnosis and repair. While prior research shows that these tools can improve coding speed (20–55%) and reduce Mean Time to Resolution (MTTR), they also introduce risks such as reduced code comprehension, “Comprehension Debt,” automation bias, and weaker performance on complex multi-step or systems-level bugs (e.g., memory leaks, race conditions).

The study addresses a research gap by experimentally comparing LLM-assisted vs traditional debugging using 50 mid-level engineers working on realistic multi-file defects in C++, Python, and Java. Participants were split into two groups: one using traditional debugging tools and the other using LLM support alongside standard tools. Performance was measured using MTTR, bug detection accuracy, false-positive fixes, and cognitive load (NASA-TLX).

Key findings show that LLM assistance reduced overall debugging time by about 24.6%, particularly improving performance on Python logic and Java concurrency bugs. However, it performed worse on C++ memory-management defects, where traditional debugging was more effective. The results suggest that LLMs are helpful for certain categories of bugs but struggle with deeply system-level, multi-layered problems.

Conclusion

This paper has presented the findings of a controlled empirical study of fifty mid-level software engineers comparing traditional and LLM-assisted debugging methodologies across complex, multi-file architectural defects. The LLM-assisted cohort achieved a 24.6% reduction in aggregate Mean Time to Resolution and a substantial reduction in reported cognitive load; however, the same cohort simultaneously exhibited reduced detection accuracy, a 2.4-fold elevation in false-positive remediation attempts, and a statistically significant performance reversal within C++ memory-management scenarios. The empirical dominance of the Iterative AI Debugging interaction pattern within Group B, coupled with behavioral evidence of Automation Bias, supports the emerging thesis that velocity gains from generative AI may be accompanied by the accrual of latent Comprehension Debt. Future work will extend this investigation along three principal axes. First, a longitudinal extension is planned in which the architectural comprehension of participants will be reassessed three and six months after the conclusion of the study, thereby directly operationalizing the Comprehension Debt construct. Second, an intervention study will evaluate whether targeted prompt-engineering training—specifically oriented toward the Generation-then-Comprehension pattern [19]—can attenuate the detection-accuracy deficit observed herein. Third, a follow-up investigation will integrate RAG-augmented LLM configurations [10] to determine whether engineered contextual scaffolding is sufficient to eliminate the observed reversal in C++ memory debugging performance. Collectively, these lines of inquiry are intended to inform the design of agentic SDLC tooling that preserves cognitive ownership while delivering genuine engineering productivity.

References

[1] N. Cardozo and K. Dam, “The Debugging Mindset,” ACM Queue, vol. 15, no. 1, 2017. [2] Coralogix, “This is what your developers are doing 75% of the time, and the cost you pay,” Coralogix Engineering Blog, 2023. [3] J. Tie, B. Yao et al., “\'Should I Give Up Now?’ Investigating LLM Pitfalls in Software Engineering,” arXiv:2411.09916, 2024. [4] A. Smith et al., “The Impact of LLM Assistants on Software Developer Productivity: A Systematic Review and Mapping Study,” arXiv:2507.03156, 2025. [5] Y. Zhou, S. Saghi et al., “Cognitive Biases in LLM-Assisted Software Development,” in Proc. 47th IEEE/ACM Int. Conf. on Software Engineering (ICSE), 2026. [6] Y. Isobe, “Measuring Developer Productivity in the LLM Era,” Industry article, 2024. [7] J. Hamade, “True Cost of AI-Generated Code: A Strategic Analysis of Comprehension Debt,” Industry white paper, 2025. [8] S. Yang et al., “Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors (DSDBench),” in Proc. EMNLP, 2025, pp. 21348–21367. [9] Microsoft Corp., “An AI-led SDLC: Building an End-to-End Agentic Software Development Lifecycle with Azure and GitHub,” Microsoft Tech Community, 2025. [10] A. Karlsson, “Task-Adapting LLMs for Software Reliability,” M.Sc. thesis, KTH Royal Institute of Technology, Stockholm, Sweden, 2026. [11] AlgoCademy Editorial, “Why Debugging Takes Longer Than Writing the Actual Code,” AlgoCademy Engineering Blog, 2024. [12] X. Liu et al., “Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs,” SMU InK Research Collection, Singapore Management Univ., 2025. [13] L. Yang et al., “Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback,” arXiv preprint, 2026. [14] Speedscale, “Essential KPIs for Software Development: Measure Success Effectively,” Speedscale Engineering Blog, 2024. [15] Virtuoso QA, “Software Testing Metrics—Types, Formulae, and Calculation,” Virtuoso QA Knowledge Base, 2024. [16] Axify, “Software Development KPIs: 32 Metrics to Track,” Industry guide, 2026. [17] Integrated Research, “How to Reduce MTTR with AI: A Guide for Enterprise IT Teams,” IR White Paper, 2026. [18] K. Park and M. Chen, “The Influence of Artificial Intelligence Tools on Learning Outcomes in Computer Programming: A Systematic Review and Meta-Analysis,” Computers, vol. 14, no. 5, 2025. [19] Anthropic Research, “How AI Assistance Impacts the Formation of Coding Skills,” Anthropic Technical Report, 2026. [20] R. Patel et al., “Can LLMs Find Bugs in Code? An Evaluation from Beginner Errors to Security Vulnerabilities in Python and C++,” arXiv:2508.16419, 2025. [21] S. Yang et al., “Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers,” ACL Anthology, EMNLP Main Track, 2025, pp. 21348–21367. [22] C. Granger, D. Khati et al., “Tricky²: Towards a Benchmark for Evaluating Human and LLM Error Interactions,” arXiv preprint, 2026. [23] Y. Ding et al., “Executing as You Generate: Hiding Execution Latency in LLM Code Generation (EG-CFG),” in Proc. NeurIPS Workshop on LLMs for Code, 2024. [24] J. Sweller, “Cognitive Load During Problem Solving: Effects on Learning,” Cognitive Science, vol. 12, no. 2, pp. 257–285, 1988. [25] METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” METR Technical Report, 2025. [26] Google Cloud / DORA, “State of AI-Assisted Software Development,” Annual DORA Report, 2025.

Copyright

Copyright © 2026 Anup Brahmne, Moksh Prajapati. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.