DebugRepair: Enhancing LLM-Based Automated Program Repair via Self-Directed Debugging
A new LLM-based bug-fixing framework that watches code run at runtime fixes 59 more bugs than the next-best approach on a standard benchmark.
The Thesis
Most automated bug-fixing tools that use large language models (LLMs) only look at the error message after a test fails — the equivalent of reading a crash report without ever running a debugger. DebugRepair changes this by inserting diagnostic print statements into failing code, watching what the program actually does at runtime, and feeding that richer evidence back to the LLM for another repair attempt. On Defects4J — a widely used benchmark of real Java bugs — the system fixes 295 bugs with DeepSeek-V3, compared to 236 for the next-best published method. The catch is that this approach depends heavily on benchmark-style test suites, and real-world codebases are messier; the paper does not demonstrate deployment outside controlled research settings.
Catalyst
LLMs capable of multi-turn, context-aware code reasoning (GPT-3.5, DeepSeek-V3, and similar) have only recently become capable enough to act on intermediate runtime traces without losing the thread of a long debugging conversation. At the same time, the Defects4J and SWE-bench benchmark ecosystems now provide standardized, reproducible evaluation harnesses that let researchers compare these techniques fairly — a precondition for cumulative progress that didn't exist at this resolution two or three years ago.
What's New
Earlier feedback-based repair systems — such as ChatRepair and SRepair — fed the LLM only the test outcome: a stack trace or a pass/fail result. That is outcome-level evidence: it shows where the program crashed but not what state variables held along the way. DebugRepair instead simulates what a human developer does in a debugger: it inserts targeted logging statements at strategic points, captures intermediate variable values, and uses those observations to guide the next repair attempt in a structured conversation. The claimed advantage is fewer incorrect patches generated by the LLM reasoning from incomplete information.
The Counter
The 295-bug result on Defects4J sounds impressive, but Defects4J is a curated benchmark with clean, well-structured test suites — exactly the conditions where injecting debugging statements is easy and reliable. Real codebases have flaky tests, external service dependencies, stateful databases, and concurrency bugs that defeat deterministic instrumentation. The paper's 'rule-based fallback' for cases where instrumentation fails is vaguely described and its frequency of use is not disclosed. The 51.3% average improvement over 'vanilla settings' is also a low bar: vanilla means feeding raw stack traces to the LLM with no other scaffolding, which is a weak baseline that any structured prompting approach can beat. The authors do not compare against human developer repair time or cost, so it's unclear whether this system is useful in practice or just benchmark-optimal. Finally, the approach inherits all the hallucination risks of the underlying LLMs — a technically plausible but semantically wrong patch that passes tests is still a bug.
Longs
- MSFT — GitHub Copilot Autofix and Azure DevOps are the most direct commercial surface for this kind of automated repair
- PLTR — enterprise code maintenance and vulnerability remediation in government/defense software pipelines
- GTLB (GitLab) — integrated CI/CD pipelines where automated patch suggestion is a product differentiator
- IGM (iShares Expanded Tech-Software ETF) — broad software tooling exposure
Shorts
- Diffblue (automated Java unit test and repair tooling) — a system that fixes more bugs with less manual configuration threatens Diffblue's value proposition in enterprise Java shops
- Static-analysis-first APR vendors (e.g., older tools in the GenProg/SPR lineage) — runtime-evidence approaches empirically outperform purely static or outcome-only methods on the same benchmarks
Enablers (Picks & Shovels)
- DeepSeek-V3 and GPT-3.5/4 APIs — the backbone LLMs the system is built on and benchmarked against
- Defects4J — the Java bug benchmark that provides reproducible evaluation; without it, the claimed 295-bug result is unverifiable
- SWE-bench — the Python equivalent benchmark used for cross-language validation
- Python and Java instrumentation runtimes — the ability to inject and execute logging statements programmatically is a core enabler
Private Watchlist
- Cursor (private) — AI-native IDE that could integrate runtime-trace-based repair directly
- Poolside AI (private) — building LLMs specifically for code with execution feedback loops
- CodiumAI (private) — automated test generation and repair tooling for enterprise dev teams
- Qodo (private, formerly CodiumAI brand extension) — code integrity tooling
Resources
The Paper
Automated Program Repair (APR) has benefited from the code understanding and generation capabilities of Large Language Models (LLMs). Existing feedback-based APR methods iteratively refine candidate patches using test execution feedback and have shown promising results. However, most rely on outcome-level failure symptoms, such as stack traces, which show how failures are observed but fail to expose the intermediate runtime states critical for root-cause analysis. As a result, LLMs often infer bug causes without sufficient runtime evidence, leading to incorrect patches. To address this limitation, we propose DebugRepair, a self-directed debugging framework for LLM-based APR. DebugRepair enhances patch refinement with intermediate runtime evidence collected through simulated debugging. It consists of three components: test semantic purification, simulated instrumentation, and debugging-driven conversational repair. Together, they reduce noisy test context, collect runtime traces through targeted debugging statements with rule-based fallback, and progressively refine candidate patches using prior attempts and newly observed runtime states. We evaluate DebugRepair on three benchmarks across Java and Python. Experiments show that DebugRepair achieves state-of-the-art performance against 15 approaches. With GPT-3.5, it correctly fixes 224 bugs on Defects4J, outperforming prior SOTA LLM-based methods by 26.2%. With DeepSeek-V3, it correctly fixes 295 Defects4J bugs, surpassing the second-best baseline by 59 bugs. Across five additional backbone LLMs, DebugRepair improves repair performance by 51.3% over vanilla settings. Ablation studies further confirm the effectiveness of all components.