When a large language model agent co-generates a bug reproduction test (BRT) and a fix for the same bug in one loop, the test can silently co-adapt to the fix: the in-trajectory fail-to-pass check passes even when both artifacts are wrong. The CoHarden framework hardens the test against LLM-synthesized mutation patches of the fix, but its own authors show the loop saturates after roughly three rounds and conjecture that test–fix coupling re-emerges because test and fix are still co-generated in lockstep. The open problem is to break this in-loop coupling — design a hardening protocol in which the generated test cannot simply tighten around what the current fix already passes, so that joint test–fix errors become visible to the loop — and to validate on SWE-bench-style bug-fix benchmarks that joint test–fix errors invisible to a lockstep loop become detectable.

Background

Automated program repair (APR) is the task of generating patches for software bugs automatically; LLM-based agents now attempt repository-scale repair starting from a natural-language bug report. Because a bug report underconstrains the intended behavior, APR systems increasingly generate a bug reproduction test (BRT) — an executable test that should fail on the buggy code and pass once the bug is fixed. The standard quality criterion for such a test is fail-to-pass (F→P): the test fails on the buggy version and passes on the golden (developer) fix. SWE-bench (and its curated subsets SWE-bench Verified and SWE-bench Lite) is the standard benchmark family: real GitHub issues in Python repositories, where a submitted patch is graded Resolved by replaying the repository's own hidden test suite.

A long-standing APR failure mode is patch overfitting: a generated patch passes the available tests yet is incorrect. The 2026 study "Beyond Fail-to-Pass" (Tan et al., arXiv:2607.19843) transfers this concern to the generated test itself. It introduces Mutation Patch Evaluation (MPE): synthesize plausible-but-incorrect mutant fixes (via four targeted operators — symptom suppression, incomplete fix, input-specific shortcut, behavior substitution) and classify a generated BRT by how it treats these mutants. A test is Rigorous if it fails the mutants (rejects plausible-but-incorrect fixes), Lax if it reproduces the symptom yet still admits some mutant (the Rigorous/Lax split is defined in the source, §2.1), and Misaligned if it rejects the golden fix, so it is not fail-to-pass at all. Only Rigorous tests reliably improve downstream repair. The paper further identifies test–fix error coupling: when the same agent loop co-generates the test and the fix, the in-trajectory F→P check can pass even when both are wrong, because the test was tuned to the very fix it validates.

To counter this, the paper proposes CoHarden, a two-phase co-generation loop: first generate a test before any fix (lax-init), then iteratively harden the test against a pool of surviving mutation patches while the fix is updated, tracking acceptance decisions across rounds in a Temporal Matrix (the previous-round test plays the role of the golden test). CoHarden reaches 69.4% Resolved and 78.9% F→P on SWE-bench Verified, but its Resolved rate plateaus after hardening round 3. The authors' stated explanation is that the loop still co-generates test and fix in lockstep, so the test tightens around what the current fix already passes while the fix updates around what the current test already accepts, leaving certain joint errors invisible to the Temporal Matrix; they name breaking this in-loop coupling as a direction for future work. Their limitations paragraph adds that the mutant pool inherits the underlying LLM's notion of plausible-but-incorrect fixes (the four operators are tuned to lax patterns observed in the study, so rare or domain-specific patterns may go undetected) and that experiments cover only Python bug-fix benchmarks and OpenHands-style agent loops (OpenHands is a widely used open-source agent framework for autonomous software engineering, serving here as the backbone that executes the repair agent).

Neighboring work does not discharge this. Agent-CoEvo (arXiv:2604.04580) co-evolves code patches and test patches as dynamic constraints but treats co-evolution as the search strategy itself, with no mechanism exposing joint errors; it appears as a baseline (58.2% Resolved) in the CoHarden paper. A Google study of dynamic cogeneration of BRTs (arXiv:2601.19066) compares cogeneration strategies and patch selectors for industrial deployment, without mutation-based hardening. On the overfitting side, a 2026 benchmarking study of patch-overfitting-detection techniques (arXiv:2603.11262) shows that post-hoc detection of overfit patches remains unreliable in practice, underscoring why an in-loop signal is wanted. Standalone BRT generation (e.g., the evolutionary generator EvoOtter, arXiv:2607.02854) improves test generation itself but does not touch the coupling between a generated test and a co-generated fix.

Problem Statement

Design and validate a hardening protocol for co-generated bug reproduction tests and fixes in LLM-based automated program repair that breaks the in-loop test–fix coupling identified by Tan et al. (arXiv:2607.19843, §6 Discussion and Limitations): the protocol must prevent the generated test from simply tightening around what the current fix already passes and the fix from simply updating around what the current test already accepts, so that joint errors — cases where the generated test and generated fix are both wrong yet the in-trajectory fail-to-pass check passes — become visible to the loop. The protocol should be evaluated on SWE-bench-style bug-fix benchmarks, the experimental setting of the source, measuring whether joint test–fix errors become visible — for example through resolved rate, fail-to-pass rate, lax acceptance, and an offline joint-error rate measured against golden artifacts as in the source's joint-failure analysis (§2.3) — with a lockstep co-generation hardening baseline as the comparison point; whether breaking the coupling also lifts the round-3 hardening saturation that the authors conjecture it causes is an empirical question the evaluation may probe rather than a required outcome. A related caveat from the source's Limitations, not part of this open question itself, is mutant-pool coverage: the four operator types (symptom suppression, incomplete fix, input-specific shortcut, behavior substitution) are tuned to commonly observed lax patterns, so rare or domain-specific plausible-but-incorrect fixes may go undetected; broadening the pool may be explored alongside the same objective but is not a condition on the decoupling protocol or on its completion. Generalization beyond Python bug-fix benchmarks and agentic loops (other languages, security patching, non-agentic LLM systems) is an explicitly untested extension in the source and may be addressed as part of the same objective but is not required for its core completion.

References

  1. Yuhao Tan, Zhibang Yang, Fangkai Yang, Yuan Yao, Yu Kang, Lu Wang, Pu Zhao, Xin Zhang, Xiaoxing Ma, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes. arXiv:2607.19843 (DOI: 10.48550/arxiv.2607.19843), 2026. https://arxiv.org/abs/2607.19843
  2. Kefan Li, Yuan Yuan, Mengfei Wang, Shihao Zheng, Wei Wang, Ping Yang, Mu Li, and Weifeng Lv. Beyond Fixed Tests: Repository-Level Issue Resolution as Coevolution of Code and Behavioral Constraints. arXiv:2604.04580, 2026. https://arxiv.org/abs/2604.04580
  3. Runxiang Cheng, Michele Tufano, José Cambronero, Renyao Wei, Sherry Shi, Grant Uy, Pat Rondon, and Franjo Ivančić. Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair. arXiv:2601.19066, 2026 (v3, August 2026). https://arxiv.org/abs/2601.19066
  4. Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel. EvoOtter: Evolutionary Reproduction Test Generator. arXiv:2607.02854, 2026. https://arxiv.org/abs/2607.02854
  5. David Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, and Federica Sarro. Unveiling Practical Shortcomings of Patch Overfitting Detection Techniques. arXiv:2603.11262, 2026. https://arxiv.org/abs/2603.11262