SWE-bench and its human-filtered subset SWE-bench Verified evaluate issue-solving agents built on large language models (LLMs) by checking generated patches against test suites, yet a patch can pass every benchmark test while failing to match what developers actually intended — and for most real-world issues the intended behavior is only partially specified by the issue text. An empirical study by Wang, Pradel, and Liu (2025) showed that 7.8% of plausible patches fail the full developer test suite, 29.6% behave differently from the developer-written oracle patch, and, crucially, 66.2% of sampled suspicious patches cannot be judged correct or incorrect at all because the issue statements under-specify the requirements. The open problem is a systematic, scalable methodology for establishing the intended behavior of ambiguous real-world issue statements and validating patch correctness against it, including a principled treatment of supplementary code changes and the integration of differentiating tests into evolving benchmark test suites.

Background

SWE-bench (Jimenez et al., 2023) is a benchmark for automated issue solving in which a system receives the natural-language issue report of a real GitHub repository and must produce a patch resolving it; a patch is scored as correct if it passes fail-to-pass tests (developer-written tests that failed before the fix and should pass after it) and pass-to-pass tests (regression tests that should keep passing). SWE-bench Verified is a 500-instance subset human-filtered by OpenAI annotators and is the de facto standard for reporting the capabilities of large-language-model (LLM) coding agents. A patch that passes these tests is called plausible; in the neighboring literature on test-based automated program repair (APR), plausible patches that do not implement the true fix are known as overfitting patches, and they have long been known to inflate reported repair rates.

Wang, Pradel, and Liu (2025) performed the most in-depth study of this phenomenon on SWE-bench Verified, analyzing plausible patches from three state-of-the-art issue-solving systems (CodeStory, LearnByInteract, and OpenHands). Their methodology centers on PatchDiff, a technique for differential patch testing: it automatically generates differentiating tests — test inputs on which the generated patch and the developer-written oracle patch behave differently — thereby exposing behavioral discrepancies that the benchmark's test suite misses. They find that the benchmark's validation ignores test files not modified in the original pull request (so 7.8% of patches count as resolved while failing the complete developer test suite), that 29.6% of plausible patches induce behavior different from the oracle patch (mostly through similar-but-divergent implementations at 46.8%, or through supplementary semantic changes — extra behavior beyond what the fix requires — at 27.3%, versus only 5.2% omitting necessary changes), and that manual inspection of a 30% sample of suspicious patches yields 28.6% certainly incorrect, 5.2% certainly correct, and 66.2% of uncertain correctness, because the issue statements do not specify the disputed implementation details. Combined, these weaknesses inflate reported resolution rates by 6.4 percentage points. Their study extends an earlier audit, SWE-bench+ (Aleithan et al., 2024), which had examined plausible patches from a single tool on the full, unfiltered SWE-bench.

The study's authors and subsequent work leave three linked issues unresolved. First, because issue statements are under-specified, there is no systematic way to determine what the developer-intended behavior actually is for ambiguous real-world tasks, which is exactly why the majority of suspicious patches remain of uncertain correctness. Second, while supplementary semantic changes correlate with incorrectness, no general decision procedure exists for telling a supplementary change that is a necessary part of the intended fix from one that is unnecessary or harmful — behavioral difference from the oracle patch is not itself evidence of incorrectness. Third, although differentiating tests and broader developer tests reveal hidden failures, it is unknown how to integrate them into evolving benchmarks in a way that scales, stays stable as repositories change, and avoids rejecting genuinely correct patches that legitimately differ from the oracle implementation.

Problem Statement

The problem, as left open by Wang, Pradel, and Liu (2025), is to establish a systematic and scalable methodology for determining the intended behavior of real-world, often under-specified issue statements and for validating the correctness of plausible patches against that behavior in SWE-bench-style benchmarks. The open problem therefore comprises three linked questions: (1) how, given an issue statement whose requirements leave implementation details ambiguous, to determine with evidence-grounded justification what the developer-intended behavior is, so that the class of patches currently of uncertain correctness (66.2% of sampled suspicious patches in the source study) receives an evidence-grounded verdict — correct, incorrect, or, only where no artifact or repository evidence pins down the developer's intent, fundamentally undecidable; (2) as an integral part of that adjudication, how to treat supplementary semantic changes in patches — extra behavior beyond what the fix requires — deciding whether such a change is a necessary component of the intended fix or an unnecessary or harmful addition against the determined intended behavior, recognizing that behavioral difference from the oracle patch is not by itself evidence of incorrectness; and (3) how to integrate the resulting validation signals, such as differentiating tests and full developer test suites, into benchmark test suites in a way that remains sustainable and robust as benchmarks and underlying repositories evolve and avoids falsely rejecting genuinely correct patches that legitimately differ from the oracle implementation. These three questions share one overarching objective — trustworthy patch-correctness validation for automated issue solving.

References

  1. You Wang, Michael Pradel, Zhongxin Liu. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study. arXiv:2503.15223. https://arxiv.org/abs/2503.15223
  2. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770. https://arxiv.org/abs/2310.06770
  3. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, Song Wang. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992. https://arxiv.org/abs/2410.06992
  4. Gustavo A. Oliva, Gopi Krishnan Rajbahadur, Aaditya Bhatia, Haoxiang Zhang, Yihao Chen, Zhilong Chen, Arthur Leung, Dayi Lin, Boyuan Chen, Ahmed E. Hassan. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation. arXiv:2507.09108. https://arxiv.org/abs/2507.09108
  5. Manyi Wang, Junjielong Xu, Pinjia He. PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks. arXiv:2607.28587. https://arxiv.org/abs/2607.28587
  6. Chenglin Li, Yisen Xu, Zehao Wang, Shin Hwei Tan, Tse-Hsun Chen. Probe to Generate: Program Variant-Guided Test Augmentation for Repository-Level Repair Benchmarks. arXiv:2604.01518. https://arxiv.org/abs/2604.01518
  7. Kefan Li, Yuan Yuan, Mengfei Wang, Shihao Zheng, Wei Wang, Ping Yang, Mu Li, Weifeng Lv. Beyond Fixed Tests: Repository-Level Issue Resolution as Coevolution of Code and Behavioral Constraints. arXiv:2604.04580. https://arxiv.org/abs/2604.04580
  8. Yuhao Tan, Zhibang Yang, Fangkai Yang, Yuan Yao, Yu Kang, Lu Wang, Pu Zhao, Xin Zhang, Xiaoxing Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang. Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes. arXiv:2607.19843. https://arxiv.org/abs/2607.19843
  9. Yangfan Wu, Haozhe Wang, Huanyu Yang, Jianmin Ji, Fangzhen Lin. SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories. arXiv:2608.09799. https://arxiv.org/abs/2608.09799
  10. Sherry Shi, Renyao Wei, Michele Tufano, José Cambronero, Runxiang Cheng, Franjo Ivančić, Pat Rondon. Towards a Human-in-the-Loop Framework for Reliable Patch Evaluation Using an LLM-as-a-Judge. arXiv:2511.10865. https://arxiv.org/abs/2511.10865
  11. KeFan Li, Mengfei Wang, Hengzhi Zhang, Zhichao Li, Yuan Yuan, Mu Li, Xiang Gao, Hailong Sun, Chunming Hu, Weifeng Lv. InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution. arXiv:2511.16004. https://arxiv.org/abs/2511.16004
  12. Yanuo Ma, Ben Kereopa-Yorke, Ben Schultz. Building to the Test: Coding Agents Deliver What You Check, Not What You Requested. arXiv:2606.28430. https://arxiv.org/abs/2606.28430
  13. Thanosan Prathifkumar, Noble Saji Mathews, Meiyappan Nagappan. Does SWE-Bench-Verified Test Agent Ability or Model Memory? arXiv:2512.10218. https://arxiv.org/abs/2512.10218
  14. Amirali Sajadi, Kostadin Damevski, Preetha Chatterjee. How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench. arXiv:2507.02976. https://arxiv.org/abs/2507.02976