Evaluations of large language model (LLM) software-engineering agents on issue-resolving benchmarks such as SWE-bench Verified may be inflated by memorization: the public GitHub issues and gold patches underlying the benchmark were likely present in models' training data. Diagnostic evidence (file-path identification from issue text alone and verbatim function reproduction) shows state-of-the-art models perform far better on SWE-bench Verified instances than on comparable but non-benchmark tasks, indicating training–evaluation overlap. The open problem is to make such evaluation reliable: establish validated construction principles and standardized temporal-freshness protocols for issue-resolving benchmarks that prevent training–evaluation overlap, develop black-box contamination-detection methods that work without access to training data or model internals, and quantify the prevalence and magnitude of benchmark memorization across model families, languages, and repository ecosystems. Numerous ad hoc remedies exist (live-updatable pipelines, benchmark mutation, from-scratch fresh tasks, session-diversity detection), but no accepted standard or validated detection methodology has emerged.

Background

An LLM-based software-engineering agent takes a natural-language bug report (a GitHub issue) for a real repository and must produce a patch that fixes it; the dominant evaluation suite is SWE-bench, whose instances are real issues from popular open-source Python repositories, each graded by the repository's own tests, and whose human-validated 500-instance subset SWE-bench Verified has become the de facto leaderboard for coding agents. Because SWE-bench instances are mined from public GitHub data that predates the training cutoffs of frontier models, the benchmark is exposed to data contamination — the presence of evaluation instances (issues, discussions, or gold patches) in a model's training corpus — so a high score can reflect memorization (recall of the specific issue and its merged fix) rather than genuine problem-solving ability. Liang, Garg, and Moghaddam (2025, 'The SWE-Bench Illusion', arXiv:2506.12286) probed this with two diagnostic tasks: file-path identification, where models identify the buggy file from the issue description alone (up to 76% top-1 accuracy on SWE-bench Verified versus up to 53% on comparable tasks from repositories not in SWE-bench, a gap that is hard to explain without prior exposure), and function reproduction, where models regenerate the fixed function from the current buggy file plus the issue, scored by consecutive 5-gram overlap with the gold patch (materially higher on SWE-bench Verified and Full than on external benchmarks). They observed a graduated decay of memorization signals from the most-curated subset to less-curated ones and a repository bias (large performance drops on popular repositories absent from the benchmark), concluding that the field urgently needs evaluation frameworks with temporal controls (freshness of evaluation data relative to model training cutoffs), cross-repository validation, contamination-resistant benchmark construction, and systematic cross-benchmark analysis. Prior mitigation strategies have two main branches. Task mutation alters existing benchmark instances (e.g., semantic or syntax-level transformations of problems) to expose overfitting to specific formulations; metric-based probing analyzes model outputs or statistics — perplexity, negative log-likelihood, n-gram overlap, or membership-inference methods such as Min-K% Prob, which flags a text as likely-seen if its low-probability tokens are too rare — but most of these require model log-probabilities or internals that commercial API-only models do not expose, and n-gram overlap alone is noisy because any functionally correct patch shares text with the gold patch. A separate line builds temporally fresh benchmarks, following LiveCodeBench's template of continuously collecting new problems and evaluating models only on items published after their training cutoffs. What no existing work provides is the combination the source paper calls for: accepted construction principles and freshness protocols (cutoff conventions, update cadences, deduplication) for issue-resolving benchmarks, detection methods that operate black-box on commercial models, and a systematic quantification of memorization across the model ecosystem.

Problem Statement

Develop and validate a contamination-resistant evaluation framework for LLM-based issue-resolving (software-engineering) agents, so that reported scores measure transferable problem-solving rather than memorization of public issue–patch data. The problem, as posed by the concluding call of Liang et al. (arXiv:2506.12286) together with its acknowledged limitations, concerns three linked aspects of this single objective: (1) construction principles and (semi-)automated pipelines for issue-resolving benchmark suites that prevent training–evaluation overlap — including temporal-freshness gating of instances against declared model training cutoffs, automated deduplication against plausible training corpora, and cross-repository and multilingual coverage; (2) a black-box contamination-detection methodology applicable to commercial API-only models without access to training data or model internals; and (3) application of such benchmarks and detectors to quantify how the prevalence and magnitude of benchmark memorization vary across model families, languages, and repository ecosystems.

References

  1. Liang, Shanchao; Garg, Spandan; Moghaddam, Roshanak Zilouchian. 2025. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv:2506.12286, https://arxiv.org/abs/2506.12286
  2. Jain, Naman; Han, King; Gu, Alex; Li, Wen-Ding; Yan, Fanjia; Zhang, Tianjun; Wang, Sida; Solar-Lezama, Armando; Sen, Koushik; Stoica, Ion. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974, https://arxiv.org/abs/2403.07974
  3. Zan, Daoguang; Huang, Zhirong; Liu, Wei; Chen, Hanwu; Zhang, Linhao; Xin, Shulin; Chen, Lu; Liu, Qi; Zhong, Xiaojian; Li, Aoyan; Liu, Siyao; Xiao, Yongsheng; Chen, Liangqiang; Zhang, Yuyu; Su, Jing; Liu, Tianyu; Long, Rui; Shen, Kai; Xiang, Liang. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. arXiv:2504.02605, https://arxiv.org/abs/2504.02605
  4. Badertdinov, Ibragim; Golubev, Alexander; Nekrashevich, Maksim; Shevtsov, Anton; Karasik, Simon; Andriushchenko, Andrei; Trofimova, Maria; Litvintseva, Daria; Yangel, Boris. 2025. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. arXiv:2505.20411, https://arxiv.org/abs/2505.20411
  5. Zhang, Linghao; He, Shilin; Zhang, Chaoyun; Kang, Yu; Li, Bowen; Xie, Chengxing; Wang, Junhao; Wang, Maoquan; Huang, Yufan; Fu, Shengyu; Nallipogu, Elsie; Lin, Qingwei; Dang, Yingnong; Rajmohan, Saravan; Zhang, Dongmei. 2025. SWE-bench Goes Live! arXiv:2505.23419, https://arxiv.org/abs/2505.23419
  6. Adamenko, Pavel; Ivanov, Mikhail; Valeev, Aidar; Levichev, Rodion; Zadorozhny, Pavel; Lopatin, Ivan; Babaev, Dmitry; Fenogenova, Alena; Malykh, Valentin. 2025. SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks. arXiv:2507.11059, https://arxiv.org/abs/2507.11059
  7. Garg, Spandan; Steenhoek, Benjamin; Huang, Yufan. 2025. Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation. arXiv:2510.08996, https://arxiv.org/abs/2510.08996
  8. Prathifkumar, Thanosan; Mathews, Noble Saji; Nagappan, Meiyappan. 2025. Does SWE-Bench-Verified Test Agent Ability or Model Memory? arXiv:2512.10218, https://arxiv.org/abs/2512.10218
  9. Huang, Wenqi; Lee, Charley; Tng, Leonard; Ge, Serena. 2026. DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks. arXiv:2607.07946, https://arxiv.org/abs/2607.07946
  10. Shilov, Evgeny. 2026. RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications. arXiv:2607.06411, https://arxiv.org/abs/2607.06411
  11. Song, Tae-Eun. 2026. Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis. arXiv:2603.21454, https://arxiv.org/abs/2603.21454
  12. Rajput, Prateek Kumar; Bonkoungou, Abdoul Aziz; Djiré, Alberick Euraste; Tang, Xunzhu; Song, Yewei; Olatunji, Iyiola Emmanuel; Diallo, El Hacen; Klein, Jacques; Bissyandé, Tegawendé F. 2026. Memorization Diagnostics for Code LLMs Should be Scale-Aware. arXiv:2608.12771, https://arxiv.org/abs/2608.12771
  13. Shi, Weijia; Ajith, Anirudh; Xia, Mengzhou; Huang, Yangsibo; Liu, Daogao; Blevins, Terra; Chen, Danqi; Zettlemoyer, Luke. 2023. Detecting Pretraining Data from Large Language Models. arXiv:2310.16789, https://arxiv.org/abs/2310.16789
  14. Oren, Yonatan; Meister, Nicole; Chatterji, Niladri; Ladhak, Faisal; Hashimoto, Tatsunori B. 2023. Proving Test Set Contamination in Black Box Language Models. arXiv:2310.17623, https://arxiv.org/abs/2310.17623
  15. Golchin, Shahriar; Surdeanu, Mihai. 2023. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. arXiv:2308.08493, https://arxiv.org/abs/2308.08493
  16. Sainz, Oscar; Campos, Jon Ander; García-Ferrero, Iker; Etxaniz, Julen; Lopez de Lacalle, Oier; Agirre, Eneko. 2023. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. arXiv:2310.18018, https://arxiv.org/abs/2310.18018
  17. Xu, Cheng; Guan, Shuhao; Greene, Derek; Kechadi, M-Tahar. 2024. Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244, https://arxiv.org/abs/2406.04244
  18. Gorinova, Maria I.; Baker, Macey; Heineike, Amy; Shaposhnikov, Maksim; Willoughby, Rob; Knox, Dru. 2026. Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering. arXiv:2606.17799, https://arxiv.org/abs/2606.17799