A 2026 ICLR study (arXiv:2510.02386) showed that benchmark contamination in large reasoning models (LRMs) becomes nearly undetectable by existing methods through two realistic pathways: supervised fine-tuning (SFT) contamination of a base model is concealed by subsequent PPO-style reinforcement learning with importance sampling and clipping (e.g., GRPO) on clean data, and final-stage SFT contamination with chain-of-thought (CoT) data on advanced LRMs leaves memorization-based detectors near random, because LRM generalization raises confidence even on similar unseen non-members. The open problem is to develop contamination-detection methods, and trustworthy audit protocols, whose assumptions replace the outdated premise that contamination equals verbatim memorization with lower loss, and that remain demonstrably effective under both concealment pathways.

Background

Large reasoning models (LRMs) are language models that generate long chains of thought (CoT) — step-by-step reasoning text produced before the final answer — and are typically built in two stages: supervised fine-tuning (SFT), where the model is trained by likelihood maximization on (question, response) pairs, often distilled from a stronger model, and reinforcement learning (RL) post-training with verifiable rewards, where the model generates candidate solutions and is optimized from reward feedback. Widely used RL algorithms such as GRPO (Group Relative Policy Optimization, introduced with DeepSeekMath, arXiv:2402.03300) and PPO (Proximal Policy Optimization) use importance-sampling ratios between the current and old policy that are clipped to a bounded range, which stabilizes training. Benchmark contamination means that evaluation questions (e.g., from AIME, AMC, OlympiadBench, GPQA-Diamond, Minerva Math) were included in the training data, inflating leaderboard scores. Detecting it is usually formalized as membership inference: given model access, distinguish 'member' questions (seen in training) from 'non-member' questions, measured by AUROC (area under the receiver operating characteristic curve; 50% is random guessing). Classical detectors are memorization-driven: they exploit lower per-token loss on members (e.g., Min-K%, arXiv:2310.16789, which scores a sample by the k least-likely tokens), generation-based or perturbation-based probes, or reference-model comparisons — all resting on the assumption that training samples get distinctly higher likelihood than unseen ones. The source study (Wang, Li, Ko and Zhang, arXiv:2510.02386, ICLR 2026) showed this assumption fails for LRMs in two practically central scenarios. Stage I (pre-LRM): when a base model is SFT-contaminated and then further trained with GRPO on clean data, the member/non-member log-probability distributions collapse onto each other, and ten representative detectors — reference-free (LOSS, Min-K%, Max-K%), reference-based, generation-based, perturbation-based — degrade monotonically with RL steps, the reference-free ones falling to near-random AUROC (≈50%) after as few as 156 steps and extensive GRPO expected to render all of them near random; the authors' theoretical analysis and ablations attribute this concealment to the PPO-style importance-sampling/clipping term itself, so a broad class of RL algorithms shares the effect (plain rejection-sampling fine-tuning without clipping does not conceal). Stage II (post-LRM): SFT contamination with CoT applied to advanced LRMs as the final training step barely leaves evidence for any existing detector, because without exposure to non-members the contaminated model also gains confidence on unseen questions distributionally similar to the training set — generalization, not memorization, drives the signal. The authors conclude that detectors relying solely on log-probabilities, and mitigations such as minor benchmark modifications, are inadequate, and they call for (i) detection methods that go beyond memorization-driven assumptions and account for long-CoT reasoning and LRM generalization capacity, and (ii) trustworthy evaluation protocols, including release of intermediate training checkpoints so each training stage can be monitored.

Problem Statement

Develop contamination-detection methods for large reasoning models — together with the trustworthy evaluation and audit protocols they require — that remain effective under the two contamination pathways demonstrated in arXiv:2510.02386, where existing memorization-driven detectors degrade to near-random performance: (1) Stage I: a base model is SFT-contaminated on benchmark data and then further trained with PPO-style RL (importance sampling with clipping, as in GRPO) on clean data, which collapses the member/non-member log-probability separability that current detectors rely on; and (2) Stage II: contamination with chain-of-thought data is applied to an advanced LRM as its final training stage, where LRM generalization raises confidence on distributionally similar unseen non-members and thereby evades memorization-based signals. Following the source study's call, such methods should replace the assumption that contamination manifests as verbatim memorization with assumptions that explicitly account for long-CoT reasoning and LRM generalization, and should operate without access to the released training CoT — the benchmark question–answer pairs and the model's generated outputs — while log-probability or intermediate-checkpoint access, whenever an audit releases it, may serve only as an optional, non-required enhancement. The associated protocol component asks what operational audit design — including intermediate checkpoint release, access models, and decision rules with characterized validity and statistical power — makes a 'no evidence of contamination' conclusion trustworthy for an LRM leaderboard entry. A resolution requires both components, assessed on both pathways: (a) a detection method whose member/non-member discrimination is demonstrably above the 50% chance level on Stage I and on Stage II alike (not merely on one), under black-box query access alone (benchmark question–answer pairs plus generated outputs) and measured on the final, post-concealment model so that a released pre-RL checkpoint does not by itself satisfy this bound, achieving a stage-wise AUROC of at least 0.65 whose confidence interval excludes 0.50, evaluated against matched controls, and (b) an audit-protocol design with an explicit validity and statistical-power analysis that, under a stated black-box adversary and access model, attains a minimum detectable contamination fraction at or below 10% of the benchmark at a target minimum power of at least 0.8, reporting power as a function of the contamination fraction rather than at a single solver-chosen point, and stating its assumptions and resulting detection limits; a single unified method spanning both pathways and a per-pathway suite of methods both satisfy (a). The two components jointly serve one objective: restoring the verifiability of LRM evaluation in the presence of adversaries who can exploit RL training itself to conceal contamination.

References

  1. Wang, Han; Li, Haoyu; Ko, Brian; Zhang, Huan (2026). On The Fragility of Benchmark Contamination Detection in Reasoning Models. arXiv:2510.02386, DOI 10.48550/arXiv.2510.02386, accepted at ICLR 2026. https://arxiv.org/abs/2510.02386
  2. Tao, Yongding; Wang, Tian; Dong, Yihong; Liu, Huanyu; Zhang, Kechi; Hu, Xiaolong; Li, Ge (2025). Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models. arXiv:2510.09259, DOI 10.48550/arXiv.2510.09259. https://arxiv.org/abs/2510.09259
  3. Liu, Yule; Zhang, Heyi; Zheng, Jinyi; Sun, Zhen; Peng, Zifan; Wei, Jiaheng; Cong, Tianshuo; Yang, Yilong; He, Xinlei (2025). Auditing Data Membership in Reinforcement Learning With Verifiable Rewards. arXiv:2511.14045, DOI 10.48550/arXiv.2511.14045. https://arxiv.org/abs/2511.14045
  4. Zhang, Hongbo; Yang, Yue; Yan, Jianhao; Bao, Guangsheng; Zhang, Yue; Zhang, Yue (2026). Detecting RLVR Training Data via Structural Convergence of Reasoning. arXiv:2602.11792, DOI 10.48550/arXiv.2602.11792. https://arxiv.org/abs/2602.11792
  5. Gwak, Minju; Kwak, Minseo; Lee, Dongseok; Son, Guijin; Ritter, Alan; Kim, Jaehyung (2026). LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training. arXiv:2605.29888, DOI 10.48550/arXiv.2605.29888. https://arxiv.org/abs/2605.29888
  6. Lan, Yifan; Cao, Yuanpu; Wang, Hanyu; Lin, Lu; Chen, Jinghui (2026). The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation. arXiv:2605.21856, DOI 10.48550/arXiv.2605.21856. https://arxiv.org/abs/2605.21856
  7. Yang, Zhen; Wang, Mengqi; Zhao, Gengda; Zhou, Mo; Wang, Jianwei; Zhang, Wenjie (2026). Calibrating Post-Training Feature Shifts for LLM Data Contamination Detection. arXiv:2608.10462, DOI 10.48550/arXiv.2608.10462. https://arxiv.org/abs/2608.10462
  8. Shihab, Ibne Farabi; Akter, Sanjeda; Sharma, Anuj (2026). When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits. arXiv:2608.07914, DOI 10.48550/arXiv.2608.07914. https://arxiv.org/abs/2608.07914
  9. Lin, Jiuheng; Zhang, Chen; Feng, Yansong (2026). To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation. arXiv:2606.29481, DOI 10.48550/arXiv.2606.29481. https://arxiv.org/abs/2606.29481
  10. Wu, Mingqi; Zhang, Zhihao; Dong, Qiaole; Xi, Zhiheng; Zhao, Jun; Jin, Senjie; Fan, Xiaoran; Zhou, Yuhao; Lv, Huijie; Zhang, Ming; Fu, Yanwei; Liu, Qin; Zhang, Songyang; Zhang, Qi (2026). Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. arXiv:2507.10532, DOI 10.48550/arXiv.2507.10532, AAAI 2026. https://arxiv.org/abs/2507.10532
  11. Shi, Weijia; Ajith, Anirudh; Xia, Mengzhou; Huang, Yangsibo; Liu, Daogao; Blevins, Terra; Chen, Danqi; Zettlemoyer, Luke (2023). Detecting Pretraining Data from Large Language Models. arXiv:2310.16789, DOI 10.48550/arXiv.2310.16789, ICLR 2024. https://arxiv.org/abs/2310.16789
  12. Shao, Zhihong; Wang, Peiyi; Zhu, Qihao; Xu, Runxin; Song, Junxiao; Bi, Xiao; Zhang, Haowei; Zhang, Mingchuan; Li, Y. K.; Wu, Y.; Guo, Daya (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, DOI 10.48550/arXiv.2402.03300. https://arxiv.org/abs/2402.03300