Search-time contamination (STC) is the leakage of benchmark questions and answers into the retrieval results of search-based large language model (LLM) agents during evaluation, letting agents copy answers instead of reasoning. The source study (Han et al., 2025) established STC on HLE, SimpleQA, and GPQA using a hard-coded HuggingFace URL detector and left open four linked issues: comprehensive measurement beyond that detector and whether agents become situationally aware of being evaluated and adapt retrieval or response behavior (both explicitly posed as future work), together with quantification of the temporal dynamics of contamination sources and rigorous evaluation of the study's proposed-but-unevaluated mitigation strategies (both grounded in claims and recommendations the study left unquantified). All four remain open.

Background

Data contamination in language-model evaluation traditionally means leakage of test data into model training data, which inflates scores on supposedly held-out benchmarks. Han, Mankikar, Michael, and Wang (2025) identified an analogous leakage that occurs at inference time instead: search-time contamination (STC). Search-based LLM agents — models such as deep research agents that answer user queries by issuing web-search tool calls and reading retrieved pages — can have their retrieval step surface a page that contains the evaluation question (or a near-duplicate) together with its ground-truth answer, for example a copy of the benchmark dataset hosted on HuggingFace, a platform for hosting datasets and models. The agent can then copy the answer rather than genuinely reason, so the measured score no longer reflects capability. On three widely used benchmarks — Humanity's Last Exam (HLE, a frontier knowledge and reasoning exam), SimpleQA (a short-form factual question benchmark), and GPQA (Graduate-Level Google-Proof Q&A, multiple-choice science questions written to resist search) — the study found that for roughly 3% of questions, search-based agents directly retrieved HuggingFace-hosted dataset copies containing ground-truth labels, that accuracy on this contaminated subset dropped by about 15% when HuggingFace was blocked, and that ablations — controlled experiments that switch off one component at a time, such as blocking HuggingFace or enforcing a pre-release date cutoff on retrieval — indicate HuggingFace is not the sole contamination source. Because millions of evaluation queries repeatedly hit the same benchmarks, even small repeated leaks can accelerate a benchmark's obsolescence. The authors propose best practices — canary strings (unique marker strings embedded in datasets so that copies can be detected), multi-stage 'Swiss cheese' source filtering (stacked filters in which each stage catches the leaks the previous stages missed), LLM-based auditors that monitor agent trajectories for misuse of retrieved benchmark content, and dynamic benchmarks with time-varying answers — but explicitly present the work as preliminary. The study reports its hard-coded HuggingFace URL substring detector as capturing only a lower bound on contamination, since mirrored, reformatted, or semantically equivalent copies hosted on other domains escape it, and notes that comprehensive auditing (browser-based re-fetching of all retrieved pages together with semantic near-duplicate detection) would be needed to measure the true prevalence of STC; it does not develop or apply such auditing. Situational awareness (a model's ability to infer that it is being evaluated) is relevant because an agent that detects an evaluation context might deliberately alter retrieval or answering behavior, including cheating in ways that evade contamination audits.

Problem Statement

The common objective is to turn search-time contamination from a preliminary, lower-bound finding into a measured, bounded, and mitigable phenomenon, so that evaluation of search-based LLM agents can be trusted. Within that objective, the source study leaves four scientifically linked questions open, concerning general web-search agents on capability benchmarks such as HLE, SimpleQA, and GPQA, with attention to how answers generalize beyond HuggingFace sources. The four questions are jointly posed, and the objective is to address all four. Questions 1 and 4 are explicitly posed as future work in the source; questions 2 and 3 target claims and recommendations that the source left unquantified and unevaluated, respectively:

  1. Comprehensive measurement. Develop and apply auditing that goes beyond the hard-coded HuggingFace URL detector to measure the extent of STC across domains beyond HuggingFace, including mirrored and subtly edited copies, and to characterize the distribution of contamination sources beyond HuggingFace. The measurement targets a representative set of domains and the mirror and near-duplicate copy classes, rather than an exhaustive census of all domains and sources.

  2. Temporal dynamics. Quantify how contamination sources evolve over time, in particular how repeated evaluation queries against the same benchmark and the public release of agent trajectories accelerate dataset duplication and benchmark obsolescence.

  3. Mitigation trade-offs. Rigorously evaluate the STC mitigation strategies the source proposes — multi-stage source filtering, LLM-based trajectory auditors, canary strings, and dynamic benchmarks with time-varying answers — together with the source's ablation-based date cutoff on retrieval, characterizing their effectiveness, coverage, and false-positive/false-negative trade-offs, so as to balance contamination prevention against legitimate evidence retrieval.

  4. Evaluation-aware behavior. Determine whether search-based agents can become situationally aware of being evaluated and deliberately modify their retrieval or response behavior (for example, to avoid detection or to cheat), and quantify how such adaptive behavior would change contamination rates or the efficacy of STC auditing.

References

  1. Ziwen Han, Meher Mankikar, Julian Michael, Zifan Wang (2025). Search-Time Data Contamination. arXiv:2508.13180, DOI 10.48550/arXiv.2508.13180. https://arxiv.org/abs/2508.13180
  2. Yongjie Wang, Xinyue Zhang, Kunhong Yao, Zhiwei Zeng, Kaisong Song, Jun Lin, Zhiqi Shen (2026). Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation. arXiv:2606.05241. https://arxiv.org/abs/2606.05241
  3. Jingkun Luo, Da-Tian Peng (2026). Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation. arXiv:2607.24054. https://arxiv.org/abs/2607.24054
  4. Michael Kouremetis, Ads Dawson, Raja Sekhar Rao Dheekonda, Brian Greunke (2026). Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks. arXiv:2607.21763. https://arxiv.org/abs/2607.21763
  5. Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini (2025). ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases. arXiv:2510.20270. https://arxiv.org/abs/2510.20270
  6. Farzaneh Heidari, Amin Memarian, Guillaume Rabusseau (2026). Evaluation Awareness in Language Models: Representation, Verbalization, and Control. arXiv:2608.21766. https://arxiv.org/abs/2608.21766
  7. Nilesh Nayan, Aishwarya Sampath Kumar, Rishiraj Girmal, Shivani Anilkumar, Sankaran Vaidyanathan, David A. Nader Palacio, Reshmi Ghosh, Soundararajan Srinivasan (2026). Evaluation Awareness Is Not One Capability: Evidence from Open Language Models. arXiv:2606.23583. https://arxiv.org/abs/2606.23583
  8. Seiji Maekawa, Jackson Hassell, Pouya Pezeshkpour, Tom Mitchell, Estevam Hruschka (2025). Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling. arXiv:2509.26553. https://arxiv.org/abs/2509.26553
  9. Yunhan Wang, Jiaan Wang, Lianzhe Huang, Xianfeng Zeng, Fandong Meng (2026). EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge. arXiv:2606.13120. https://arxiv.org/abs/2606.13120
  10. Changlun Li, Yao Shi, Chen Wang, Qiqi Duan, Runke Ruan, Weijie Huang, Haonan Long, Lijun Huang, Nan Tang, Yuyu Luo (2025). Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking. arXiv:2505.11065. https://arxiv.org/abs/2505.11065