Bitterwolf, Müller and Hein (ICML 2023) showed that widely used out-of-distribution (OOD) test sets for ImageNet-1K are heavily contaminated with in-distribution (ID) content — in some cases more than 50% of the images — and argued that automating the removal of ID contamination is circular: a filter reliable enough to clean challenging OOD data would itself be an OOD detector that already solves the problem the benchmark is meant to pose. Their clean benchmark, NINCO, therefore consists of 5,879 images checked individually by hand, and later clean benchmarks (ImageNet-OOD) have likewise relied on manual curation, while the automated alternative (C-OOD's WordNet class filtering) demonstrably retains severe contamination. The open problem is to construct clean and challenging (near-)OOD test sets at scale by a general automated procedure that removes every image containing an ID-class object — including incidental background objects, not just class-label overlap — without circular reliance on OOD-detection competence and without stripping the challenging near-ID samples, together with a validated estimate of residual contamination; or, alternatively, to formalize and prove the impossibility of doing so. Resolving it would make near-OOD evaluation possible at the scale of far-OOD corpora and remove a systematic bias from reported progress in OOD detection.

Background

Out-of-distribution (OOD) detection for an image classifier is the task of deciding, at test time, whether an input belongs to none of the classes the classifier was trained on — the in-distribution (ID) classes. Performance is typically reported as AUROC (area under the receiver-operating characteristic curve) or FPR95 (the false-positive rate on OOD inputs at the operating point where 95% of ID inputs are correctly accepted), measured on fixed benchmark OOD test sets. Near-OOD test sets contain classes semantically close to the ID classes (the hardest and most safety-relevant regime); far-OOD test sets contain clearly unrelated imagery. For the ImageNet-1K (IN-1K) ID task, Bitterwolf, Müller and Hein (ICML 2023) found that the standard test OOD sets — including sets inherited from open-set recognition (the classification setting where test inputs may belong to classes unseen during training and must be rejected) — suffer severe ID contamination: in some cases more than 50% of the images contain an object from an IN-1K class. Contamination arises in two modes: class-level semantic overlap, where the OOD class label itself overlaps an ID class (e.g. "quail" against the ID class "bobwhite quail"), and incidental contamination, where an ID object merely appears in an otherwise-OOD image (in the background or as a co-occurring object). Both inflate the apparent false-positive rate of every detector, because flagging such images as OOD is actually a correct ID decision, which systematically distorts detector rankings. As the fix, the paper introduced NINCO (No ImageNet Class Objects), 5,879 images in 64 OOD classes, each image individually checked by hand against the WordNet glosses and keyword lists that define the IN-1K classes (ImageNet classes correspond to WordNet synsets, each with a dictionary-style gloss and a list of synonymous keywords). Its cleanliness rule is non-permissive: an OOD test sample "cannot fall into the definition of any keyword of any IN-1K class", a sample is removed if an ID object is at least partially visible, and a sample for which it is ambiguous whether an ID object is present is not included in the cleaned dataset; the human checkers were shown the top-5 IN-1K predictions of a Vision Transformer for each image, so the construction itself already mixed model output with per-sample human inspection. Difficulty in the paper is measured detector-relatively, as the FPR at 95% TPR obtained per OOD class, and the stated aim for NINCO was "a variety of classes that are challenging, diverse and, most importantly, not actually categorically ID to begin with". The paper's central negative observation is a circularity argument: automated filtering of challenging OOD data "would require a strong detector that already solves the problems that the dataset is meant to pose", so "it is impossible to construct a clean and challenging OOD dataset without manually checking the OOD samples for ID contamination". The known automated attempt at scale — the C-OOD benchmark of Galil, Dabbah and El-Yaniv (ICLR 2023), built from ImageNet-21K (the larger ImageNet release with roughly 21,000 classes, which contains the IN-1K classes) by filtering classes according to their WordNet distance from the ID classes — was documented by Bitterwolf et al. to retain severe contamination of both modes. Subsequent clean benchmarks reverted to manual curation: ImageNet-OOD (Yang, Zhang and Russakovsky, ICLR 2024) was built from ImageNet-21K through roughly twenty hours of manual class selection plus a six-hour human review on top of WordNet-based candidate exclusion, and the OpenOOD v1.5 benchmark suite (Zhang et al.) adopts the hand-curated NINCO unchanged as its ImageNet near-OOD component. Manual checking caps benchmark size at a few thousand to a few tens of thousands of images, so near-OOD evaluation at the scale of far-OOD corpora remains out of reach, and later work (Long et al., NeurIPS 2024) shows that ID membership is gradational — a Sorites-style borderline problem — rather than sharply decidable. The question is whether a general automated procedure can replace manual inspection, or whether the circularity argument can be made into a precise impossibility theorem.

Problem Statement

Given ImageNet-1K as the in-distribution image classification dataset, with the class definitions that fix what counts as an ID object — its WordNet synsets, glosses and keyword lists — and a large candidate corpus of natural images, is there a general automated procedure that produces a clean and challenging OOD test set from that corpus, in the following sense: (1) every retained image is free of ID content, meaning it contains no object falling under any ID class definition, judged by the source's non-permissive rule — an ID object that is at least partially visible disqualifies the image, and an image for which it is ambiguous whether an ID object is present is not retained — so that not only class-label overlap but incidental ID objects appearing in the background, partially depicted, or co-occurring with the OOD subject are caught; (2) the retained set is challenging and not merely clean: its classes and images are semantically near the ID classes, in the near-OOD regime the benchmark exists to probe, and the set it produces is large — beyond the few thousand to few tens of thousands of images that manual per-sample curation delivers — so that a pipeline retaining only clearly far or unrelated imagery, or a degenerate output of a handful of images (for which (1) holds vacuously), does not answer the problem; (3) the procedure is non-circular, meaning its filtering signal does not reduce to the OOD-detection competence the benchmark is meant to measure — operationally, the filtering signal is not the ID task's own learned ID/OOD score, while general-purpose recognition components prompted with the ID class definitions (open-vocabulary object recognition, vision-language models prompted with class definitions) are admissible — and in particular it does not achieve cleanliness by discarding exactly the challenging near-ID samples that give the benchmark its value; (4) it scales to corpus sizes far beyond what manual per-sample inspection permits, with automated per-image cost and no per-sample human adjudication of ID content: pre-existing human-produced resources (WordNet glosses and keyword lists, models pretrained on human-labelled data) and a fixed-size human audit of the output are admissible, but human effort that grows with the corpus by deciding, image by image, whether ID content is present is not, since that is precisely the manual checking whose necessity the source asserts; and (5) its output carries a validated estimate of residual contamination — the fraction of retained images that in fact violate (1), measured on a human-audited sample — which quantifies how often the procedure's decisions fail without relaxing (1), the target property of its decision rule. Evidence about the effect of residual contamination on detector rankings, for example by comparison against a manually verified reference benchmark such as NINCO, bears on the source's finding that contamination systematically underestimates true detection performance and unrightfully punishes stronger detectors.

Bitterwolf, Müller and Hein (2023) assert that this is impossible without manual checking; the problem is therefore either to exhibit such a procedure with convincing validation, or to make the impossibility claim precise and prove it. The source states the claim only as an informal argument — automation of filtering for challenging OOD data "would require a strong detector that already solves the problems that the dataset is meant to pose" — so on the impossibility branch the model of admissible filtering procedures and of "detector competence" has to be supplied by the answer, and it must be adequate to the claim rather than chosen to make the claim true: it must admit every automated procedure realizable with currently available general-purpose recognition components (open-vocabulary object recognition, vision-language models prompted with class definitions) together with the sublinear human assistance allowed in (4); it must be able to express the source's own construction — per-sample human inspection assisted by model predictions — as a procedure within the model; and the impossibility must survive enlarging the admissible class in these ways. A theorem obtained by restricting admissibility, for example to filters built from detector-derived scores or to procedures denied pretrained models and human-labelled data, establishes only a statement about that restriction, not the source's claim. The scope of the ID task is likewise the source's: ImageNet-1K with its WordNet class definitions.

References

  1. Julian Bitterwolf, Maximilian Müller, Matthias Hein (2023). In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation. ICML 2023. arXiv:2306.00826, DOI:10.48550/arXiv.2306.00826. https://arxiv.org/abs/2306.00826
  2. Ido Galil, Mohammed Dabbah, Ran El-Yaniv (2023). A framework for benchmarking class-out-of-distribution detection and its application to ImageNet. ICLR 2023. arXiv:2302.11893. https://arxiv.org/abs/2302.11893
  3. William Yang, Byron Zhang, Olga Russakovsky (2024). ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms. ICLR 2024. arXiv:2310.01755. https://arxiv.org/abs/2310.01755
  4. Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, Yiran Chen, Hai Li (2023). OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. arXiv:2306.09301. https://arxiv.org/abs/2306.09301
  5. Xingming Long, Jie Zhang, Shiguang Shan, Xilin Chen (2024). Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox. NeurIPS 2024. arXiv:2406.09867. https://arxiv.org/abs/2406.09867
  6. Vishnu Bindu Balachandran (2026). Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks. arXiv:2607.19393. https://arxiv.org/abs/2607.19393
  7. Gerhard Krumpl, Henning Avenhaus, Horst Possegger (2026). ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research. WACV 2026. arXiv:2601.10802, DOI:10.1109/WACV61042.2026.00813. https://arxiv.org/abs/2601.10802
  8. Ari Spiesberger, Juan J. Vazquez, Nicky Pochinkov, Tomáš Gavenčiak, Peli Grietzer, Gavin Leech, Nandi Schoots (2026). Soft Contamination Means Benchmarks Test Shallow Generalization. arXiv:2602.12413. https://arxiv.org/abs/2602.12413