Near-out-of-distribution (near-OOD) benchmarks can be silently contaminated when a benchmark designates a class the model was trained on as the "OOD" class: the class's training data stays inside the detector's in-distribution fit set, and OOD scores invert below chance. Balachandran (2026) distilled this failure into a leak fingerprint — near-perfect supervised decodability (OracleLDA at or above 0.95) co-occurring with collapsed unsupervised detection (best unsupervised AUROC below 0.65) — validated it on 52 controlled settings with ResNet-50 and ViT-B/16 on CIFAR-10/100, and audited 24 standard near/far OOD pairs at CIFAR scale plus text and document settings, finding standard cross-dataset construction clean. That paper explicitly leaves a larger survey across ImageNet-scale near-OOD suites to future work, and the reliability boundary of its single-threshold rule (false fires on intrinsically hard clean pairs, missed leaks on feature-peripheral classes) has been mapped only at CIFAR scale. The open problem is to audit the published ImageNet-scale near-OOD benchmark suites for fit-set leakage with the fingerprint and its confirmatory fit-set-exclusion contrast: adjudicate every firing as a genuine leak or an intrinsically hard clean pair, report which suites are contaminated and by what mechanisms, and establish where the fingerprint's decision rule holds at this scale.

Background

Out-of-distribution (OOD) detection asks a classifier to flag inputs that do not belong to its in-distribution (ID) training classes, and it is a standard component of reliable open-world systems. Post-hoc detectors score OOD-ness on a frozen network without retraining: output-based scores such as the maximum softmax probability, and feature-based scores such as the Mahalanobis distance, which measures how far an input's penultimate-layer embedding (the network's last hidden-layer representation) lies from the ID class means, scaled by the ID feature covariance. Near-OOD denotes the regime where the OOD test classes are semantically close to the ID classes (for example CIFAR-100 versus CIFAR-10, two standard small-image benchmarks with 100 and 10 object classes), as opposed to far-OOD (for example natural images versus digit images); Ren et al. (2021) showed that Mahalanobis-style scores degrade exactly in this near regime, and benchmark suites such as OpenOOD (Yang et al., 2022) and OpenOOD v1.5 (Zhang et al., 2023) standardize near- and far-OOD evaluation, including an ImageNet-scale track that underlies much of the field's leaderboard comparisons.

A key evaluation hazard in this area is benchmark-construction leakage. Some near-OOD splits are built by designating a class the model was trained on as the "OOD" class. The detector's fit set — the data used to fit its ID model, such as class means and covariances in feature space — then contains the designated OOD class's training data, so the OOD test points sit inside the fitted ID region and the detector is penalized for correctly ranking them as familiar. Balachandran (2026) encountered this as a below-chance AUROC (area under the receiver operating characteristic curve, a standard threshold-free ranking metric where 0.5 is chance level) of 0.326 while auditing a perturbation-based detector on a deployed multimodal document benchmark (Constantinou et al., 2026); deleting the class from training and retraining 35 models across two domains raised the score to 0.911. The paper distills the contamination into a leak fingerprint: a designated split fires when supervised decodability is near-perfect — OracleLDA, the AUROC of the best label-supervised linear discriminant (Fisher/LDA) reader of the model's embedding space, at or above 0.95 — while every unsupervised detector collapses (best unsupervised AUROC below 0.65). The intuition is that a leaked class is perfectly separable given labels but lies inside the one-class fit set, so no unsupervised score can find it. On a controlled battery of 52 settings (20 leaked, 32 clean) spanning ResNet-50 (a 50-layer residual convolutional network) and ViT-B/16 (a base-size Vision Transformer operating on 16-by-16-pixel image patches) on CIFAR-10/100, the fingerprint attained sensitivity 18/20 and specificity 31/32: the missed leaks were the feature-peripheral CIFAR-10 "cat" class, which an unsupervised detector ranks as borderline-OOD even when leaked, and the single false fire was the intrinsically hard clean pair CIFAR-100 (ID) versus CIFAR-10 (OOD), whose unsupervised detection collapses with no leak at all. The paper therefore identifies the confirmatory signal as the fit-set-exclusion contrast — refitting the detector with the suspect class's data excluded from the fit set and checking whether unsupervised detection recovers — which was perfect (20/20) on the battery and requires only cheap detector refits rather than backbone retraining, but is reliable only in the penultimate-embedding space and not in the weaker perturbation-signature space, where genuine leave-class-out retraining is needed. An in-the-wild audit of 24 standard near/far OOD pairs (CIFAR-scale cross-dataset pairs plus text and document settings) fired on exactly the one predicted hard pair and on no far-OOD pair, indicating that standard cross-dataset construction does not introduce this leak.

Adjacent results frame but do not perform the ImageNet-scale survey. Bitterwolf et al. (2023) audited ImageNet-scale OOD test sets and found severe contamination of a different kind — ID objects mislabeled as OOD inside far-OOD test sets, which inflates false positives — and released the hand-checked NINCO benchmark (a test OOD set whose samples are each verified to contain no ImageNet-class object); that mechanism sits in the OOD test data, whereas the fit-set leak sits in the ID training data and inverts scores rather than inflating them. Ascarate et al. (2026) independently documented score-direction instability when a held-out class overlaps the normal mixture in class-split anomaly detection, proposing an unsupervised k-nearest-neighbor overlap index; Balachandran (2026) shows that index is blind to separable-cluster leaks (chance-level separation on the 52-setting battery), so the two diagnostics are complementary. On the theory side, Balachandran (2026) explains the persistent gap between supervised and unsupervised detection with two results for Gaussian signatures with shared covariance — an ID-only detector's AUROC depends on the OOD mean shift only through a whitened distance, and whitening misallocates its detection budget to noise directions — corroborated by Li et al. (2025) and Janiak et al. (2025). The validation of the fingerprint, however, is limited to the four fine-tuned CIFAR-scale backbones, and the paper states that a larger survey across ImageNet-scale near-OOD suites is left to future work; other near-OOD suites may contain contamination mechanisms not seen in its 24-pair audit, whose coverage the paper itself does not claim to be exhaustive.

Problem Statement

Audit the published ImageNet-scale near-OOD benchmark suites for fit-set leakage — the contamination in which a benchmark designates a class the model was trained on as the OOD class, leaving that class's training data inside the detector's in-distribution fit set — using the leak fingerprint and its confirmatory contrast introduced by Balachandran (2026). Concretely, the audit comprises: (i) fixing an explicit, field-representative list of published ImageNet-scale near-OOD suites (including, at minimum, the near-OOD test sets of the OpenOOD v1.5 ImageNet track such as NINCO, and any near-OOD benchmarks built from ImageNet-1k by class-subset or fine-grained constructions where a trained class is repurposed as OOD), together with the ImageNet-trained backbones audited, and stating which published suites were excluded and why; (ii) computing, for each suite, the fingerprint's quantities — the supervised decodability OracleLDA and the best unsupervised detection AUROC in the model's penultimate-embedding space under the correct ID-train-only fit — and applying the firing rule (OracleLDA at least 0.95 with best unsupervised AUROC below 0.65); (iii) adjudicating every firing with the fit-set-exclusion contrast (refit the detector with the suspect class excluded from the fit set and test whether unsupervised detection recovers), supplemented by leave-class-out retraining where the contrast is inconclusive, distinguishing genuine leaks from intrinsically hard clean pairs in the documented false-fire mode; and (iv) reporting which suites are contaminated and by what mechanism, whether any contamination mechanism absent at CIFAR scale appears, and where the single-threshold rule's scope ends at ImageNet scale. The common objective binding these sub-goals is a determination of whether the near-OOD evaluation landscape at ImageNet scale — the scale that carries the field's leaderboard weight — rests on contaminated constructions, together with a validated statement of the fingerprint's operating characteristics at that scale. A clean outcome — no genuine leaks found, with explicit coverage and adjudication — would resolve the question negatively.

References

  1. Vishnu Bindu Balachandran (2026). Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks. arXiv:2607.19393. https://arxiv.org/abs/2607.19393
  2. Alejandro Ascarate, Leo Lebrat, Rodrigo Santa Cruz, Clinton Fookes, and Olivier Salvado (2026). Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection. arXiv:2606.02601. https://arxiv.org/abs/2606.02601
  3. Julian Bitterwolf, Maximilian Müller, and Matthias Hein (2023). In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation. ICML 2023; arXiv:2306.00826. https://arxiv.org/abs/2306.00826
  4. Christos Constantinou, Georgios Ioannides, Aman Chadha, Aaron Elkins, and Edwin Simpson (2026). Out of distribution detection with attention head masking for multimodal document classification. Scientific Reports 16:2449, doi:10.1038/s41598-025-32328-9. https://doi.org/10.1038/s41598-025-32328-9
  5. Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu (2022). OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. NeurIPS 2022 Datasets and Benchmarks Track; arXiv:2210.07242. https://arxiv.org/abs/2210.07242
  6. Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li (2023). OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. arXiv:2306.09301. https://arxiv.org/abs/2306.09301
  7. Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan (2021). A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection. arXiv:2106.09022. https://arxiv.org/abs/2106.09022
  8. Yucen Lily Li, Daohan Lu, Polina Kirichenko, Shikai Qiu, Tim G. J. Rudner, C. Bayan Bruss, and Andrew Gordon Wilson (2025). Out-of-Distribution Detection Methods Answer the Wrong Questions. ICML 2025 (extended version); arXiv:2507.01831. https://arxiv.org/abs/2507.01831
  9. Denis Janiak, Jakub Binkowski, and Tomasz Kajdanowicz (2025). A Geometry-Based View of Mahalanobis OOD Detection. arXiv:2510.15202. https://arxiv.org/abs/2510.15202
  10. Xingming Long, Jie Zhang, Shiguang Shan, and Xilin Chen (2024). Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox. arXiv:2406.09867. https://arxiv.org/abs/2406.09867
  11. Max Gutbrod, David Rauber, Danilo Weber Nunes, and Christoph Palm (2025). OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detection. arXiv:2503.16247. https://arxiv.org/abs/2503.16247