Out-of-distribution (OoD) detection benchmarks for object detectors are contaminated in two documented ways: OoD-only test images contain in-distribution (ID) objects (Type-1 error), and ID-only calibration/test images contain unlabeled OoD objects (Type-2 error). Wu et al. (TPAMI 2026; arXiv:2503.07330) showed that in the standard PASCAL-VOC/OpenImages and BDD-100K/OpenImages setups roughly 13% of objects in OoD test sets are actually ID, inflating the calibrated threshold metric FPR95 (the rate at which an OoD filter accepts predicted boxes in OoD-only test images as ID, evaluated at the threshold τ\tau calibrated so that the filter's error rate on the ID-only calibration set stays below 5%) by up to 9.85 percentage points, and that per-prediction ID/OoD labeling is impractical at the roughly 200,000-prediction scale of BDD-100K. Their own remedy is an image-level automated screener plus optional manual inspection, whose residual contamination is not quantified. The open problem is to construct and validate, at benchmark scale, a per-box labeling policy or automatic curation pipeline that eliminates both contamination types, with the elimination certified by an audit whose per-box ground truth is not produced by the curation mechanism itself and whose audited sample is drawn at random by a rule fixed in advance.

Background

Object detectors such as those in the You Only Look Once (YOLO) family of real-time detectors predict, for every candidate region, an objectness score expressing how likely the region is to contain any object of the trained categories, plus a class label. On inputs containing objects outside the training categories (out-of-distribution, OoD, objects), detectors still fire confidently — so-called OoD hallucinations — which is a safety concern in applications such as autonomous driving. A large family of post-hoc OoD filters — scoring functions applied to an already-trained detector without retraining, such as maximum-softmax-probability (MSP), energy-based, Mahalanobis-distance (distance of a box's feature vector to the ID feature distribution), k-nearest-neighbor (KNN), and box-abstraction monitors — score each predicted bounding box and reject boxes whose score exceeds a threshold τ\tau.

The threshold τ\tau is conventionally calibrated by FPR95: following Wu et al. (Eqs. (3)-(4)), it is set on an ID-only calibration set so that the filter's error rate on that set remains below 5%, where that error rate counts every predicted box whose OoD score reaches τ\tau as a fraction of all predictions in the calibration set, not only the detector's correct ID detections. This protocol presupposes two dataset guarantees. First, every image in the OoD-only test set must be free of ID objects; otherwise a detector that correctly finds an ID object in an 'OoD' image is scored as making a false-positive hallucination. Second, every object in the ID-only calibration/test sets must be labeled, so that a filter correctly rejecting an unlabeled OoD object is not scored as making an error on an ID object. Wu et al. formalized the violations as the Type-1 data error (an image in the OoD-only test set contains at least one ID object) and the Type-2 data issue (an image in the ID-only set contains at least one detected box whose ground truth is 'ood' rather than an ID class). They quantified both on the widely used OoD benchmarks built from OpenImages (a large labeled image collection) for the PASCAL-VOC and BDD-100K (a large-scale driving-image dataset) in-distribution tasks: 104 of 1,852 OoD test images (5.62%) contain ID objects (12.75% and 13.77% of all detected objects for the VOC and BDD tasks), inflating FPR95 by 9.85 and 5.58 percentage points respectively, and 16.52% of BDD-100K images and 8.69% of VOC images contain at least one unlabeled OoD object. Because BDD-100K alone yields roughly 200,000 detector predictions, the authors state that assigning each prediction an accurate ID/OoD label is impractical, and they only approximate the Type-2 quantification with a pre-trained screening model. The Type-2 issue also biases the calibrated threshold itself: unlabeled OoD objects inside the ID calibration set push τ\tau outward, weakening the filter.

In response, the authors built new Near-OoD and Far-OoD benchmarks (about 1,000 images each, curated by a pre-trained YOLO screening model plus manual inspection in the first version, and by an automated image-level pipeline built on the FiftyOne data-curation and annotation tool and the YOLOE open-vocabulary detector — a detector that, unlike a fixed-vocabulary detector, can be prompted with arbitrary class names at inference time — discarding any candidate image in which an ID class is detected with confidence at least 0.25, with optional human-in-the-loop verification in the journal version). They also fine-tune detectors on proximal OoD data to suppress objectness on OoD objects, but that mitigation line is separate from the benchmark-curation question. The curation remedies remain image-level and single-screener: the pipeline's own residual contamination rate is not measured, and per-box OoD labels are not produced for ID sets at scale. That benchmark content moves the reported metric is not hypothetical: in the paper's ID-outlier analysis, removing visually atypical ID objects from the ID test set lowers FPR95 for every compared filter and reverses the ordering of two of them (BAM and KNN). Because FPR95 is a tail statistic — τ\tau is placed so that only 5% of the calibration predictions exceed it — a small number of contaminated boxes surviving inside that tail can shift it disproportionately.

Problem Statement

For out-of-distribution (OoD) detection benchmarks in object detection, in the framework of Wu et al. (TPAMI 2026; arXiv:2503.07330): construct and validate, at the scale of standard benchmarks such as PASCAL-VOC and BDD-100K with OpenImages-derived OoD sets (on the order of 10^5 detector predictions), a per-box labeling policy or automatic curation pipeline that eliminates both Type-1 contamination (in-distribution objects present in OoD-only test images) and Type-2 contamination (unlabeled OoD objects in ID-only calibration and test sets) — going beyond image-level screening with a single open-vocabulary detector. Elimination is understood in the source's own terms: ID objects are removed within the OoD-only dataset, and OoD objects are labeled in bounding boxes within the ID-only dataset, so that an OoD filter's correct rejection of them is counted as correct rather than as an error. Since no trusted per-box oracle exists, this must be certified by an audit whose per-box ground truth is not produced by the curation mechanism itself. The audited statistic is the residual contamination rate, measured per predicted box as the fraction of all predicted boxes in the curated benchmark whose ground-truth ID/OoD category disagrees with the benchmark's assignment, where a Type-1-contaminated box is an ID object inside an OoD-only test image and a Type-2-contaminated box is an OoD object inside an ID-only calibration or test image. Elimination is achieved, and the task is resolved, when, on a random sample drawn from the full curated benchmark by a rule fixed in advance, the 95% upper confidence bound on this rate lies below 1% for the combined Type-1 and Type-2 rate and, separately, for each of the two types, with the sample large enough that a true residual rate of 1% would be detected with probability at least 0.8. Residual contamination below these bounds, rather than a literal zero, defines elimination. The curated benchmark must remain fit for the evaluation it serves: curation may remove or relabel only objects identified as contaminated (Type-1 ID objects removed from OoD-only images, Type-2 OoD objects labeled in ID-only images), every in-distribution class and every out-of-distribution category present before curation must remain represented, including the semantically close near-OoD cases, and the curated benchmark must retain the original order of scale (on the order of 10^5 detector predictions), with the original and curated per-class predicted-box counts reported, so that the reported FPR95 values and the comparison of OoD-detection methods stay meaningful. The objective is contamination-free OoD benchmarks in the certified sense defined above, whose reported FPR95-style metrics are trustworthy.

References

  1. Changshun Wu, Weicheng He, Chih-Hong Cheng, Xiaowei Huang, Saddek Bensalem. Revisiting Out-of-Distribution Detection in Real-Time Object Detection: From Benchmark Pitfalls to a New Mitigation Paradigm. IEEE Transactions on Pattern Analysis and Machine Intelligence 48(5):5243-5260, 2026. DOI: 10.1109/TPAMI.2025.3650695, https://doi.org/10.1109/TPAMI.2025.3650695 (arXiv:2503.07330, https://arxiv.org/abs/2503.07330; first posted March 2025 as v1 under the title 'Mitigating Hallucinations in YOLO-based Object Detection Models: A Revisit to Out-of-Distribution Detection')
  2. Changshun Wu, Weicheng He, Xiaowei Huang, Saddek Bensalem. SPK: Eliciting Structured Prior Knowledge for Interpretable Out-of-Distribution Detection in Real-Time Object Detection. arXiv:2608.19080, 2026. DOI: 10.48550/arXiv.2608.19080, https://arxiv.org/abs/2608.19080
  3. Daniel Montoya, Aymen Bouguerra, Alexandra Gomez-Villa, Fabio Arnez. FindMeIfYouCan: Bringing Open Set metrics to near, far and farther Out-of-Distribution Object Detection. arXiv:2506.14008, 2025. DOI: 10.48550/arXiv.2506.14008, https://arxiv.org/abs/2506.14008
  4. Ling Lin, Yang Bai, Heng Su, Congcong Zhu, Yaoxing Wang, Yang Zhou, Huazhu Fu, Jingrun Chen. OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models. arXiv:2602.18094, 2026. DOI: 10.48550/arXiv.2602.18094, https://arxiv.org/abs/2602.18094
  5. Vishnu Bindu Balachandran. Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks. arXiv:2607.19393, 2026. DOI: 10.48550/arXiv.2607.19393, https://arxiv.org/abs/2607.19393
  6. Hejer Ammar, Nikita Kiselov, Guillaume Lapouge, Romaric Audigier. Open-set object detection: towards unified problem formulation and benchmarking. arXiv:2411.05564, 2024. DOI: 10.48550/arXiv.2411.05564, https://arxiv.org/abs/2411.05564