Selecting the best pretrained model for an unlabeled target dataset without exhaustively labeling it is the goal of pool-based active model selection. The MODEL SELECTOR framework (Okanovic et al., AISTATS 2025) made this practical: it models the relationship between candidate-model predictions and true labels with a single flip parameter ϵ\epsilon, tunes ϵ\epsilon from model votes without labels, and greedily queries the examples with maximal mutual information about which model is best. The authors themselves flag the theory gaps this leaves: the single-parameter channel is a simplification misaligned with the assumptions under which greedy information maximization is provably near-optimal, and model-vote (noisy) labels can tune ϵ\epsilon but cannot identify the best model. Follow-up work has answered parts of this agenda empirically — CODA (ICCV 2025) replaces the single parameter with per-classifier confusion matrices, and the same group's SELECT-LLM (2026) extends selection to large language models using only model outputs — but explicitly leaves theoretical guarantees for the selection rule and robustness to noisy or biased annotation oracles open. The surviving open problem is to put information-greedy active model selection on provable, annotation-realistic foundations: near-optimality or label-complexity guarantees (or negative results) for greedy information-gain acquisition in the black-box, multi-model, multi-class, pool-based setting under noise models richer than the single flip channel, and a characterization of whether the best model can be identified at all without trusted ground-truth labels.

Background

Public model zoos now host thousands of pretrained classifiers, so a practitioner deploying a model on a new target dataset faces a selection problem: which candidate achieves the highest accuracy on this data? Standard validation answers this by labeling a held-out set, which is exactly the cost one wants to avoid. Pool-based active model selection addresses this directly: given a pool of nn unlabeled target examples, a set of mm candidate models, and a labeling budget b≪nb \ll n, choose which examples to label so as to identify the best model (or one within a small accuracy gap of it) with as few labels as possible.

MODEL SELECTOR (Okanovic et al., AISTATS 2025; arXiv:2410.13609) is an influential instance of this idea. It assumes that the true label of an example is generated by taking the prediction of the (unknown) best model and flipping it independently with a single noise probability ϵ\epsilon — a symmetric-noise channel, generalizing the binary symmetric channel to KK classes, that compresses all class-conditional and model-specific error structure into one global parameter. Under this proxy likelihood, the method maintains a Bayesian posterior over which candidate is best and greedily queries the unlabeled example whose label has the largest mutual information with the best-model indicator, updating the posterior after each observation. The parameter ϵ\epsilon is tuned without any labels by building a "noisy oracle" from the distribution of model votes (predicted classes across the candidate pool). Across 18 model collections on 16 datasets (over 1,500 pretrained models), this reduces labeling cost by up to 94.15% relative to the strongest baseline, but each queried example still requires a trusted ground-truth label.

Greedy mutual-information (information-gain) acquisition has provable near-optimality only in stylized settings: Chen, Hassani, Karbasi, and Krause (COLT 2015) proved that greedy sequential information maximization is near-optimal under specific structural conditions on the observation channel, and Chattopadhyay, Haeffele, Vidal, and Geman (ICML 2024) derived performance bounds for active binary testing with information maximization. MODEL SELECTOR's own text concedes that its single-parameter channel is "a major simplification" of the setting studied by Chen et al. and misaligned with their modeling assumptions, and that small ϵ\epsilon produces overconfident posterior updates (each observation rescales a model's posterior by a factor (1−ϵ)/ϵ(1-\epsilon)/\epsilon). The paper asserts, without proof, that the cited guarantees "generally apply" to its case. Empirically it also shows a second limitation: the model-vote noisy oracle can tune ϵ\epsilon, but it cannot identify the best model — the best model selected using noisy vote labels trails the true best model by up to 15.93 accuracy points across its 18 collections (its Table 5), leading to the conclusion that labels are required to identify the best model.

Later work has advanced parts of this agenda empirically. CODA (Kay et al., ICCV 2025) replaces the single flip parameter with a Dawid–Skene-style model — the classical framework in which each annotator is represented by a confusion matrix of per-category error rates — so each candidate classifier carries its own per-category error profile, and it derives informative unsupervised priors from model consensus; it is markedly more label-efficient but proves no guarantees, and its expected-information-gain acquisition empirically loses to simpler rules on some tasks. SELECT-LLM (Durmazkeser, Okanovic, Kirsch, Hoefler, Gürel; 2025–2026), a single framework introduced in 2025 under the name LLM SELECTOR and renamed in its 2026 revision, extends active model selection to generative language models, using only generated model outputs (a posterior-weighted function of pairwise similarities between candidate responses) and an external annotation oracle; that revision explicitly restricts itself to a reliable reference-based oracle and lists theoretical guarantees for the selection rule and robustness to noisy or biased feedback sources as open future work. Thus the empirically successful paradigm — greedy information gain over a best-model posterior, driven by model-vote structure — still lacks any guarantee valid for the setting where it is actually used, and the necessity of trusted labels remains uncharacterized.

Problem Statement

The problem is the surviving open core of the theory gaps flagged by Okanovic et al. (AISTATS 2025) and left open by later work: their single-parameter channel is misaligned with the assumptions under which greedy information maximization is provably near-optimal, and model-vote labels empirically fail to identify the best model, while SELECT-LLM (2026) explicitly lists theoretical guarantees for the selection rule and robustness to noisy or biased feedback sources as open future work, after later literature empirically addressed the richer-noise-model and generative-model strands. The goal is to place information-greedy pool-based active model selection over black-box model pools on provable, annotation-realistic foundations. Concretely, in the setting of nn unlabeled pool examples, m≥2m \geq 2 black-box candidate models (hard predictions, or generated outputs scored against an annotation rule), budget b≪nb \ll n, and an observation/annotation channel whose noise may be richer than a single global flip parameter — per-model and per-category error structure such as confusion-matrix (Dawid–Skene) models, correlated errors across models, or noisy and biased annotation oracles — two jointly linked questions are open:

(a) Acquisition-rule guarantees. For the greedy mutual-information (expected-information-gain) selection rule over the best-model posterior, establish provable performance guarantees valid in this actual setting — for example, a near-optimality approximation ratio of greedy versus the best adaptive policy, or label-complexity bounds for identifying the best model or a within-δ\delta-of-best model (one whose accuracy on the target data is within δ\delta of the best candidate's) — under explicitly stated noise/annotation models; or establish the complementary negative results, such as model pools and noise regimes in which greedy information gain is far from optimal or no policy within the budget can identify the best model.

(b) Annotation-sufficiency (label-free identification). Characterize whether the best model can be identified without trusted ground-truth labels by exploiting structure in model votes: give conditions on the candidate pool under which a noisy oracle built from model votes identifies the best (or within-δ\delta-of-best) model, algorithms with provable guarantees for such label-free identification, or lower bounds showing that trusted labels are necessary.

The two questions serve one objective — reliable label-efficient model selection: (a) certifies that the query rule spends the annotation budget near-optimally, and (b) determines what the annotations must be (trusted labels versus exploitable model-vote structure). Both questions are posed: each is resolved by establishing any one of the alternatives listed for it — a positive guarantee or bound, or the complementary negative or impossibility result — under explicitly stated noise/annotation models.

References

  1. Patrik Okanovic, Andreas Kirsch, Jannes Kasper, Torsten Hoefler, Andreas Krause, Nezihe Merve Gürel. "All models are wrong, some are useful: Model Selection with Limited Labels." AISTATS 2025, PMLR 258:2035–2043. arXiv:2410.13609, DOI: 10.48550/arxiv.2410.13609, https://arxiv.org/abs/2410.13609
  2. Yuxin Chen, S. Hamed Hassani, Amin Karbasi, Andreas Krause. "Sequential Information Maximization: When is Greedy Near-optimal?" COLT 2015, PMLR 40:338–363, http://proceedings.mlr.press/v40/Chen15b.html
  3. Aditya Chattopadhyay, Benjamin David Haeffele, René Vidal, Donald Geman. "Performance Bounds for Active Binary Testing with Information Maximization." ICML 2024, PMLR 235:6346–6371, https://proceedings.mlr.press/v235/chattopadhyay24a.html
  4. A. P. Dawid, A. M. Skene. "Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm." Applied Statistics 28(1):20–28, 1979. DOI: 10.2307/2346806, https://doi.org/10.2307/2346806
  5. Justin Kay, Grant Van Horn, Subhransu Maji, Daniel Sheldon, Sara Beery. "Consensus-Driven Active Model Selection." arXiv:2507.23771 (2025); also ICCV 2025. DOI: 10.48550/arXiv.2507.23771, https://arxiv.org/abs/2507.23771
  6. Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel. "Large Language Model Selection with Limited Annotations." arXiv:2510.09418 (2026; v1, 2025, titled "Active Model Selection for Large Language Models"). DOI: 10.48550/arXiv.2510.09418, https://arxiv.org/abs/2510.09418