Perez, Kiela, and Cho (2021) showed that classical model-selection criteria — leave-one-out cross-validation (CV), minimum description length (MDL), and their variants — select language-model prompts and fine-tuning hyperparameters barely better than random when only a handful of labeled examples exist, a setting they called true few-shot learning; selection variance grows with model size, and reliable hyperparameter selection appeared to require thousands of labeled examples. The question that work raised is whether model selection for language models can be made reliable with few or no labeled validation examples, and how many labels reliability requires. Follow-up literature has produced strong label-free prompt-selection criteria (mutual information and its variants, perplexity, loss-landscape flatness), zero-label selection and routing schemes, and label-efficient active model ranking, but none is reliable across all tasks, domains, and model scales; selection for fine-tuning hyperparameters and a general theory of validation-sample requirements remain open. The open problem is a selection procedure, scaling law, or impossibility characterization that addresses the true few-shot regime in its full generality — across tasks, domains, model scales, and the three selection targets (prompts, fine-tuning hyperparameters, and models).

Background

Large pretrained language models (LMs) can be adapted to a new task without gradient updates by conditioning on a natural-language prompt — a textual template that reformats task inputs, optionally together with a few worked examples shown in the model's context (in-context learning). Performance is notoriously sensitive to design choices: prompt phrasing, the choice and ordering of in-context examples, and, when the model is fine-tuned on the small dataset, optimization hyperparameters. Standard practice therefore tunes these choices on a labeled validation set. Perez, Kiela, and Cho (2021) isolated the setting where no such held-out set exists, calling it true few-shot learning: every labeled example available must serve both for fitting the model and for making each design decision. They evaluated classical model-selection criteria for this job — leave-one-out cross-validation (CV), which scores a candidate by its loss when predicting each held-out example after fitting on the rest, and minimum description length (MDL), which picks the candidate that compresses the labels of the example set most effectively — together with appendix variants: Bayesian CV (marginalizing the held-out loss over a posterior), an interpolation MDLβ between CV and MDL, and joint log-probability of input and label. All performed near randomly on average across tasks and model sizes, at times selecting candidates clearly worse than a random choice; selection-based gains fell 5–7% short of the best candidate identified with a large labeled set, the variance of gains grew with model size (largest for GPT-3 175B), and selecting better-than-average fine-tuning hyperparameters appeared to require several thousand labeled examples. They concluded that model selection, rather than learning itself, is the binding constraint in low-data regimes, and suggested that alternative assumptions — meta-learning over many tasks, transfer from related labeled datasets, or use of unlabeled data — are more promising than raw true few-shot selection. Subsequent work has attacked parts of this question along four lines. Label-free selection criteria score candidate prompts using only the LM's output probabilities on unlabeled inputs: mutual information between input and model output (Sorensen et al. 2022), prompt perplexity (Gonen et al. 2022), and a flatness measure of the loss landscape (Shen et al. 2023); a unified analysis (Yang et al. 2023) shows these are largely variants of the mutual-information criterion and pushes label-free selection effectiveness close to the oracle across 13 tasks. Zero-label selection methods avoid labels altogether via prompt-ensemble pseudo-labels (Liao et al. 2022) or weak-supervision graphical models over model outputs for routing among LLMs (Guha et al. 2024). A third line treats the problem as label-efficient evaluation, actively choosing which few validation examples to annotate in order to rank models (Hu et al. 2023; Okanovic et al. 2024; Durmazkeser et al. 2025), or estimating task accuracy from unlabeled data (Fu et al. 2023; Li et al. 2024). Finally, Schick and Schütze (2021) showed that PET, a method that fine-tunes on many prompts and averages over them rather than selecting one, performs strongly without a dev set — sidestepping rather than solving the selection problem. No line, however, establishes selection that is reliable across tasks, domains, model scales, and selection targets, and no general theory of the validation-sample requirement exists.

Problem Statement

The problem, as posed by Perez et al. (2021) and only partially addressed since, is whether model selection for language models can be made reliable in the true few-shot regime — where at most a handful of labeled examples are available and no large held-out validation set exists, so every labeled example must serve both for fitting and for making design decisions. Following Perez et al. (2021), selection counts as reliable when the generalization loss of the selected candidate is low both in expectation and in variance across training sets and random factors, so that improvement over the average candidate is obtained with high probability rather than merely on average. Four facets share one coherent objective, reliable selection under extreme label scarcity across the three selection targets — prompts, fine-tuning hyperparameters, and models — and a complete answer must address it as such rather than a proxy: (1) Does there exist a model-selection criterion or practical procedure — beyond cross-validation, minimum description length, and their tested variants — that, using at most a handful of labeled examples (down to none for prompt and model selection, though fine-tuning hyperparameter selection requires at least a small labeled budget) and unlabeled data, selects prompts, fine-tuning hyperparameters, and models whose test performance approaches that obtained by selection on a large labeled validation set, reliably across tasks, domains, model scales, and selection targets? (2) What is the sample complexity of reliable selection — how many labeled validation examples are needed, as a function of the selection target (prompt versus fine-tuning hyperparameter versus model), the task, the candidate-set size, and the model? (3) How should the validation budget scale with model capacity, given the observation that selection variance grows with model size? (4) Can purely label-free selection — via model output probabilities, pseudo-labels, or prompt ensembles — close the gap to held-out-tuned selection performance? These four facets are not four separately mandatory deliverables but the sub-questions through which the single shared objective is attacked; a complete answer is any one of the admissible forms below that resolves it across all three selection targets. Alternative assumptions that go beyond the strict true few-shot regime — meta-learning across many tasks, transfer from related labeled datasets, or data-augmentation schemes — were recommended by Perez et al. (2021) as promising complementary directions rather than as part of this question; an answer that relies on them addresses a relaxed setting and constitutes complementary progress, not a resolution of the strict-regime question posed here. An admissible answer may take the form of a validated selection procedure with evidence of approaching held-out-tuned performance at a stated label budget, a quantitative scaling law or theory characterizing when reliable low-label selection is possible, or a formal impossibility or lower-bound result delimiting the settings in which no criterion can be reliable; in whichever form, it must address the shared objective across the selection targets, so that coverage of a single target in isolation — for example, in-distribution prompt templates alone — constitutes partial progress rather than a complete answer.

References

  1. Ethan Perez, Douwe Kiela, Kyunghyun Cho. True Few-Shot Learning with Language Models. arXiv:2105.11447, 2021 (NeurIPS 2021). DOI: 10.48550/arXiv.2105.11447. https://arxiv.org/abs/2105.11447
  2. Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, David Wingate. An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels. arXiv:2203.11364, 2022 (ACL 2022). DOI: 10.48550/arXiv.2203.11364. https://arxiv.org/abs/2203.11364
  3. Chonghua Liao, Yanan Zheng, Zhilin Yang. Zero-Label Prompt Selection. arXiv:2211.04668, 2022. DOI: 10.48550/arXiv.2211.04668. https://arxiv.org/abs/2211.04668
  4. Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, Luke Zettlemoyer. Demystifying Prompts in Language Models via Perplexity Estimation. arXiv:2212.04037, 2022. DOI: 10.48550/arXiv.2212.04037. https://arxiv.org/abs/2212.04037
  5. Lingfeng Shen, Weiting Tan, Boyuan Zheng, Daniel Khashabi. Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency. arXiv:2305.10713, 2023. DOI: 10.48550/arXiv.2305.10713. https://arxiv.org/abs/2305.10713
  6. Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lee, Minjoon Seo. Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis. arXiv:2305.14877, 2023 (TACL 2024). DOI: 10.48550/arXiv.2305.14877. https://arxiv.org/abs/2305.14877
  7. Bangzheng Li, Ben Zhou, Xingyu Fu, Fei Wang, Dan Roth, Muhao Chen. FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimation. arXiv:2406.11243, 2024. DOI: 10.48550/arXiv.2406.11243. https://arxiv.org/abs/2406.11243
  8. Harvey Yiyun Fu, Qinyuan Ye, Albert Xu, Xiang Ren, Robin Jia. Estimating Large Language Model Capabilities without Labeled Test Data. arXiv:2305.14802, 2023. DOI: 10.48550/arXiv.2305.14802. https://arxiv.org/abs/2305.14802
  9. Timo Schick, Hinrich Schütze. True Few-Shot Learning with Prompts — A Real-World Perspective. arXiv:2111.13440, 2021. DOI: 10.48550/arXiv.2111.13440. https://arxiv.org/abs/2111.13440
  10. Zhengyu Hu, Jieyu Zhang, Yue Yu, Yuchen Zhuang, Hui Xiong. How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking. arXiv:2312.01619, 2023. DOI: 10.48550/arXiv.2312.01619. https://arxiv.org/abs/2312.01619
  11. Patrik Okanovic, Andreas Kirsch, Jannes Kasper, Torsten Hoefler, Andreas Krause, Nezihe Merve Gürel. All models are wrong, some are useful: Model Selection with Limited Labels. arXiv:2410.13609, 2024. DOI: 10.48550/arXiv.2410.13609. https://arxiv.org/abs/2410.13609
  12. Neel Guha, Mayee F. Chen, Trevor Chow, Ishan S. Khare, Christopher Ré. Smoothie: Label Free Language Model Routing. arXiv:2412.04692, 2024. DOI: 10.48550/arXiv.2412.04692. https://arxiv.org/abs/2412.04692
  13. Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel. Active Model Selection for Large Language Models. arXiv:2510.09418, 2025. DOI: 10.48550/arXiv.2510.09418. https://arxiv.org/abs/2510.09418