Quantitative theory for the fundamental limits of model selection: stability–consistency relations, finite-sample optimality of adaptive (bridge) criteria, and the cross-validation splitting-ratio paradox

A 2021 survey chapter by Ding, Yang, and Tarokh identifies a missing quantitative theory unifying the fundamental limits of inference (identifying a true model) and prediction (estimating efficiently) in model selection. The open problem comprises three jointly supporting gaps: (1) no quantitative relation between the stability (reproducibility) of a selected model and the selection consistency of the procedure has been established; (2) the conditions under which the bridge criterion or related adaptive AIC–BIC-interpolating criteria achieve optimal finite-sample performance are unknown, as are extensions of their asymptotic guarantees to high-dimensional regression, nonlinear models, and dependent data; and (3) the cross-validation paradox — better estimation of prediction error does not imply better identification of the better modeling procedure — lacks a finite-sample theory of how the train/validation splitting ratio affects the probability of correct procedure selection. These three gaps remain open.

Background

Model selection is the task of choosing, from a family of candidate statistical models of increasing complexity, the model to use for inference and prediction. The two classical benchmarks are the Akaike information criterion (AIC), which selects the model minimizing $-2\ell_{n,m}+2d_m$ (twice the negative maximized log-likelihood plus twice the number of parameters $d_m$), and the Bayesian information criterion (BIC), which replaces the penalty $2d_m$ by $d_m\log n$. Their strengths are famously complementary and, as Yang (2005) proved in a foundational impossibility result, irreconcilable in general: AIC is asymptotically efficient (its prediction loss is asymptotically as small as the best candidate's) in non-parametric settings where no finite-dimensional candidate is true, but inconsistent when a finite-dimensional true model is among the candidates; BIC is consistent in that parametric case (it selects the true model with probability tending to one) but inefficient in non-parametric settings; and no selection criterion can achieve both when candidate dimensions are fixed.

The bridge criterion (BC) of Ding, Tarokh, and Yang (2018) escapes this dilemma in a restricted but important setting: autoregressive (AR) order selection, where the candidate dimension grows with the sample size $n$. The BC minimizes $-2\ell_{n,m}+c_n(1+2^{-1}+\dots+d_m^{-1})$ with default $c_n=n^{2/3}$, searching only over models of dimension at most the AIC-selected dimension; the harmonic penalty interpolates between the AIC and BIC penalty levels, motivated by an asymptotic result that the one-step underfitting loss of an AR approximation decays like $\chi^2_1/d$ in model dimension $d$. The BC is provably selection-consistent in parametric frameworks and asymptotically efficient in both parametric and non-parametric frameworks — but the proofs are asymptotic, tied to AR order selection (with a companion analysis for linear regression in an unpublished manuscript cited by the chapter), and say little about finite-sample behavior. A related diagnostic, the parametricness index of Liu and Yang (2011), uses the distances among the dimensions selected by BC, AIC, and BIC to estimate whether the data-generating mechanism is well approximated by a finite-dimensional candidate.

A separate thread concerns cross-validation (CV), in which data are split into a training set (used to fit each candidate) and a validation set (used to score it); for selecting among modeling procedures — entire algorithms or selection rules, not just among nested parametric models — Yang (2007) and Zhang and Yang (2015) studied when CV is consistent, i.e., when the probability that CV picks the genuinely better procedure tends to one. Zhang and Yang (2015) showed that consistency requires the validation size $n_v$ to diverge faster than the training size $n_t$ (the ratio $n_v/n_t\to\infty$), so that common schemes — leave-one-out CV, $K$-fold CV, or holdout splits at a fixed ratio such as 70/30 — may be inconsistent for procedure selection; the chapter terms the resulting phenomenon (better estimation of prediction error does not imply better procedure selection) the cross-validation paradox, and illustrates it on a gene-expression dataset where growing both sets at a fixed ratio decreased selection accuracy. Finally, the chapter stresses that a selected model on which one wishes to base scientific inference should be stable — reproducible under resampling or perturbation of the data — yet no quantitative theory links such model stability to selection consistency.

The chapter's closing outlook states that a unified perspective integrating these fundamental limits of inference and prediction remains an active research area, and names the three gaps above as the specific open directions.

Problem Statement

Develop the missing quantitative theory that integrates the fundamental limits of inference and prediction in model selection, as identified in the closing outlook of Ding, Yang, and Tarokh (2021). The problem comprises three jointly supporting open directions:

(1) A stability–consistency relation. Fix a formal notion of model-selection stability (the reproducibility of the selected model under resampling or perturbation of the data — the source's informal notion) and establish a quantitative relation between it and selection consistency: for example, bounds or equivalences showing how the degree of stability of a selection procedure constrains, or is constrained by, the probability that it selects the true model in a parametric framework (or a prediction-optimal model in a non-parametric framework), at finite sample sizes. The stability notion must be defined solely from the observable behavior of the selection procedure under a non-degenerate resampling or perturbation scheme — one under which the stability measure does not reduce by construction to the probability of correct selection — and without reference to the true model or to the event of selecting it, so that the quantitative relation is a substantive constraint rather than a consequence of the definition itself.

(2) Finite-sample optimality and extended scope of adaptive criteria. Identify the conditions — on sample size, candidate dimension, signal strength, and error distribution — under which the bridge criterion (or related adaptive criteria whose penalty interpolates between the AIC and BIC levels) achieves provably optimal performance at finite samples, where the sense of optimality (for example an oracle or minimax sense relative to the class of candidate models) is stated explicitly and is not defined so as to make the optimality claim automatic; and extend the bridge criterion's asymptotic guarantees (selection consistency in parametric frameworks, asymptotic efficiency in both parametric and non-parametric frameworks), currently established for autoregressive order selection, to the more complex settings of high-dimensional regression, non-linear models, and dependent data.

(3) A splitting-ratio theory for procedure selection. Quantify how the train/validation splitting ratio affects the probability that cross-validation correctly identifies the better of two modeling procedures at finite sample sizes — sharpening the known asymptotic necessary condition that the validation size must dominate the training size ($n_v/n_t\to\infty$) into finite-sample guidance, including the practically relevant regime of limited samples.

The common scientific objective is a single quantitative account of the inference–prediction trade-off in model selection; direction (1) supplies its stability ingredient, (2) its optimality ingredient, and (3) its validation ingredient. The three directions may be advanced separately: a resolution of the task requires quantitative results for all three, while a complete result for a proper subset of them is partial progress rather than a resolution. The settings listed in direction (2) describe the scope of that direction, so results for a representative subset of those settings suffice. Any resulting theory must remain compatible with the known asymptotic impossibility results for fixed candidate dimensions.

References

1. Jie Ding, Yuhong Yang, and Vahid Tarokh, 'Fundamental Limits in Model Selection for Modern Data Analysis', in Information-Theoretic Methods in Data Science, Cambridge University Press, 2021, pp. 359–382. DOI: 10.1017/9781108616799.013, https://doi.org/10.1017/9781108616799.013 (source chapter)
2. Jie Ding, Vahid Tarokh, and Yuhong Yang, 'Bridging AIC and BIC: A New Criterion for Autoregression', IEEE Transactions on Information Theory 64(6), 2018, pp. 4024–4043. DOI: 10.1109/TIT.2017.2717599, https://doi.org/10.1109/TIT.2017.2717599
3. Yongli Zhang and Yuhong Yang, 'Cross-validation for selecting a model selection procedure', Journal of Econometrics 187(1), 2015, pp. 95–112. DOI: 10.1016/j.jeconom.2015.02.006, https://doi.org/10.1016/j.jeconom.2015.02.006
4. Yuhong Yang, 'Can the strengths of AIC and BIC be shared? A conflict between model identification and regression estimation', Biometrika 92(4), 2005, pp. 937–950. DOI: 10.1093/biomet/92.4.937, https://doi.org/10.1093/biomet/92.4.937
5. Yuhong Yang, 'Consistency of cross validation for comparing regression procedures', The Annals of Statistics 35(6), 2007, pp. 2450–2473. DOI: 10.1214/009053607000000514, https://doi.org/10.1214/009053607000000514
6. Wei Liu and Yuhong Yang, 'Parametric or nonparametric? A parametricness index for model selection', The Annals of Statistics 39(4), 2011, pp. 2074–2102. DOI: 10.1214/11-AOS899, https://doi.org/10.1214/11-AOS899
7. Jie Ding, Mohammad Noshad, and Vahid Tarokh, 'Order Selection of Autoregressive Processes Using Bridge Criterion', 2015 IEEE International Conference on Data Mining Workshop (ICDMW), 2015, pp. 615–622. DOI: 10.1109/ICDMW.2015.216, https://doi.org/10.1109/ICDMW.2015.216
8. Jie Ding, Enmao Diao, Jiawei Zhou, and Vahid Tarokh, 'On Statistical Efficiency in Learning', IEEE Transactions on Information Theory 67(4), 2021, pp. 2488–2506. DOI: 10.1109/TIT.2020.3047620, arXiv:2012.13307, https://arxiv.org/abs/2012.13307
9. Jiawei Zhang, Yuhong Yang, and Jie Ding, 'Information criteria for model selection', WIREs Computational Statistics 15(5), 2023. DOI: 10.1002/wics.1607, https://doi.org/10.1002/wics.1607
10. V. Roshan Joseph, 'Optimal ratio for data splitting', Statistical Analysis and Data Mining: The ASA Data Science Journal 15(4), 2022, pp. 531–538. DOI: 10.1002/sam.11583, arXiv:2202.03326, https://arxiv.org/abs/2202.03326
11. Kenichiro McAlinn and Kōsaku Takanashi, 'Optimal Hold-Out Size in Cross-Validation', 2025. arXiv:2511.12698, https://arxiv.org/abs/2511.12698
12. Ching-Kang Ing, 'Model selection for high-dimensional linear regression with dependent observations', The Annals of Statistics 48(4), 2020, pp. 1959–1980. DOI: 10.1214/19-aos1872, https://doi.org/10.1214/19-aos1872
13. William Kengne, 'On consistency for time series model selection', Statistical Inference for Stochastic Processes 26(2), 2023, pp. 437–458. DOI: 10.1007/s11203-022-09284-6, https://doi.org/10.1007/s11203-022-09284-6
14. Chor-yiu Sin and Shu-Hui Yu, 'Order selection for possibly infinite-order non-stationary time series', AStA Advances in Statistical Analysis 103(2), 2019, pp. 187–216. DOI: 10.1007/s10182-018-00333-1, https://doi.org/10.1007/s10182-018-00333-1
15. Wei Sun, Junhui Wang, and Yixin Fang, 'Consistent Selection of Tuning Parameters via Variable Selection Stability', Journal of Machine Learning Research 14, 2013, pp. 3419–3440. https://www.jmlr.org/papers/v14/sun13b.html
16. Yang Feng, Tengfei Li, and Zhiliang Ying, 'Likelihood adaptively modified penalties', Applied Stochastic Models in Business and Industry 35(2), 2018, pp. 330–353. DOI: 10.1002/asmb.2322, https://doi.org/10.1002/asmb.2322
17. Adrien Saumard and Fabien Navarro, 'Finite sample improvement of Akaike's Information Criterion', 2018. arXiv:1803.02078, https://arxiv.org/abs/1803.02078
18. Pratik Patil, Yuchen Wu, and Ryan J. Tibshirani, 'Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent', 2024. arXiv:2402.16793, https://arxiv.org/abs/2402.16793
19. George I. Austin, Itsik Pe'er, and Tal Korem, 'Distributional bias compromises leave-one-out cross-validation', Science Advances 11(48), 2025. DOI: 10.1126/sciadv.adx6976, arXiv:2406.01652, https://arxiv.org/abs/2406.01652
20. Kaizhi Yu and Tielai Tao, 'Consistent Model Selection Procedure for Random Coefficient INAR Models', Entropy 25(8), 2023, article 1220. DOI: 10.3390/e25081220, https://doi.org/10.3390/e25081220
21. Kare Kamila, 'Data-driven model selection for same-realization predictions in autoregressive processes', Annals of the Institute of Statistical Mathematics 75(4), 2023, pp. 567–592. DOI: 10.1007/s10463-022-00855-1, https://doi.org/10.1007/s10463-022-00855-1
22. Michael C. Chung, Alen Zacharia, and Juan Guan, 'An information-based model selection criterion for data-driven model discovery', 2026. arXiv:2602.21393, https://arxiv.org/abs/2602.21393