Non-asymptotic theory of optimal cross-validation data-split ratios beyond simple models: finite-sample characterization, conflicting objectives, and LOO-versus-large-validation regimes

When a fixed dataset of N samples must be divided among training, validation, and test roles, how large a fraction should each part receive? Larsen and Goutte (1999) derived exact non-asymptotic answers only for a Gaussian location-parameter model: for the design/test split under hold-out cross-validation the optimal test fraction $\gamma$ approaches 1 with $1-\gamma = O(N^{-1/3})$, while for K-fold and permutation cross-validation the mean-squared error of the generalization-error estimate is minimized at leave-one-out; they also showed, in the same tractable model, that the objectives of accurate error estimation, best achieved generalization, and consistent model selection demand conflicting split ratios, with sharp phase transitions between regimes. The general question they left open remains open: a finite-sample (non-asymptotic) characterization of optimal split ratios for realistic high-dimensional or nonlinear models, a principled formalization of the trade-off among the three conflicting objectives, and a map of the regimes in which leave-one-out or large validation fractions are optimal. Later literature has resolved only special cases - optimal training size $n/2$ under a variance-based criterion (Afendras and Markatou, 2019), the $\sqrt{p}:1$ train/test ratio for well-specified linear regression (Joseph, 2022), random-matrix test-set sizing for linear and ridge regression (Dubbs, 2024, 2025), and a split proportional to the square root of the matrix dimension for covariance estimation (Lamrani et al., 2025) - while 2025 work still describes the underlying finite-sample split theory as not well understood.

Background

Cross-validation (CV) is the standard tool for estimating the generalization error of a fitted model - its expected loss on a future independent sample - and for selecting among candidate models, whenever data are too scarce for analytic error estimates to be trusted. The data must first be split. In hold-out cross-validation (HO) the $N$ samples are split once into a design set of size $(1-\gamma)N$ and a test set of size $\gamma N$, where $\gamma$ is the split ratio; the model is fit on the design set and its error is estimated on the test set. In K-fold cross-validation (KCV) the data are partitioned into $K$ disjoint subsets, each subset in turn serving as the test set for a model fit on the remaining data, and the $K$ error estimates are averaged; leave-one-out (LOO) is the special case $K=N$. Permutation (or Monte-Carlo) cross-validation (PCV) repeats the two-set split many times at random and averages. A further, nested distinction matters: the design/test split serves final error assessment, while the training/validation split of the design set serves model and hyperparameter selection.

The quality of a split ratio can be measured by at least three distinct objectives: (i) estimation accuracy, e.g. the mean squared error (MSE) of the CV error estimate around the minimum achievable error; (ii) the generalization error of the model ultimately selected or trained; and (iii) selection consistency - the probability that the correct model is selected tending to one. These objectives conflict. For estimating the error of a pattern classifier, Highleyman (1962) already sought the variance-minimizing partition; Kearns (1997) derived bounds on CV error with direct consequences for the training-test split; Guyon, Makhoul, Schwartz, and Vapnik (1998) asked what test-set size yields good error-rate estimates; Blum, Kalai, and Langford (1999) proved that K-fold and progressive CV estimates beat the hold-out. On the consistency side, Shao (1993) proved that for variable selection in linear regression, LOO is inconsistent and leave-$n_v$-out CV is consistent only if $n_v$ grows to infinity with $n_v/N$ approaching 1 - that is, consistency demands reserving asymptotically most of the data for validation. Larsen and Goutte (1999) made the conflict concrete and exact in the one model tractable at finite $N$, a Gaussian location-parameter model (estimating the mean of a normal distribution): there, the MSE of the HO estimate is minimized at $\gamma$ approaching 1 with $1-\gamma = O(N^{-1/3})$ (so most data should asymptotically be reserved for testing), KCV/PCV MSE is minimized at LOO for every $N$, and under model selection the optimal $\gamma$ undergoes phase transitions - toward small validation fractions when best generalization is the goal, toward $\gamma$ near 1 when consistency is the goal, with the transition threshold scaling as $\sigma/\sqrt{N}$, where $\sigma$ is the noise level. Simulation on autoregressive (AR) filters - models whose output is a weighted sum of their own past outputs plus noise - showed qualitatively similar behavior; a preliminary experiment fitting a neural network to the Hénon map, a standard chaotic dynamical system used as a nonlinear time-series benchmark, was summarized, with details deferred to the workshop and never archived. Theoretical study remained restricted to simple models or asymptotics, which is exactly the gap identified by Larsen and Goutte (1999). Subsequent work has chipped away at special model classes - the optimal training size $n/2$ for smooth losses in Efron's q-class (a family of convex loss functions, including squared error, satisfying a curvature condition introduced by Efron) under a resampling-effectiveness criterion (Afendras and Markatou, 2019); the $\sqrt{p}:1$ train/test ratio, with $p$ the number of predictors, for a well-specified linear model with homoscedastic (constant-variance) noise under random subsampling (Joseph, 2022), whose companion work addresses which rows to allocate rather than how many (Joseph and Vakayil, 2022); random-matrix-theory test-set sizing for Gaussian linear regression, where the optimal training size is the root of a quartic in the sample size and dimension (Dubbs, 2024), extended to ridge regression (Dubbs, 2025); and a closed-form holdout error for covariance-matrix estimation, with optimal split proportional to the square root of the matrix dimension, under a white inverse-Wishart ensemble - the inverse-Wishart being a standard probability distribution on covariance matrices (Lamrani, Bongiorno, and Potters, 2025). Related splitting-ratio questions are being solved domain-by-domain elsewhere - calibration-set sizing in split conformal prediction, a distribution-free method for constructing prediction intervals whose calibration set plays the role of a validation set (Das, Yaghooti, Kuffner, and Lahiri, 2026), and the splitting ratio of the split likelihood ratio test, a universal hypothesis test that fits the null model on one part of the data and evaluates the likelihood ratio on the other (Strieder and Drton, 2022) - while the survey of Arlot and Celisse (2010) documents how fold-count and split choices remain largely heuristic, and Bates, Hastie, and Tibshirani (2024) clarify what CV actually estimates under model selection. What still does not exist is a general finite-sample theory covering realistic nonlinear or high-dimensional models and reconciling the conflicting objectives.

Problem Statement

Establish a general non-asymptotic (finite-sample) theory of optimal data-split ratios for cross-validation-based generalization estimation and model selection, in the setting of Larsen and Goutte (1999): a fixed dataset of $N$ independent samples is split with ratio $\gamma$ among design and test (and, within the design set, training and validation) roles, and the split may be judged by any of three objectives - the MSE of the cross-validation error estimate around the minimum achievable error, the generalization error of the model the procedure ultimately selects, or the probability of consistent model selection. The theory should deliver, with proofs valid at finite $N$ rather than only asymptotically: (1) a characterization of the optimal split ratio $\gamma^*$ as a function of sample size, model class, and objective for realistic model classes beyond the tractable location-parameter case - for example, high-dimensional regularized regression beyond the already-solved linear and ridge cases, general nonlinear regression, or neural networks under a pinned training protocol (architecture, optimizer, and stopping rule held fixed while the split ratio varies); (2) a principled formalization and resolution of the demonstrated conflict among the three objectives - for instance, a canonical compound objective with optimality guarantees, or a proof that no single split ratio can jointly optimize them, together with a characterization of the resulting Pareto frontier of split ratios; and (3) a determination of the regimes - in terms of model class, task, and noise level - in which leave-one-out (small validation fractions) versus large validation or test fractions are optimal for K-fold and permutation cross-validation, extending beyond the empirical examples (autoregressive filters, Hénon-map network) available in Larsen and Goutte (1999). These three components jointly support one objective: replacing heuristic split conventions with model-dependent, finite-sample-optimal prescriptions, extending the open questions in Larsen and Goutte (1999). Answers restricted to a new special model class are admissible as progress but do not by themselves complete the problem unless the model class is broad enough, or the technique general enough, that the remaining cases follow the same pattern.

References

1. Larsen, J., Goutte, C. (1999). On optimal data split for generalization estimation and model selection. In: Proceedings of the IEEE Workshop on Neural Networks for Signal Processing IX (NNSP-99), pp. 225-234. DOI: 10.1109/NNSP.1999.788141. https://doi.org/10.1109/NNSP.1999.788141 (open-access copy at https://backend.orbit.dtu.dk/ws/files/5381207/Larsen.pdf)
2. Highleyman, W. H. (1962). The design and analysis of pattern recognition experiments. Bell System Technical Journal 41(2):723-744. DOI: 10.1002/j.1538-7305.1962.tb02426.x. https://doi.org/10.1002/j.1538-7305.1962.tb02426.x
3. Kearns, M. (1997). A bound on the error of cross validation using the approximation and estimation rates, with consequences for the training-test split. Neural Computation 9(5):1143-1161. DOI: 10.1162/neco.1997.9.5.1143. https://doi.org/10.1162/neco.1997.9.5.1143
4. Guyon, I., Makhoul, J., Schwartz, R., Vapnik, V. (1998). What size test set gives good error rate estimates? IEEE Transactions on Pattern Analysis and Machine Intelligence 20(1):52-64. DOI: 10.1109/34.655649. https://doi.org/10.1109/34.655649
5. Shao, J. (1993). Linear model selection by cross-validation. Journal of the American Statistical Association 88(422):486-494. DOI: 10.1080/01621459.1993.10476299. https://doi.org/10.1080/01621459.1993.10476299
6. Blum, A., Kalai, A., Langford, J. (1999). Beating the hold-out: bounds for k-fold and progressive cross-validation. In: Proceedings of the Twelfth Annual Conference on Computational Learning Theory (COLT '99), pp. 203-208. DOI: 10.1145/307400.307439. https://doi.org/10.1145/307400.307439
7. Burman, P. (1989). A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods. Biometrika 76(3):503-514. DOI: 10.1093/biomet/76.3.503. https://doi.org/10.1093/biomet/76.3.503
8. Arlot, S., Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys 4:40-79. DOI: 10.1214/09-SS054. https://doi.org/10.1214/09-SS054
9. Yang, Y. (2007). Consistency of cross validation for comparing regression procedures. The Annals of Statistics 35(6):2450-2473. DOI: 10.1214/009053607000000514. https://doi.org/10.1214/009053607000000514
10. Afendras, G., Markatou, M. (2019). Optimality of training/test size and resampling effectiveness in cross-validation. Journal of Statistical Planning and Inference 199:286-301. DOI: 10.1016/j.jspi.2018.07.005. https://doi.org/10.1016/j.jspi.2018.07.005 (preprint: arXiv:1511.02980, https://arxiv.org/abs/1511.02980, posted under the title "Optimality of Training/Test Size and Resampling Effectiveness of Cross-Validation Estimators of the Generalization Error")
11. Joseph, V. R. (2022). Optimal ratio for data splitting. Statistical Analysis and Data Mining: The ASA Data Science Journal 15(4):531-538. DOI: 10.1002/sam.11583. https://doi.org/10.1002/sam.11583 (preprint: arXiv:2202.03326, https://arxiv.org/abs/2202.03326)
12. Joseph, V. R., Vakayil, A. (2022). SPlit: an optimal method for data splitting. Technometrics 64(2):166-176. DOI: 10.1080/00401706.2021.1921037. https://doi.org/10.1080/00401706.2021.1921037
13. Dubbs, A. (2024). Test set sizing via random matrix theory. Operations Research Forum 5(1), Article 17. DOI: 10.1007/s43069-024-00292-1. https://doi.org/10.1007/s43069-024-00292-1 (preprint: arXiv:2112.05977, https://arxiv.org/abs/2112.05977)
14. Dubbs, A. (2025). Test set sizing for the ridge regression. arXiv:2504.19231. https://arxiv.org/abs/2504.19231
15. Lamrani, L., Bongiorno, C., Potters, M. (2025). Optimal data splitting for holdout cross-validation in large covariance matrix estimation. arXiv:2503.15186. https://arxiv.org/abs/2503.15186
16. Bates, S., Hastie, T., Tibshirani, R. (2024). Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association 119(546):1434-1445. DOI: 10.1080/01621459.2023.2197686. https://doi.org/10.1080/01621459.2023.2197686 (online first 2023)
17. Strieder, D., Drton, M. (2022). On the choice of the splitting ratio for the split likelihood ratio test. Electronic Journal of Statistics 16(2). DOI: 10.1214/22-EJS2099. https://doi.org/10.1214/22-EJS2099
18. Das, S., Yaghooti, B., Kuffner, T. A., Lahiri, S. N. (2026). On optimal data splitting for split conformal prediction. arXiv:2606.31600. https://arxiv.org/abs/2606.31600