Curth, Jeffares, and van der Schaar (NeurIPS 2023) showed that the apparent double descent of trees, boosting, and random-Fourier-feature linear regression arises from silently transitioning between two entangled complexity axes — for example, supervised principal components versus unsupervised excess features — and that plotting a generalized, test-time effective number of parameters folds these curves back into classical U-shapes. They left open whether this mechanism-level account extends to deep learning, and whether the label-free effective-parameter criterion supports reliable model selection. Later work partially answered the first question: the same authors extended the metric to neural networks through telescoping first-order approximations of SGD training and found that test-time effective parameters shrink past the interpolation threshold in width-wise deep double descent; independent measures (random-X degrees of freedom, gradient-alignment complexity) now both rival and partly contest the diagnostic, giving opposite verdicts on the same experiments. This record asks for a general account — empirical and/or theoretical — of deep double descent in terms of effective complexity across architectures and training regimes, together with systematic validation of label-free effective-parameter model selection.

Background

Classical statistical learning theory predicts a U-shaped relationship between model complexity and test error: as complexity grows, error first falls (reducing bias) and then rises (growing variance). Double descent is the empirically observed deviation in which, when complexity is measured by the number of parameters pp and grows past the sample size nn, test error rises to a peak near the interpolation threshold — the smallest complexity at which the training set is fit exactly — and then descends a second time. Belkin, Hsu, Ma, and Mandal (2019) demonstrated the phenomenon across classical methods (random Fourier features, trees, boosting, random forests), and Nakkiran, Kaplun, Bansal, Yang, Barak, and Sutskever (2019) established it as a pervasive property of deep neural networks, where it also appears as a function of training time and label noise.

A smoother is a predictor whose prediction at any input is a fixed linear combination of the training targets, written f^(x0)=∑is^i(x0) yi\hat{f}(x_{0})=\sum_{i}\hat{s}_{i}(x_{0})\,y_{i} with smoothing weights s^(x0)\hat{\mathbf{s}}(x_{0}); linear regression, kernel regression, and nearest-neighbor methods are all smoothers. For smoothers, classical statistics measures complexity by the effective number of parameters pe=∑i∥s^(xi)∥2p_{e}=\sum_{i}\|\hat{\mathbf{s}}(x_{i})\|^{2}, which equals the raw parameter count for ordinary linear regression. Curth, Jeffares, and van der Schaar (2023) generalized this to any set of inputs I0\mathcal{I}_{0}:

ps^0=n∣I0∣∑j∈I0∥s^(xj0)∥2p^{0}_{\hat{\mathbf{s}}}=\frac{n}{|\mathcal{I}_{0}|}\sum_{j\in\mathcal{I}_{0}}\|\hat{\mathbf{s}}(x^{0}_{j})\|^{2}

so that the train-time value ps^trainp^{\text{train}}_{\hat{\mathbf{s}}} recovers pep_{e}, while the test-time value ps^testp^{\text{test}}_{\hat{\mathbf{s}}} measures the smoothing the fitted model applies on unseen inputs (for kk-nearest-neighbor regression it equals n/kn/k).

Using this lens, they showed that the double-descent experiments for trees, boosting, and random-Fourier-feature (RFF) regression — in which inputs are mapped through random sinusoidal features and linear regression is fit by principal-components regression (PCR), i.e. on the top principal components of the feature matrix — conflate two distinct mechanisms of increasing the raw parameter count: a supervised axis (the number of principal components PPCP^{PC} actually fit, or the number of leaves per tree) and an unsupervised axis (the number of excess features Pex=Pϕ−PPCP^{ex}=P^{\phi}-P^{PC} that only enrich the basis, or the number of boosting stages). Each axis alone exhibits the classical U-shape in test error; the apparent second descent appears exactly where an experiment silently switches from one mechanism to the other, so its location is not intrinsically tied to the interpolation threshold p=np=n. Plotting test error against ps^testp^{\text{test}}_{\hat{\mathbf{s}}} folds the apparent double-descent curves back into convex, traditional shapes. Related earlier results point in the same direction: Hastie, Montanari, Rosset, and Tibshirani (2022) derived exact high-dimensional risk curves for ridgeless (minimum-norm) linear regression; Poggio, Kur, and Banburski (2019) observed that the condition number of a random feature matrix — the ratio of its largest to smallest singular value, the standard measure of how sensitive a linear system is to noise — peaks at square aspect ratio, tracing the double-descent peak; Maddox, Benton, and Wilson (2020) proposed the effective dimensionality of learned weight matrices as a replacement for parameter counting in deep models; and Nakkiran et al. (2019) introduced a train-time effective model complexity. What none of these provides is a test-time, effective-parameter account of double descent in deep learning.

The source paper deliberately restricted itself to non-deep methods, and its conclusion poses the natural next questions: whether neural networks also possess multiple implicitly entangled complexity axes; whether the linear-regression/min-norm insights, combined with known connections between random-feature models and two-layer networks and between stochastic gradient descent and min-norm solutions, yield a mechanism-switching explanation of deep double descent; and whether the label-free, test-time effective-parameter criterion can be systematized into a reliable model-selection method. Separately, the paper shows — only empirically, in an appendix — that unsupervised basis enrichment improves the conditioning of the regression basis; the source presents this as supporting evidence rather than as an open question, and this record retains it only as background. This record treats the two future-work questions as one problem: establishing whether the effective-parameter resolution of double descent is a general property of statistical learners or specific to the non-deep smoothers studied.

Problem Statement

The problem is to determine whether the effective-parameter account of double descent — complexity measured by what a trained model actually uses on unseen inputs, rather than by raw parameter counts — generalizes beyond the non-deep smoothers studied by Curth, Jeffares, and van der Schaar (2023). The overarching objective is to settle whether their resolution of double descent is a general property of statistical learners or an artifact of the particular methods studied, through two jointly supporting deliverables.

(1) Deep-learning diagnostic and theory. Identify and measure the complexity axes that a trained neural network actually uses on unseen inputs, and determine whether these diagnostics account for deep double descent and benign overfitting (the phenomenon in which models that interpolate the training data nonetheless generalize well) across width and depth sweeps, architectures, loss functions, and training regimes. The theoretical part of this deliverable is whether a mechanism-switching explanation of the kind established for trees, boosting, and RFF regression provably arises in neural-network settings.

(2) Label-free model selection. Assess systematically whether a test-time effective-parameter criterion can be used for model selection without held-out labels: does selecting among zero-training-error (or otherwise comparable) models by minimizing ps^testp^{\text{test}}_{\hat{\mathbf{s}}} reliably predict generalization across datasets, model classes, and hyperparameter grids, and what are the statistical properties, robustness, and failure modes of such a criterion?

These two deliverables bear jointly on a single question: is the effective-parameter resolution of double descent a general property of statistical learners, or is it confined to the non-deep smoothers studied by Curth, Jeffares, and van der Schaar (2023) and liable to fail or mislead in identifiable deep-learning regimes?

References

  1. Alicia Curth, Alan Jeffares, Mihaela van der Schaar. A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). arXiv:2310.18988, DOI: 10.48550/arXiv.2310.18988. https://arxiv.org/abs/2310.18988
  2. Alan Jeffares, Alicia Curth, Mihaela van der Schaar. Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2411.00247. https://arxiv.org/abs/2411.00247
  3. Pratik Patil, Jin-Hong Du, Ryan J. Tibshirani. Revisiting optimism and model complexity in the wake of overparameterized machine learning. 2024. arXiv:2410.01259. https://arxiv.org/abs/2410.01259
  4. Oskar Allerbo, Thomas B. Schön. A Rigorous, Tractable Measure of Model Complexity. 2026. arXiv:2605.21167. https://arxiv.org/abs/2605.21167
  5. Oskar Allerbo, Thomas B. Schön. Is Supervised Learning Really That Different from Unsupervised? (accepted at AISTATS 2026). 2025. arXiv:2505.11006. https://arxiv.org/abs/2505.11006
  6. Daniel Gedon, Antonio H. Ribeiro, Thomas B. Schön. No Double Descent in Principal Component Regression: A High-Dimensional Analysis. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:15271-15293, 2024. https://proceedings.mlr.press/v235/gedon24a.html
  7. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever. Deep Double Descent: Where Bigger Models and More Data Hurt. 2019. arXiv:1912.02292. https://arxiv.org/abs/1912.02292
  8. Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences 116(32):15849-15854, 2019. DOI: 10.1073/pnas.1903070116. https://doi.org/10.1073/pnas.1903070116
  9. Wesley J. Maddox, Gregory Benton, Andrew Gordon Wilson. Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited. 2020. arXiv:2003.02139. https://arxiv.org/abs/2003.02139
  10. Trevor Hastie, Andrea Montanari, Saharon Rosset, Ryan J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics 50(2), 2022. DOI: 10.1214/21-AOS2133. https://doi.org/10.1214/21-AOS2133
  11. Tomaso Poggio, Gil Kur, Andrzej Banburski. Double descent in the condition number. 2019. arXiv:1912.06190. https://arxiv.org/abs/1912.06190
  12. Alicia Curth, Alan Jeffares, Mihaela van der Schaar. Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers. 2024. arXiv:2402.01502. https://arxiv.org/abs/2402.01502
  13. Amanda Olmin, Fredrik Lindsten. Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks. 2024. arXiv:2407.09845. https://arxiv.org/abs/2407.09845
  14. Rylan Schaeffer, Mikail Khona, Zachary Robertson, Akhilan Boopathy, Kateryna Pistunova, Jason W. Rocks, Ila Rani Fiete, Oluwasanmi Koyejo. Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle. 2023. arXiv:2303.14151. https://arxiv.org/abs/2303.14151
  15. Alan Jeffares, Mihaela van der Schaar. Not All Explanations for Deep Learning Phenomena Are Equally Valuable. (ICML 2025). 2025. arXiv:2506.23286. https://arxiv.org/abs/2506.23286