A rigorous quantitative theory of the three-factor mechanism for test-error divergence at the interpolation threshold

The tutorial by Schaeffer et al. (2023) explains double descent in ordinary linear regression through three simultaneously extreme factors — small-but-nonzero singular values of the training features near the interpolation threshold, large residual-error projections along those singular modes, and test features varying along the same modes — but deliberately leaves the mechanism at the level of geometric intuition. The open problem is to convert this explanation into a rigorous quantitative theory: (1) non-asymptotic, finite-sample probabilistic characterizations of the smallest non-zero singular value of realistic (non-Gaussian, structured) feature matrices across the interpolation threshold; (2) precise conditions under which the mechanism does or does not produce double descent in finite-width, finite-depth neural networks trained with stochastic gradient descent (SGD) and feature learning — where the learned representation adapts during training — beyond the infinite-width kernel limit; and (3) a characterization of when one, two, or many test-error descents occur. Precise asymptotic theories exist for stylized linear and random-features models, and constructions show that arbitrarily many descents are possible in specific families, but no existing result supplies the required finite-sample singular-value laws, finite-width conditions, or general multiple-descent characterization.

Background

In supervised regression, the test (out-of-sample) error of a learned model classically decreases and then increases with model complexity — the bias-variance trade-off. Modern overparameterized models violate this: as the number of fit parameters $P$ grows relative to the number of training data $N$, test error often peaks sharply and then falls again, a phenomenon called double descent. The peak occurs near the interpolation threshold, the regime $P \approx N$ where the model just barely acquires enough capacity to fit the training data exactly (zero training error). For linear regression trained by gradient descent, the relevant estimator is the minimum-norm least-squares interpolant: the interpolating solution of smallest parameter norm, which gradient descent converges to implicitly.

The singular value decomposition (SVD) writes the $N \times P$ matrix $X$ of training features as $X = U\Sigma V^{\top}$, where the singular values on the diagonal of $\Sigma$ measure how much variance the training data carry along each orthogonal direction of feature space. For matrices with independent identically distributed entries, the Marchenko–Pastur distribution from random matrix theory describes the limiting shape of the singular-value spectrum; its lower edge approaches zero as $N$ approaches $P$, which is why the smallest non-zero singular value is expected to be most extreme exactly at the interpolation threshold.

Schaeffer, Khona, Robertson, Boopathy, Pistunova, Rocks, Fiete and Koyejo (2023) gave a widely read tutorial-level account of double descent in ordinary linear regression. Writing the test prediction error as a sum over singular modes, they identified three multiplicative factors that, when simultaneously extreme, create double descent: (i) the training features contain small-but-nonzero variance in some singular direction; (ii) from the perspective of the model class, the residual errors of the best possible model have large projection along that mode; and (iii) the test features vary significantly along that mode. They verified the mechanism causally: ablating any one of the three factors in linear regression on synthetic and real datasets removes double descent. However, they deliberately avoided the technical random-matrix-theory derivation behind factor (i), restricted the mathematical analysis to linear regression, and extended the intuition to deep networks only via infinite-width limits — regimes in which network training reduces exactly to linear or kernel regression on fixed features (the Neural Network Gaussian Process and Neural Tangent Kernel limits), so that no genuine feature learning — adaptation of the learned representation during training, as opposed to fitting on fixed features — occurs. They also noted, citing earlier empirical work, that different settings can exhibit one, three, or more descents, without characterizing when.

A substantial adjacent literature already provides rigor of a different kind. Precise asymptotic analyses of the minimum-norm interpolator in high-dimensional linear and random-features regression (Hastie, Montanari, Rosset and Tibshirani; Mei and Montanari; Rocks and Mehta; Poggio, Kur and Banburski on the condition number) characterize the double-descent risk curve exactly in the limit $N, P \to \infty$ at fixed ratio, using Marchenko–Pastur-type tools. Classical random-matrix results (e.g., Rudelson and Vershynin on the smallest singular value of random rectangular matrices) give finite-sample bounds for Gaussian or subgaussian entries. Non-asymptotic risk bounds for interpolating linear regression under general covariance spectra (Bartlett, Long, Lugosi and Tsigler's benign overfitting) delineate when interpolation is compatible with good prediction. None of these, however, gives a non-asymptotic law for the smallest singular values of realistic structured data matrices near rectangularity, treats finite-width networks outside the kernel limit with realistic optimization dynamics, or characterizes the number of descents across model families.

Problem Statement

Establish a rigorous quantitative version of the three-factor mechanism of Schaeffer et al. (2023) for test-error divergence at the interpolation threshold, replacing the tutorial's geometric intuition with precise mathematics. The objective comprises three linked deliverables, each quantifying one acknowledged limitation of the source:

1. Finite-sample singular-value law. Give a probabilistic characterization — exact finite-$N, P$ laws or tight non-asymptotic bounds — of the smallest non-zero singular value of training-feature matrices arising from realistic (non-Gaussian, structured, possibly heavy-tailed spectral) data, as the parameter-to-data ratio crosses the interpolation threshold, sufficient to quantify when factor (i) holds and with what magnitude. Existing results cover the Gaussian/subgaussian case and the $N, P \to \infty$ asymptotic regime; the target is the finite-sample, structured-data regime exemplified by the tutorial's real-data experiments. The class is not fixed to a single family: any non-Gaussian, structured matrix class is admissible, provided the result names that class concretely — an explicit generative model or a named dataset together with its feature map — so that factor (i) can be evaluated against it.

2. Finite-width extension. Derive explicit conditions — in terms of architecture, width and depth, initialization, optimizer and step sizes, and data distribution — under which the three-factor mechanism provably does, or provably does not, predict double descent in finite-width, finite-depth neural networks, including the effects of stochastic gradient descent (SGD) dynamics, nonlinearity, and feature learning (adaptation of the learned representation during training). Because the source defines the three factors via the singular value decomposition of a fixed training-feature matrix — or of fixed kernel features in the infinite-width limits — this component includes making precise which objects play the role of the singular modes once the learned representation is no longer fixed, and in what exact sense the mechanism does or does not predict the test-error divergence. Results stated only for linear regression, infinite-width kernel limits, or gradient-flow idealizations of SGD address parts of this component but not the stated target.

3. Multiple descents. Characterize when one, two, or more descents occur: which properties of the data spectrum, the model family, and the inductive bias of the learning algorithm — the preference among equally fitting solutions imposed by the algorithm and architecture — determine the number and location of test-error peaks. Constructions showing that an arbitrary number of peaks is achievable in specific families answer the possibility question but not the characterization question.

The three deliverables jointly serve one objective: a predictive, quantitative theory of when interpolation-driven test-error divergence occurs and how large it is, of which the tutorial's three factors are the informal skeleton. Each is an independent open front, and resolving any one of them alone does not resolve the objective.

References

1. Rylan Schaeffer, Mikail Khona, Zachary Robertson, Akhilan Boopathy, Kateryna Pistunova, Jason W. Rocks, Ila Rani Fiete, Oluwasanmi Koyejo. Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle. arXiv:2303.14151, DOI 10.48550/arXiv.2303.14151. https://arxiv.org/abs/2303.14151
2. Trevor Hastie, Andrea Montanari, Saharon Rosset, Ryan J. Tibshirani. Surprises in High-Dimensional Ridgeless Least Squares Interpolation. arXiv:1903.08560, DOI 10.48550/arXiv.1903.08560. https://arxiv.org/abs/1903.08560. Also The Annals of Statistics 50(2):949–986, 2022, DOI 10.1214/21-AOS2133
3. Song Mei, Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. arXiv:1908.05355, DOI 10.48550/arXiv.1908.05355. https://arxiv.org/abs/1908.05355. Also Communications on Pure and Applied Mathematics 75(4):667–766, 2022, DOI 10.1002/cpa.22008
4. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever. Deep Double Descent: Where Bigger Models and More Data Hurt. arXiv:1912.02292, DOI 10.48550/arXiv.1912.02292. https://arxiv.org/abs/1912.02292
5. Lin Chen, Yifei Min, Mikhail Belkin, Amin Karbasi. Multiple Descent: Design Your Own Generalization Curve. arXiv:2008.01036, DOI 10.48550/arXiv.2008.01036. https://arxiv.org/abs/2008.01036 (also NeurIPS 2021)
6. Tengyuan Liang, Alexander Rakhlin, Xiyu Zhai. On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels. arXiv:1908.10292, DOI 10.48550/arXiv.1908.10292. https://arxiv.org/abs/1908.10292 (also Proceedings of the 33rd Conference on Learning Theory (COLT 2020), PMLR 125:2683–2711)
7. Yue Li, Yuting Wei. Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent. arXiv:2110.09502, DOI 10.48550/arXiv.2110.09502. https://arxiv.org/abs/2110.09502
8. Theodor Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv:2204.10425, DOI 10.48550/arXiv.2204.10425. https://arxiv.org/abs/2204.10425
9. Xuran Meng, Jianfeng Yao, Yuan Cao. Multiple Descent in the Multiple Random Feature Model. arXiv:2208.09897, DOI 10.48550/arXiv.2208.09897. https://arxiv.org/abs/2208.09897
10. Antoine Bodin, Nicolas Macris. Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures. arXiv:2212.06757, DOI 10.48550/arXiv.2212.06757. https://arxiv.org/abs/2212.06757
11. Kevin Han Huang, Haoyu Ye, Somak Laha, Morgane Austern. The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression. arXiv:2607.24041, DOI 10.48550/arXiv.2607.24041. https://arxiv.org/abs/2607.24041
12. Mark K. Transtrum, Gus L. W. Hart, Tyler J. Jarvis, Jared P. Whitehead. eGAD! double descent is explained by Generalized Aliasing Decomposition. arXiv:2408.08294, DOI 10.48550/arXiv.2408.08294. https://arxiv.org/abs/2408.08294
13. Sidak Pal Singh, Aurelien Lucchi, Thomas Hofmann, Bernhard Schölkopf. Phenomenology of Double Descent in Finite-Width Neural Networks. arXiv:2203.07337, DOI 10.48550/arXiv.2203.07337. https://arxiv.org/abs/2203.07337 (arXiv comment: Published at ICLR 2022)
14. Chris Yuhao Liu, Jeffrey Flanigan. Understanding the Role of Optimization in Double Descent. arXiv:2312.03951, DOI 10.48550/arXiv.2312.03951. https://arxiv.org/abs/2312.03951
15. Fanghui Liu, Johan A. K. Suykens, Volkan Cevher. On the Double Descent of Random Features Models Trained with SGD. arXiv:2110.06910, DOI 10.48550/arXiv.2110.06910. https://arxiv.org/abs/2110.06910 (also NeurIPS 2022)
16. Hao Yu. Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenon. arXiv:2503.02129, DOI 10.48550/arXiv.2503.02129. https://arxiv.org/abs/2503.02129
17. Mark Rudelson, Roman Vershynin. The smallest singular value of a random rectangular matrix. arXiv:0802.3956, DOI 10.48550/arXiv.0802.3956. https://arxiv.org/abs/0802.3956. Also Communications on Pure and Applied Mathematics 62(12):1707–1739, 2009, DOI 10.1002/cpa.20294
18. Peter L. Bartlett, Philip M. Long, Gábor Lugosi, Alexander Tsigler. Benign overfitting in linear regression. arXiv:1906.11300, DOI 10.48550/arXiv.1906.11300. https://arxiv.org/abs/1906.11300. Also Proceedings of the National Academy of Sciences 117(48):30063–30070, 2020, DOI 10.1073/pnas.1907378117
19. Jason W. Rocks, Pankaj Mehta. Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models. arXiv:2010.13933, DOI 10.48550/arXiv.2010.13933. https://arxiv.org/abs/2010.13933. Also Physical Review Research 4:013201, 2022, DOI 10.1103/PhysRevResearch.4.013201
20. Tomaso Poggio, Gil Kur, Andrzej Banburski. Double descent in the condition number. arXiv:1912.06190, DOI 10.48550/arXiv.1912.06190. https://arxiv.org/abs/1912.06190
21. Tomas McKelvey. Analysis of Interpolating Regression Models and the Double Descent Phenomenon. arXiv:2304.08113, DOI 10.48550/arXiv.2304.08113. https://arxiv.org/abs/2304.08113 (also IFAC World Congress 2023)
22. Rylan Schaeffer, Zachary Robertson, Akhilan Boopathy, Mikail Khona, Ila Fiete, Andrey Gromov, Sanmi Koyejo. Divergence at the Interpolation Threshold: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle. NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning (M3L). https://openreview.net/forum?id=6V11PG7w5l.