Non-asymptotic and dynamical completion of the Hessian-based theory of double descent in finite-width neural networks

The influence-function analysis of Singh, Lucchi, Hofmann and Schölkopf (2022) — influence functions measure how infinitesimally perturbing the data shifts a fitted optimum — ties the double descent of finite-width neural networks, i.e. the peak and subsequent renewed descent of test error as the parameter count grows through the interpolation threshold (the point where the network just fits the training set exactly), to the smallest non-zero eigenvalue of the training-loss Hessian at the trained optimum: it proves an asymptotic divergence of their population-risk lower bound at the interpolation threshold for the mean-squared-error (MSE) loss, and it empirically discovers that for the cross-entropy loss the entire Hessian spectrum collapses to (near) zero near p≈n, the basis of their 'vanishing Hessian hypothesis' about stochastic gradient descent (SGD) dynamics. The paper itself leaves open the finite-sample and dynamical completion of this picture, which this task pursues through three technical cores: non-asymptotic finite-(n,p) versions of the Hessian-based bounds; a formal characterization of the vanishing-Hessian phenomenon for cross-entropy and of the training dynamics producing it; and conditions under which the optimal solution lies in the over- rather than under-parameterized regime. Two further side-questions raised by the source — the consequences for flatness-based generalization reasoning (the view that flatter minima generalize better) and the practical computability of the bounds — are out of scope for this task and serve as context only.

Background

In classical statistical learning, the test (population) error of a trained model first falls and then rises with model complexity, tracing the bias-variance trade-off curve. Modern over-parameterized deep networks violate this picture: as the number of parameters $p$ grows relative to the number of training samples $n$, test error follows the classical curve only until the model can exactly fit (interpolate) the training set — the interpolation threshold — where it peaks, and then descends again. This "double descent" curve was systematized empirically by Belkin, Hsu, Ma and Mandal (PNAS 2019) and Nakkiran, Kaplun, Bansal, Yang, Barak and Sutskever (2019), and its theory was developed almost entirely for linear and kernel regression — e.g., the exact asymptotic analysis of minimum-norm interpolation in random regression designs by Hastie, Montanari, Rosset and Tibshirani (2019) and the random-features analysis of Mei and Montanari (2019). Infinite-width neural networks can be pulled into this framework via the Neural Tangent Kernel (NTK) — the kernel induced by the network's Jacobian at initialization, in whose "lazy" regime training behaves like kernel regression — but this does not describe finite-width networks, where the Jacobian moves substantially during training (feature learning), and it disregards the effect of the loss function.

Singh, Lucchi, Hofmann and Schölkopf (2022) attacked the finite-width problem with influence functions — the classical robust-statistics tool that measures how infinitesimally perturbing the data shifts a fitted optimum. For a network with $p$ parameters trained to convergence on $n$ samples, they derived an exact first-order influence-function expression for the population risk, and a lower bound of the form

$\mathcal{L}(\boldsymbol{\theta}^{\star}) \,\geq\, \widetilde{\mathcal{L}}_{S}(\boldsymbol{\theta}^{\star}) \,+\, \frac{1}{n+1}\,\frac{\sigma^{2}_{\text{min}}\,\alpha\,\lambda_{\min}\!\left(\mathbf{C}^{\widetilde{D}}_{\bm{f}}(\boldsymbol{\theta}^{\star})\right)}{\lambda_{r}\!\left(\mathbf{H}^{S}_{\mathcal{L}}(\boldsymbol{\theta}^{\star})\right)},$

where $\boldsymbol{\theta}^{\star}$ are the trained parameters, $\mathbf{H}^{S}_{\mathcal{L}}(\boldsymbol{\theta}^{\star})$ is the Hessian (matrix of second derivatives) of the empirical training loss at the optimum, $\lambda_{r}$ denotes its smallest non-zero eigenvalue under the source's convention $\lambda_{1}\geq\cdots\geq\lambda_{r}$ with $r=\operatorname{rank}\!\left(\mathbf{H}^{S}_{\mathcal{L}}(\boldsymbol{\theta}^{\star})\right)$ — so that $\lambda_{r}\!\left(\mathbf{H}^{S}_{\mathcal{L}}(\boldsymbol{\theta}^{\star})\right)$ is written as $0$ when the Hessian collapses entirely, as in the source's Fact 6 for cross-entropy near $p\approx n$ — $\mathbf{C}_{\bm{f}}$ is the covariance of the network's Jacobians (gradients of outputs with respect to parameters), $\sigma^{2}_{\min}$ bounds from below the residual output-gradient norm under the data distribution, and $\widetilde{\mathcal{L}}_{S}$ is a "one-sample training loss" term. The bound thus ties generalization to the decay of the smallest non-zero Hessian eigenvalue; in the source it is Theorem 4, obtained under Assumption A1 (the residual output-gradient norm $\sigma^{2}_{(\bm{x},\bm{y})}=\|\nabla_{\bm{f}}\bm{\ell}(\bm{f}_{\boldsymbol{\theta}^{\star}}(\bm{x}),\bm{y})\|^{2}$ is non-zero with probability $\alpha$) in the limit of vanishing external regularization $\lambda\to0$ and for a single output, $K=1$. Under additional assumptions — the functional (second-derivative-of-the-network) part of the Hessian vanishes at the optimum, the Jacobian-covariance spectrum at the optimum is comparable to that at initialization, and the Jacobian columns are sub-Gaussian (their tails are dominated by those of a Gaussian) at initialization — their Theorem 5 shows for the mean-squared-error (MSE) loss, in the proportional asymptotic regime $p,n\to\infty$ with $p/n$ constant, that this lower bound diverges at the interpolation threshold: double descent is captured by the vanishing of $\lambda_{r}(\mathbf{H})$. They also explain the empirically observed threshold locations ($p\approx Kn$ for MSE with $K$ output classes, via a Hessian-rank argument), the accentuating effect of label noise, and why suitable explicit regularization removes the peak.

Two phenomena remain unexplained at the theory level. First, everything in Theorem 5 is asymptotic; finite-$(n,p)$ behavior is verified only empirically (down to a few thousand parameters, on MNIST-1D, a lightweight one-dimensional analogue of the MNIST digit-classification dataset introduced by Greydanus and Kobak 2020), because evaluating the bound requires the full Hessian spectrum, whose exact computation costs $O(p^{3})$. Second, for the cross-entropy loss used in classification, the Hessian of the loss with respect to the network output is $\operatorname{diag}(\bm{p})-\bm{p}\bm{p}^{\top}$ where $\bm{p}=\operatorname{softmax}(\bm{f})$ is the vector of predicted class probabilities; this matrix has rank $K-1$ for $K$ classes, which naively places the threshold at $p\approx(K-1)n$, yet empirically the test loss peaks at $p\approx n$ and — remarkably — the entire Hessian spectrum, not just its smallest non-zero eigenvalue, collapses to (near) zero near the threshold, consistent with predicted probabilities approaching one-hot values (all probability mass on a single class) as the training loss vanishes. The authors therefore posit the "vanishing Hessian hypothesis": near $p\approx n$, stochastic gradient descent (SGD) finds critical points with (nearly) zero Hessian for networks trained with cross-entropy, and over-parameterization beyond the interpolation threshold helps completely avoid or significantly decelerate convergence to such critical points.

This hypothesis sits in tension with the classical flat-minima view of generalization (Hochreiter and Schmidhuber 1997; Keskar, Mudigere, Nocedal, Smelyanskiy and Tang 2016) — that flatter minima generalize better — since near $p\approx n$ the Hessian is the flattest possible; with the reparameterization critique of Hessian flatness (Dinh, Pascanu, Bengio and Bengio 2017); and with the observation of Granziol (2020) that cross-entropy-trained low-loss solutions with large weights have small Hessian-based flatness measures regardless of generalization. The paper's discussion also settles a conjecture of Kuzborskij, Szepesvári, Rivasplata, Rannen-Triki and Pascanu (2021) in the negative direction: the Hessian at the optimum, not the penultimate-feature covariance, is the relevant spectral object for neural-network double descent. What remains open, as the paper itself states, is to convert this asymptotic, spectral phenomenology into a finite-sample and dynamical theory.

Problem Statement

The problem is to complete, at the finite-sample and dynamical level, the Hessian-based account of double descent in finite-width neural networks established asymptotically by Singh, Lucchi, Hofmann and Schölkopf (2022). Its three technical cores serve the single objective of turning the asymptotic spectral phenomenology into a rigorous theory valid at practical network sizes; the source lists the corresponding questions as separate open directions:

1. Non-asymptotic Hessian-based bounds. Establish finite-(n,p) guarantees, in the source's architecture-agnostic setting of general finite-width networks rather than for a single solvable model class, for the smallest non-zero eigenvalue $\lambda_{r}$ of the training-loss Hessian at the trained optimum and for the population-risk lower bound of the source's Theorems 4-5: quantitative statements of how the bound behaves at and near the interpolation threshold for finite $n$ and $p$, recovering Theorem 5 in the proportional asymptotic limit, with rates describing how closely the asymptotic divergence is approached at practical sizes.

2. The vanishing Hessian for cross-entropy. This core comprises four sub-claims, each with independent standing:

   • 2.1 (Criterion). Prove or refute the source's vanishing Hessian hypothesis: give checkable conditions under which gradient-based training (SGD or gradient descent) of a finite-width network with the cross-entropy loss near the interpolation threshold $p\approx n$ converges to critical points whose Hessian (nearly) vanishes — or, should the hypothesis fail, conditions under which it fails.
   • 2.2 (Decay rate). Determine how $\lambda_{r}$ behaves with training time and with $p/n$ in this regime, including whether and how fast it decays toward zero.
   • 2.3 (Over-parameterization). Determine whether over-parameterization beyond the threshold avoids or decelerates convergence to such critical points.
   • 2.4 (Threshold location). Give the sharper characterization of the cross-entropy phenomenon requested in the source's discussion: why the risk peaks at $p\approx n$ rather than the naive rank-based location $p\approx(K-1)n$.

3. Regime selection. Determine checkable conditions on the data distribution, architecture, and loss — conditions on these primitives rather than a restatement of which regime attains the lower risk — under which the risk-optimal (global) optimum of a neural network lies in the over-parameterized regime rather than the under-parameterized one.

The source also raises two further side-questions, which are out of scope for this task and are included as context only (they are not required deliverables): (i) whether the flat-minima-generalize-better hypothesis admits a qualification in view of the vanishing Hessian at cross-entropy interpolation, where the Hessian is the flattest possible and can therefore not by itself distinguish generalizing from non-generalizing solutions; and (ii) whether the Hessian-based lower bounds can be made practically informative — non-vacuous when the residual gradient term $\sigma^{2}_{\min}$ is small, and computable without the $O(p^{3})$ full-spectrum computation — so they can serve as predictive tools rather than only as mechanism isolators; the source treats this practical applicability as an open research area of secondary concern relative to its primary purpose of isolating the source of double descent.

References

1. Sidak Pal Singh, Aurelien Lucchi, Thomas Hofmann, Bernhard Schölkopf. "Phenomenology of Double Descent in Finite-Width Neural Networks." arXiv:2203.07337 (2022). DOI: 10.48550/arXiv.2203.07337. https://arxiv.org/abs/2203.07337
2. Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal. "Reconciling modern machine-learning practice and the classical bias-variance trade-off." Proceedings of the National Academy of Sciences 116(32):15849-15854 (2019). DOI: 10.1073/pnas.1903070116. https://doi.org/10.1073/pnas.1903070116
3. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever. "Deep Double Descent: Where Bigger Models and More Data Hurt." arXiv:1912.02292 (2019). DOI: 10.48550/arXiv.1912.02292. https://arxiv.org/abs/1912.02292
4. Trevor Hastie, Andrea Montanari, Saharon Rosset, Ryan J. Tibshirani. "Surprises in High-Dimensional Ridgeless Least Squares Interpolation." arXiv:1903.08560 (2019). DOI: 10.48550/arXiv.1903.08560. https://arxiv.org/abs/1903.08560
5. Song Mei, Andrea Montanari. "The generalization error of random features regression: Precise asymptotics and double descent curve." arXiv:1908.05355 (2019). DOI: 10.48550/arXiv.1908.05355. https://arxiv.org/abs/1908.05355
6. Sam Greydanus, Dmitry Kobak. "Scaling Down Deep Learning with MNIST-1D." arXiv:2011.14439 (2020). DOI: 10.48550/arXiv.2011.14439. https://arxiv.org/abs/2011.14439
7. Sepp Hochreiter, Jürgen Schmidhuber. "Flat Minima." Neural Computation 9(1):1-42 (1997). DOI: 10.1162/neco.1997.9.1.1. https://doi.org/10.1162/neco.1997.9.1.1
8. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima." arXiv:1609.04836 (2016). DOI: 10.48550/arXiv.1609.04836. https://arxiv.org/abs/1609.04836
9. Ilja Kuzborskij, Csaba Szepesvári, Omar Rivasplata, Amal Rannen-Triki, Razvan Pascanu. "On the Role of Optimization in Double Descent: A Least Squares Study." arXiv:2107.12685 (2021). DOI: 10.48550/arXiv.2107.12685. https://arxiv.org/abs/2107.12685
10. Quynh Nguyen, Marco Mondelli, Guido Montufar. "Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks." arXiv:2012.11654 (2020). DOI: 10.48550/arXiv.2012.11654. https://arxiv.org/abs/2012.11654
11. Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio. "Sharp Minima Can Generalize For Deep Nets." arXiv:1703.04933 (2017). DOI: 10.48550/arXiv.1703.04933. https://arxiv.org/abs/1703.04933
12. Diego Granziol. "Flatness is a False Friend." arXiv:2006.09091 (2020). DOI: 10.48550/arXiv.2006.09091. https://arxiv.org/abs/2006.09091
13. Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Talwalkar. "Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability." arXiv:2103.00065 (2021). DOI: 10.48550/arXiv.2103.00065. https://arxiv.org/abs/2103.00065
14. Grigory Khromov, Sidak Pal Singh. "Some Fundamental Aspects about Lipschitz Continuity of Neural Networks." arXiv:2302.10886 (2023). DOI: 10.48550/arXiv.2302.10886. https://arxiv.org/abs/2302.10886
15. Jim Zhao, Sidak Pal Singh, Aurelien Lucchi. "Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks." arXiv:2411.02139 (2024). DOI: 10.48550/arXiv.2411.02139. https://arxiv.org/abs/2411.02139
16. Simon Martin, Giulio Biroli, Francis Bach. "High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks." arXiv:2601.10483 (2026). DOI: 10.48550/arXiv.2601.10483. https://arxiv.org/abs/2601.10483
17. Harsh Vardhan, Hossein Taheri, Arya Mazumdar. "Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks." arXiv:2606.04429 (2026). DOI: 10.48550/arXiv.2606.04429. https://arxiv.org/abs/2606.04429
18. Neta Shoham, Liron Mor-Yosef, Haim Avron. "Flatness After All?" arXiv:2506.17809 (2025). DOI: 10.48550/arXiv.2506.17809. https://arxiv.org/abs/2506.17809
19. Jason W. Rocks, Pankaj Mehta. "Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models." Physical Review Research 4, 013201 (2022). DOI: 10.1103/PhysRevResearch.4.013201. https://doi.org/10.1103/PhysRevResearch.4.013201
20. Egor Petrov, Nikita Kiselev, Vladislav Meshkov, Andrey Grabovoy. "Closing the Curvature Gap: Full Transformer Hessians." arXiv:2510.16927 (2025). DOI: 10.48550/arXiv.2510.16927. https://arxiv.org/abs/2510.16927
21. Markus Gross, Arne P. Raulf, Christoph Räth. "Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation." arXiv:2311.14120 (2023). DOI: 10.48550/arXiv.2311.14120. https://arxiv.org/abs/2311.14120