Completing the scaling-limit (DMFT) theory of continual learning: finite-width convergence, the learning-rate-driven lazy–rich transition, depth, and mitigation scaling

The ICML 2025 paper 'The Importance of Being Lazy: Scaling Limits of Continual Learning' (Graldi, Breccia, Lanzillotta, Hofmann, Noci) established an infinite-width dynamical-mean-field-theory (DMFT) description of continual learning in the feature-learning regime, showing that increasing width helps only when it reduces feature learning, that forgetting rises sharply beyond a lazy–rich transition (LRT) in the feature-learning strength $\gamma_0$, and that performance is optimal at a critical $\gamma_0$ that transfers across scales. The paper's own discussion leaves this theory uncompleted at four linked points: the role of the quadratic learning-rate scaling in producing the sharp LRT; the finite-width convergence to the DMFT limit and the inversion of the width-benefit across the LRT; depth scaling under feature-learning-controlled parameterizations; and the scaling behavior of standard mitigation methods (experience replay, elastic weight consolidation) inside the framework. Two further author-noted limitations — the second-order truncation of the $\gamma_0$ perturbation expansion and the cubic sample/task cost of DMFT simulation — are recorded below as context bounding the theory's completeness rather than as author-posed open questions.

Background

Continual learning (CL) is the setting in which a network is trained sequentially on tasks $T_1,\dots,T_T$ and evaluated on all of them; the degradation of performance on earlier tasks as later ones are learned is catastrophic forgetting (CF), typically summarized by a forgetting metric such as the source paper's CFr (the average drop of past-task accuracy from its peak). A central fact of wide-network theory is that training dynamics depend on the parameterization, i.e. the prescription for how initialization variances and the learning rate scale with width. In the lazy or kernel regime (e.g. the Neural Tangent Parameterization, NTP), the network function stays close to its initialization and training is governed by the Neural Tangent Kernel (NTK), the kernel obtained by linearizing the network function at initialization; in the rich or feature-learning regime (e.g. the Maximal Update Parameterization, $\mu$P, the parameterization underlying practical zero-shot hyperparameter transfer), the network's features change by order-one amounts at any width. Graldi et al. (2025) interpolate between the two extremes with a single scalar $\gamma_0$, the scale of the output-layer weights in the parameterization, with $\gamma_0 \to 0$ lazy and $\gamma_0 = 1$ recovering full $\mu$P. In the lazy regime, CL theory already existed: Doan et al. (2021) showed that forgetting is controlled by the overlap matrix between the tasks' NTK blocks, and Mirzadeh et al. (2022) showed that wider networks forget less under lazy scaling. These results say nothing about the feature-learning regime relevant at practical scales. Graldi et al. (2025) therefore extended dynamical mean-field theory (DMFT), a technique from statistical physics that describes infinite-width training with feature learning through a closed system of self-consistent integral equations for order parameters (forward pre-activation fields and backward response fields), generally solved numerically, to task-sequential training. Their findings: increasing width reduces forgetting only insofar as it increases laziness; average error is optimal at a critical feature-learning level $\gamma_0^\star \approx 0.1$ (on Split-CIFAR10, a standard continual-learning benchmark that splits the CIFAR-10 image-classification dataset into per-task class subsets) that transfers across widths and depths; a lazy–rich transition (LRT) at $\gamma_0 \approx \gamma_0^{LRT}$, modulated by task similarity, separates an effectively lazy regime with low forgetting from a rich regime with sharply increasing forgetting; the width-benefit inverts across the LRT (for $\gamma_0 < \gamma_0^{LRT}$ finite-width forgetting approaches the infinite-width limit from above, for $\gamma_0 > \gamma_0^{LRT}$ from below); a perturbative expansion of the dynamics in $\gamma_0$ is computed to second order for a two-layer linear network on two tasks; and preliminary depth experiments under $\mu$P and a depth-corrected variant $\mu$P $+ 1/\sqrt{L}$ show no depth benefit on the datasets tried. Relatedly, Atanasov et al. (2025) mapped, for single-task online training, the joint scaling regimes of the feature-learning strength $\gamma$ and the learning rate $\eta$: the optimal $\eta \propto \gamma^2$ in the lazy regime but scales sub-quadratically in rich regimes, which underlies the source paper's suspicion that its own quadratic learning-rate scaling in $\gamma_0$ contributes to the sharpness of the observed LRT. The unresolved target is the completion of this scaling-limit theory at the points where the authors themselves stopped.

Problem Statement

Complete the analytical scaling-limit theory of continual learning established by Graldi et al. (2025), closing the gaps the authors themselves identify. The shared objective is a quantitative theory, at the same level of rigor as the existing infinite-width DMFT analysis, of how scale (width and depth), learning-rate scaling, and mitigation strategies determine catastrophic forgetting across the lazy-to-rich continuum. Concretely: (1) determine, theoretically and empirically, whether the sharp lazy–rich transition at $\gamma_0 \approx \gamma_0^{LRT}$ is intrinsic to the $\gamma_0$ parameterization or partly an artifact of the quadratic learning-rate scaling $\eta \propto \gamma_0^2$ used in the paper; (2) characterize analytically the finite-width convergence to the infinite-width DMFT limit — the convergence rate and its dependence on $\gamma_0$ and training time — and explain the observed inversion of the width-benefit across the LRT, where finite-width forgetting approaches the infinite-width limit from above for $\gamma_0 < \gamma_0^{LRT}$ and from below for $\gamma_0 > \gamma_0^{LRT}$; (3) characterize depth scaling in continual learning under parameterizations that decouple feature-learning magnitude from depth, of which the paper's $\mu$P and $\mu$P $+ 1/\sqrt{L}$ experiments are only preliminary; and (4) analyze standard mitigation methods — experience replay (revisiting a small stored buffer of past samples) and elastic weight consolidation (a quadratic penalty anchoring weights important for past tasks) — within the DMFT/scaling-limit framework, deriving their scaling properties with width and $\gamma_0$ and their effectiveness across lazy and rich regimes. The four components are jointly posed parts of a single theory. Two further limitations noted by the authors bound the same theory's completeness but are not posed as open questions in the source and are therefore recorded as context rather than required components: the truncation of the $\gamma_0$ perturbation expansion at second order, stated as a deliberate practical choice for the regime of interest (with $\gamma_0$ typically in $[0,1]$ and the observed transition of order $0.1$), and the cubic (in samples and tasks) computational cost of the infinite-width DMFT simulations, stated as the constraint that imposed the paper's simplified experimental setting.

References

1. Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta, Thomas Hofmann, Lorenzo Noci. The Importance of Being Lazy: Scaling Limits of Continual Learning. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), Proceedings of Machine Learning Research (PMLR) 267:20326-20367. arXiv:2506.16884, DOI 10.48550/arxiv.2506.16884. https://arxiv.org/abs/2506.16884
2. Alexander Atanasov, Alexandru Meterez, James B. Simon, Cengiz Pehlevan. The Optimization Landscape of SGD Across the Feature Learning Strength. ICLR 2025. arXiv:2410.04642, DOI 10.48550/arxiv.2410.04642. https://arxiv.org/abs/2410.04642
3. Thang Doan, Mehdi Bennani, Bogdan Mazoure, Guillaume Rabusseau, Pierre Alquier. A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix. AISTATS 2021. arXiv:2010.04003, DOI 10.48550/arxiv.2010.04003. https://arxiv.org/abs/2010.04003
4. Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, Mehrdad Farajtabar. Wide Neural Networks Forget Less Catastrophically. ICML 2022. arXiv:2110.11526, DOI 10.48550/arxiv.2110.11526. https://arxiv.org/abs/2110.11526
5. Blake Bordelon, Cengiz Pehlevan. Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks. arXiv:2304.03408, DOI 10.48550/arxiv.2304.03408. https://arxiv.org/abs/2304.03408
6. Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, Jianfeng Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. NeurIPS 2021. arXiv:2203.03466, DOI 10.48550/arxiv.2203.03466. https://arxiv.org/abs/2203.03466
7. Zonghuan Xu, Krishna Harish. Optimal Training-Time Scaling in Gradual Adaptation. arXiv:2608.04927, DOI 10.48550/arxiv.2608.04927. https://arxiv.org/abs/2608.04927
8. Matan Schliserman, Gon Buzaglo, Itay Evron, Daniel Soudry. Convergence of Continual Learning in Homogeneous Deep Networks. arXiv:2606.30559, DOI 10.48550/arxiv.2606.30559. https://arxiv.org/abs/2606.30559
9. Yuzhi Liu, Zixuan Chen, Zirui Zhang, Yufei Liu, Giulia Lanzillotta. Reactivation: Empirical NTK Dynamics Under Task Shifts. ICML 2025 Workshop on High-dimensional Learning Dynamics. arXiv:2507.16039, DOI 10.48550/arxiv.2507.16039. https://arxiv.org/abs/2507.16039
10. Amir Joudaki, Giulia Lanzillotta, Mohammad Samragh Razlighi, Iman Mirzadeh, Keivan Alizadeh, Thomas Hofmann, Mehrdad Farajtabar, Fartash Faghri. Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity. arXiv:2510.00304, DOI 10.48550/arxiv.2510.00304. https://arxiv.org/abs/2510.00304
11. Giulia Lanzillotta, Damiano Meier, Thomas Hofmann. Heads collapse, features stay: Why Replay needs big buffers. arXiv:2512.07400, DOI 10.48550/arxiv.2512.07400. https://arxiv.org/abs/2512.07400
12. Giulia Lanzillotta, Mandana Samiei, Doina Precup, Razvan Pascanu, Claire Vernade. To Retain or to Adapt? Generalizing Continual Learning. arXiv:2607.05609, DOI 10.48550/arxiv.2607.05609. https://arxiv.org/abs/2607.05609
13. Meng Ding, Jinhui Xu, Kaiyi Ji. Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective. AISTATS 2026. arXiv:2602.02767, DOI 10.48550/arxiv.2602.02767. https://arxiv.org/abs/2602.02767