Lu et al. (ICML 2025) showed empirically that, under an equal parameter constraint, deeper neural networks learn new continual-learning tasks better (plasticity) while wider networks forget less (stability), and instantiated this asymmetry as the Dual-Arch framework pairing a stability-specialized wide learner with a plasticity-specialized deep learner — but gave no account of why the asymmetry holds. Derive and validate a principled theoretical account that predicts when and why architectural scalings (depth versus width at matched parameter or compute budgets) favor stability or plasticity across data distributions and continual-learning regimes, together with the design rules it would license. Existing theory covers only components — width effects via lazy-training/feature-learning limits (Mirzadeh et al.; Graldi et al.), depth effects in a wide-network statistical-mechanics model that appear to conflict with the empirical asymmetry (Shan et al.), and architecture-agnostic plasticity-loss mechanisms (Joudaki et al.) — and no published work supplies or claims the unified predictive account. The problem is open.

Background

Continual learning (CL) asks a neural network to learn a sequence of tasks from non-stationary data while retaining what it learned before; the failure mode in which new learning erases old knowledge is catastrophic forgetting. The central tension is the stability-plasticity dilemma: stability means preserving previously learned knowledge, plasticity means the ability to acquire new knowledge. In class-incremental learning (Class-IL), the most common protocol, the model sees classes in disjoint task batches and must classify all classes at test time without knowing the task identity; performance is summarized by average accuracy over tasks, by forgetting measures such as the drop in accuracy on earlier tasks after later training (average forgetting), and by accuracy on the most recently learned task as a plasticity proxy. Most CL research adjusts training procedures (regularization, replay of stored examples, parameter isolation) while keeping the network architecture fixed. A smaller line of work studies the architecture itself. Mirzadeh et al. (ICML 2022, 'Wide Neural Networks Forget Less Catastrophically') showed that increasing width alone markedly reduces forgetting and offered mechanistic sketches via gradient orthogonality (old- and new-task gradients interfering less in wider networks), activation sparsity, and lazy training; the companion 'Architecture Matters in Continual Learning' study and ArchCraft (Lu et al., IJCAI 2024) explored depth, width, and component choices empirically, with ArchCraft crafting a CL-tailored architecture search space. Lee (2024) reported that larger models do not automatically fare better in online continual learning, where each example is seen only once in a single pass over a task stream. The source paper (Lu et al., ICML 2025, 'Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective') sharpened this line with the cleanest empirical statement so far: under an equal parameter constraint, deeper networks exhibit better plasticity while wider networks exhibit superior stability. It instantiated the finding as Dual-Arch, a plug-in framework pairing a wide, stability-specialized learner (Sta-Net) with a deep, plasticity-specialized learner (Pla-Net), coupled by knowledge distillation (training one network to match another's outputs), improving existing CL methods at up to 87% fewer parameters; it also reported a 1.39×–1.77× wall-clock training overhead from sequentially training the two learners, which it explicitly flags as a limitation. The paper offers no explanation of why the depth/width asymmetry holds, and its conclusion invites 'further study on exploring a better trade-off between stability and plasticity from an architectural perspective.' The theoretical landscape is partial and partly conflicting. The neural tangent kernel (NTK) describes training in the infinite-width lazy regime, where the network's first-order approximation to its input-output map stays fixed during training and features change little; the rich regime (strong feature learning) is the opposite limit. Mirzadeh et al.'s width analysis lives closest to the lazy limit. Shan, Li, and Sompolinsky give a statistical-mechanics theory of CL in deep, wide networks using dynamical mean-field theory (DMFT, a method that closes the equations of training dynamics in terms of a few self-consistently determined order parameters — summary statistics of the system); their order parameters capture task-feature and rule similarity, and the theory predicts, notably, that increasing depth can reduce interference between tasks — in visible tension with the equal-parameter empirics above. Graldi et al. (ICML 2025) use DMFT to study CL scaling limits, showing that increasing width helps only when it reduces feature learning (increasing 'laziness'), that a lazy-to-rich transition is modulated by task similarity, and that optimal performance sits at a critical level of feature learning that depends on task non-stationarity. On the plasticity side, Joudaki et al. formalize loss of plasticity through stable manifolds in parameter space (traps created by saturated 'frozen' units and redundant 'cloned' units), revealing a tension between simplicity biases that aid generalization and plasticity in non-stationary settings; Wang et al. prove by counterexample that popular plasticity diagnostics such as representation rank and NTK rank fail to predict future trainability and propose an 'optimization readiness' metric instead. Related mechanism-level results connect plasticity to dynamical isometry (keeping the singular values of each layer's input-output Jacobian near one) and, in the toy setting of deep matrix factorization (approximating a target matrix by a product of several learned thin matrices), to a depth-induced low-rank implicit bias, meaning that gradient descent preferentially converges to low-rank solutions as depth grows. What no existing work provides is a principled account predicting, at matched parameter (or compute) budgets, when a depth-favoring versus a width-favoring scaling will be more stable or more plastic, as a function of data-distribution properties (task similarity, distribution-shift magnitude) and CL regime (single-head versus multi-head, Class-IL, online, with or without replay), together with design rules for stability- and plasticity-specialized architectures.

Problem Statement

Derive and validate a principled theoretical account that predicts when and why architectural scalings of a neural network — in particular depth versus width under an equal parameter (or equal compute) budget — favor stability or plasticity in continual learning, as a function of data-distribution properties (e.g., task similarity and distribution-shift magnitude) and CL regime (single-head versus multi-head, class-incremental, online, with or without replay). At minimum the account should: (i) explain the empirical asymmetry of Lu et al. (ICML 2025) — deeper networks more plastic, wider networks more stable at matched parameter counts, established in class-incremental protocols with replay-based training — and the width effect of Mirzadeh et al. (ICML 2022); and (ii) yield design rules predicting which architectural scalings, or stability-/plasticity-specialized architectures such as the Sta-Net/Pla-Net pair, optimize the stability-plasticity balance in a specified regime. The source paper's associated engineering directions — reducing Dual-Arch's 1.39×–1.77× sequential-training overhead and validating Dual-Arch more broadly across model families and deployment regimes — are separate open engineering questions, not parts of this problem's scientific objective.

References

  1. Aojun Lu, Hangjie Yuan, Tao Feng, Yanan Sun. Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective. ICML 2025, PMLR v267. arXiv:2506.03951, DOI: 10.48550/arXiv.2506.03951. https://arxiv.org/abs/2506.03951 (published version: https://proceedings.mlr.press/v267/lu25t.html)
  2. Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, Mehrdad Farajtabar. Wide Neural Networks Forget Less Catastrophically. ICML 2022, PMLR v162. arXiv:2110.11526. https://arxiv.org/abs/2110.11526 (published version: https://proceedings.mlr.press/v162/mirzadeh22a.html)
  3. Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, Mehrdad Farajtabar. Architecture Matters in Continual Learning. arXiv:2202.00275 (2022). https://arxiv.org/abs/2202.00275
  4. Aojun Lu, Tao Feng, Hangjie Yuan, Xiaotian Song, Yanan Sun. Revisiting Neural Networks for Continual Learning: An Architectural Perspective (ArchCraft). IJCAI 2024, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 4651-4659. DOI: 10.24963/ijcai.2024/514. arXiv:2404.14829. https://arxiv.org/abs/2404.14829
  5. Haozhe Shan, Qianyi Li, Haim Sompolinsky. Order parameters and phase transitions of continual learning in deep neural networks. arXiv:2407.10315 (2024). https://arxiv.org/abs/2407.10315
  6. Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta, Thomas Hofmann, Lorenzo Noci. The Importance of Being Lazy: Scaling Limits of Continual Learning. ICML 2025. arXiv:2506.16884. https://arxiv.org/abs/2506.16884
  7. Amir Joudaki, Giulia Lanzillotta, Mohammad Samragh Razlighi, Iman Mirzadeh, Keivan Alizadeh, Thomas Hofmann, Mehrdad Farajtabar, Fartash Faghri. Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity. arXiv:2510.00304 (2025). https://arxiv.org/abs/2510.00304
  8. Jiuqi Wang, Jayanth Srinivasa, Claire Chen, Shuze Daniel Liu, Ali Payani, Shangtong Zhang. Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective. arXiv:2605.09044 (2026). https://arxiv.org/abs/2605.09044
  9. Eunhae Lee. The impact of model size on catastrophic forgetting in Online Continual Learning. arXiv:2407.00176 (2024). https://arxiv.org/abs/2407.00176
  10. Kathrin Korte, Christian Medeiros Adriano, Joachim Winther Pedersen, Eleni Nisioti, Sebastian Risi. Dimensionality Controls When Modularity Helps in Continual Learning. arXiv:2606.17889 (2026). https://arxiv.org/abs/2606.17889
  11. Baekrok Shin, Chulhee Yun. Implicit Bias and Loss of Plasticity in Matrix Completion: Depth Promotes Low-Rankness. arXiv:2603.04703 (2026). https://arxiv.org/abs/2603.04703
  12. Andries Rosseau, Robert Müller, Ann Nowé. Preserving Plasticity in Continual Learning via Dynamical Isometry. arXiv:2606.09762 (2026). https://arxiv.org/abs/2606.09762
  13. Supriyo Chakraborty, Krishnan Raghavan. On Understanding of the Dynamics of Model Capacity in Continual Learning. arXiv:2508.08052 (2025). https://arxiv.org/abs/2508.08052
  14. Matan Schliserman, Gon Buzaglo, Itay Evron, Daniel Soudry. Convergence of Continual Learning in Homogeneous Deep Networks. arXiv:2606.30559 (2026). https://arxiv.org/abs/2606.30559
  15. Francesco Mori, Stefano Sarao Mannelli, Francesca Mignacco. Optimal Protocols for Continual Learning via Statistical Physics and Control Theory. arXiv:2409.18061 (2024). https://arxiv.org/abs/2409.18061