Applicability of Frobenius-Isometry Reinitialization (FIRE) under Restricted Past-Data Access and at Large-Language-Model Scale

FIRE (Frobenius-Isometry Reinitialization; Han et al., International Conference on Learning Representations (ICLR) 2026 oral) reinitializes each weight matrix of a continually trained network to the point that minimizes the squared Frobenius error to the current weights subject to zero deviation from isometry, computed by Newton-Schulz iteration, an iterative matrix-polynomial scheme that approximates the orthogonal factor of the matrix. It was shown to balance stability and plasticity in continual vision, reinforcement learning, and small-scale language modeling, but only in regimes where past data remain accessible during continual training and only at the scale of GPT-0.1B (a roughly 124M-parameter GPT-style model) for continual pretraining. The open problem is to establish whether Frobenius-isometry reinitialization remains an effective stability-plasticity intervention in the regimes the original evaluation excluded: continual learning with restricted or no access to past data (online continual learning and bounded replay memory), substantially larger language models, and continual fine-tuning workflows for large language models (LLMs) including interactions with fine-tuning optimizers and learning-rate schedules. The applicability boundary in these regimes remains unresolved.

Background

Deep neural networks trained on nonstationary data must balance two competing properties: stability, the retention of previously learned knowledge, and plasticity, the ability to adapt to new tasks or data. Which failure mode dominates depends on the training regime. Conventional continual learning assumes restricted access to past data, so catastrophic forgetting - the overwriting of old knowledge by new updates - is the central concern. By contrast, foundation models and robotic agents are typically trained on ever-expanding datasets in which past data remain fully accessible; there the central concern is loss of plasticity, the progressive inability of the network to fit new targets, demonstrated even in simple supervised continual-learning settings by Dohare et al. (Nature, 2024).

A widely used family of interventions against plasticity loss is reinitialization (reset) methods, which periodically move weights back toward a fresher configuration. Shrink and Perturb (Ash and Adams, 2020) shrinks all weights toward zero and adds noise; ReDo (Sokar et al., 2023) resets only dormant units - hidden units whose activations are nearly always zero and whose gradients therefore vanish. These methods are difficult to tune: conservative resets fail to restore plasticity, while aggressive ones erase useful knowledge.

FIRE (Han et al., International Conference on Learning Representations (ICLR) 2026) makes this tradeoff principled. For each weight matrix $W$, FIRE measures stability by the Squared Frobenius Error (SFE) $\|W-\widetilde{W}\|_F^2$, the squared Frobenius-norm distance between the current weights and the reinitialization point, and plasticity by the Deviation from Isometry (DfI) $\|\widetilde{W}^\top\widetilde{W}-I\|_F^2$, i.e. how far the matrix is from having orthonormal columns (being an isometry). Near-isometry is associated, empirically and theoretically in that paper, with a smoother loss landscape, higher effective rank (more significantly nonzero singular directions), and more active units. FIRE reinitializes each weight matrix at the solution of the constrained problem $\min_{\widetilde{W}}\|W-\widetilde{W}\|_F^2$ subject to $\widetilde{W}^\top\widetilde{W}=I$, approximated efficiently by Newton-Schulz iteration, an iterative matrix-polynomial scheme that approaches the orthogonal (polar) factor of $W$ without a full singular value decomposition. In the authors' experiments, FIRE outperformed naive training and standard reinitialization baselines in continual visual learning (CIFAR-10/100 and Tiny-ImageNet splits under random and class-incremental protocols (the stream is split into sequential chunks, mixing classes randomly in the former and introducing disjoint new classes in the latter) with residual networks (ResNet), Vision Transformers (ViT), and VGG convolutional architectures), continual reinforcement learning (the soft actor-critic algorithm SAC on HumanoidBench and the deep Q-network algorithm DQN on Atari), and continual pretraining of GPT-0.1B, a roughly 124M-parameter GPT-style language model warm-started from WikiText-103 and continually trained on an OpenWebText/WikiText-103 mixture.

Crucially, all of these evaluations assume access to past data: continual training keeps revisiting earlier data (data mixtures in language modeling, replay buffers in reinforcement learning), so plasticity loss rather than forgetting is the dominant failure mode, and the interaction of FIRE with forgetting-dominated or larger-scale regimes is untested. The authors state this past-data assumption as the main limitation of the work and additionally note that the language-model experiments used only relatively small models in continual pretraining, leaving restricted-data scenarios, larger models, and continual fine-tuning as the boundaries of the current study.

Problem Statement

The problem, stated by Han et al. as the boundary of their FIRE study, is to establish whether Frobenius-isometry reinitialization - resetting each weight matrix to the point minimizing the squared Frobenius error to the current weights subject to zero deviation from isometry - remains an effective way to balance stability and plasticity outside the regime in which it was demonstrated. Three linked regimes are excluded by the original evaluation and together define this applicability boundary: (1) restricted or no access to past data, i.e. continual learning protocols such as pure online continual learning (each example seen at most once) or bounded replay memory (a size-limited buffer of stored past examples), where catastrophic forgetting - the loss of previously learned knowledge - competes with plasticity loss and the past-data-accessible setup no longer applies; (2) substantially larger language models, i.e. continual pretraining well beyond the GPT-0.1B scale (roughly 124M parameters) used in the source, asking whether the same per-layer orthogonalization preserves performance and efficiency at foundation-model scale (models with billions of parameters pretrained on broad data); and (3) continual fine-tuning workflows for large language models (LLMs), i.e. applying the reinitialization not only to pretraining but to continual fine-tuning, including how it interacts with fine-tuning optimizers and learning-rate schedules. The substance of this boundary is whether Frobenius-isometry reinitialization helps, harms, or is immaterial in these regimes. The three parts share one scientific objective - determining the applicability boundary of isometry-based reinitialization for stability-plasticity control - and each is independently refutable: a controlled comparison that characterizes whether the reinitialization helps, harms, or is immaterial within one regime establishes that part of the boundary while leaving the remaining parts open.

References

1. Isaac Han, Sangyeon Park, Seungwon Oh, Donghu Kim, Hojoon Lee, Kyung-Joong Kim. FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff. ICLR 2026 (oral); arXiv:2602.08040, DOI 10.48550/arxiv.2602.08040. https://arxiv.org/abs/2602.08040
2. Toan Nguyen, Yang Liu, Trung Le, Celso de Melo, Flora D. Salim. FOGO: Forgetting-aware Orthogonalization Optimizer. arXiv:2606.10406 (2026), DOI 10.48550/arxiv.2606.10406. https://arxiv.org/abs/2606.10406
3. Lirui Luo, Guoxi Zhang, Hongming Xu, Rongqing Li, Cong Fang, Lifeng Fan. Continual Reasoning Gym: Diagnosing and Harnessing Shared Reasoning in Continual RLVR. arXiv:2608.18574 (2026), DOI 10.48550/arxiv.2608.18574. https://arxiv.org/abs/2608.18574
4. Tyler Kastner, Nimrod De La Vega, Amir-massoud Farahmand. SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning. arXiv:2608.18319 (2026), DOI 10.48550/arxiv.2608.18319. https://arxiv.org/abs/2608.18319
5. Donghu Kim, Youngdo Lee, Hojoon Lee, Johan Obando-Ceron, Byungkun Lee, Aaron Courville, Pablo Samuel Castro, Jaegul Choo, Clare Lyle. V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control. RLC 2026; arXiv:2608.07870, DOI 10.48550/arxiv.2608.07870. https://arxiv.org/abs/2608.07870
6. Jordan T. Ash, Ryan P. Adams. On Warm-Starting Neural Network Training. NeurIPS 2020; arXiv:1910.08475. https://arxiv.org/abs/1910.08475
7. Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, Richard S. Sutton. Loss of plasticity in deep continual learning. Nature 632:768-774 (2024), DOI 10.1038/s41586-024-07711-7. https://doi.org/10.1038/s41586-024-07711-7
8. Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, Utku Evci. The Dormant Neuron Phenomenon in Deep Reinforcement Learning. ICML 2023; arXiv:2302.12902. https://arxiv.org/abs/2302.12902