Minimal alignment requirements and deletion guarantees for residual feature alignment unlearning of pre-trained models

Residual Feature Alignment (RFA) unlearning removes the influence of chosen training data from a pre-trained model by inserting LoRA bypass branches at intermediate layers: residual features are driven to zero on retained data and toward the retained-set average feature on forgotten data, and the same objective can be trained through an algebraically equivalent teacher-student feature-distillation loop adopted for coding convenience. The method works empirically across image classifiers, text classifiers, and language models, but its authors leave open how much alignment machinery it actually needs and what deletion guarantee it provides: which minimal subset of intermediate layers suffices, how the alignment weight $\gamma$ should be set in principle, what computational overhead the training implementation carries, whether the unlearned model is indistinguishable from a retrained one (or can be distinguished by stronger attacks), and how the method behaves under sequential deletion requests. This problem asks for a principled resolution of this alignment-requirement, efficiency, and guarantee trade-off: a theory-backed characterization of how little intermediate-layer alignment achieves effective unlearning, together with an account of what the achieved unlearning actually guarantees.

Background

Machine unlearning is the task of removing the influence of a chosen subset of training data (the forget set $\mathcal{D}_f$) from a trained model while preserving performance on the remaining data (the retain set $\mathcal{D}_r$), motivated by privacy regulation such as the right to erasure in the EU General Data Protection Regulation (GDPR) and by the removal of harmful or outdated content. Exact unlearning requires the unlearned model to be statistically indistinguishable from a gold-standard model retrained from scratch on $\mathcal{D}_r$; because retraining large models is prohibitively expensive, and partition-and-retrain schemes such as SISA (Sharded, Isolated, Sliced, Aggregated; Bourtoule et al., arXiv:1912.03817) require storage and training-time choices unavailable for already-trained pre-trained models, most work on pre-trained models pursues approximate unlearning: modifying parameters so that the model merely behaves approximately like the retrained one. A standard empirical yardstick is the membership inference attack (MIA), in which an adversary trains a classifier to decide whether a sample was in the training set; an unlearned model is considered good if its MIA profile on $\mathcal{D}_f$ is close to that of the retrained model.

LoRA (Low-Rank Adaptation; Hu et al., arXiv:2106.09685) is a parameter-efficient fine-tuning technique that freezes pre-trained weights $W$ and learns only a low-rank update $\Delta W = BA$ added in parallel, so that fine-tuning touches a small fraction of the parameters and the update can be merged into $W$ afterward with no added inference cost. LoRA's initialization (with $B = 0$) makes the update start at zero, which gives training a smooth starting point.

The source work (Qin, Zhu, Wang, Zhou; IEEE TDSC 23(3):5797–5813, 2026; arXiv:2411.08443) proposes Residual Feature Alignment (RFA) unlearning. At selected intermediate layers, the model's features are decomposed into a frozen pre-trained component $x'_k = Wx_{k-1}$ and a trainable residual component $\Delta x'_k = BAx_{k-1}$ produced by a LoRA bypass. On $\mathcal{D}_r$ the residual is aligned to zero, so retained-data features are unchanged; on $\mathcal{D}_f$ the residual is aligned so that the summed feature moves toward the average intermediate feature of $\mathcal{D}_r$, which erases forget-set class structure while disturbing the retain-set feature distribution as little as possible. The total loss combines these per-layer alignment terms, weighted by a hyperparameter $\gamma$, with an output-level task loss. For implementation, the objective can be recast as feature distillation in a teacher-student architecture — an algebraically equivalent reformulation adopted by the source to facilitate code implementation: the frozen original model acts as teacher, and the LoRA-equipped student aligns its intermediate features with the teacher's; the source's direct objective and its Algorithm 1 express the same alignment targets without a separate teacher network. The method is evaluated on sample-level and class-level unlearning for image classification (ResNet18, a standard 18-layer convolutional network, and a Vision Transformer (ViT), on CIFAR-10 and Fashion-MNIST), text classification, text generation (GPT-2, a small pre-trained transformer language model), and, in the published version, safety unlearning on the WMDP benchmark (a multiple-choice benchmark measuring hazardous biosecurity/cybersecurity/chemistry knowledge) with the Qwen2.5-3B open 3-billion-parameter language model, using accuracy, perplexity (the standard measure of how well a language model predicts held-out text), activation distance, feature distance (the distance between the unlearned and original models' intermediate features), MIA, and runtime.

The paper itself documents the unresolved issues. Its conclusion states as future work "achieving unlearning with as few intermediate layer alignments as possible", balancing unlearning performance against computational efficiency; its single ablation on ResNet18/CIFAR-10 found that placing LoRA only in deep layers roughly matches placing it in all layers, but it warns that optimal layer selection is architecture- and task-dependent. The teacher-student implementation is acknowledged to add the cost of a teacher forward pass (extra computation and memory). The role of $\gamma$ is characterized only empirically (larger $\gamma$ shrinks feature distance roughly linearly; effects on forget-set accuracy differ between sample and class unlearning), without prescriptive settings or theory. The method is an approximate-unlearning technique: its privacy evidence is empirical MIA only, with no formal indistinguishability-from-retraining guarantee, and all experiments are one-shot, although the introduction notes that deletion requests arrive frequently in practice.

Adjacent results sharpen, but do not answer, these questions. Recent analyses show that output-level success can mask incomplete forgetting: many unlearning methods achieve low forget-set accuracy mainly through a misalignment between last-layer features and the classifier, while hidden features stay discriminative and simple linear probing (training a linear readout on frozen features) recovers near-original accuracy (Gao et al., arXiv:2604.08271); forget-class structure can be recovered by unsupervised clustering attacks (Le et al., arXiv:2607.23934); and in class-level unlearning, merely suppressing classification-head bias terms can satisfy conventional metrics while leaking forgotten labels (Zheng et al., arXiv:2605.08730); complementary critiques show that standard evaluations of LLM unlearning are themselves inadequate (Yuan et al., arXiv:2410.08109). Separately, continual (sequential) unlearning — handling deletion requests that arrive one after another without eroding retained knowledge — has become an active subfield with its own LoRA-based methods (Gao et al., arXiv:2407.10223; Rahulamathavan et al., arXiv:2604.12526; Feng et al., arXiv:2601.13578), and verified unlearning with retracing guarantees exists for partitioned multi-teacher distillation (Quan et al., arXiv:2503.22539). None of these analyzes the RFA construction itself.

Problem Statement

For Residual Feature Alignment (RFA) unlearning — the construction of Qin et al. in which LoRA residual branches at intermediate layers are aligned to zero on the retain set $\mathcal{D}_r$ and to the retain-set average feature on the forget set $\mathcal{D}_f$, with per-layer alignment losses weighted by $\gamma$ alongside an output task loss, and with an algebraically equivalent frozen-teacher feature-distillation loop available for implementation — determine (a) which minimal subset of intermediate layers suffices and a principled criterion for selecting it, and (b) what deletion guarantee the resulting model provides — whether the unlearned model is indistinguishable from the retrained one, and what adversaries stronger than those used in the source's evaluation can recover; the setting of $\gamma$, the implementation cost, and behaviour under sequential deletion are open sub-questions bearing on the same requirement-versus-cost-versus-guarantee trade-off.

The source explicitly poses as future work "achieving unlearning with as few intermediate layer alignments as possible, aiming to balance unlearning performance and computational efficiency". The central question is therefore which subsets of the intermediate layers carrying LoRA alignment suffice for effective unlearning, how small such a subset can be, and whether there is a principled criterion for selecting it for a given model architecture and unlearning task — the source's own placement ablation (ResNet18/CIFAR-10, class unlearning) finds deep-only placement roughly comparable to all-layer placement, but warns that the optimal layer selection may be highly dependent on the specific model architecture and unlearning task, so that a single placement strategy is not universally applicable. A candidate subset is a nonempty set of intermediate layers carrying LoRA residual branches: dropping intermediate-layer alignment entirely would abandon the RFA construction rather than minimize it. A subset of intermediate layers suffices for effective unlearning when the unlearned model is at least as close to the gold-standard model retrained from scratch on $\mathcal{D}_r$ — in retain/forget/test accuracy, perplexity for generative tasks, feature and activation distance, and MIA profile — as the source's all-layer construction is on each of these metrics. "Minimal" then follows the minimum-cardinality order: among the subsets that suffice, the fewest aligned intermediate layers qualify, matching the source's goal of "as few intermediate layer alignments as possible". Computational cost decreases monotonically with the number of aligned layers, so it is reported alongside this comparison rather than traded against it.

Three further limitations acknowledged by the source bear on the same requirement-versus-cost-versus-guarantee trade-off and remain open for this method: (i) the role of the alignment weight $\gamma$ in trading intermediate-layer alignment against the output-task loss is characterized only by single-setting empirical trends, without prescriptive guidance or theory; (ii) the deletion guarantee is empirical only — the privacy evidence consists of membership inference attacks, with no formal indistinguishability-from-retraining bound, and it is unknown whether stronger adversaries (feature probing, clustering, model extraction) than those used in the source's evaluation recover forgotten information that the output-level metrics miss; and (iii) all experiments are one-shot, so the method's behavior under sequential (continual) deletion requests — where repeated unlearning steps may compound utility loss or interference between LoRA updates — is unexamined, although deletion requests arrive frequently in practice. On the cost side of the trade-off, the source also acknowledges that the teacher-student implementation requires the additional output of the teacher network, increasing computational and memory overhead; since its direct objective and Algorithm 1 already express the alignment targets without a separate teacher network, the open efficiency question is at what implementation cost a given unlearning level can be achieved, not whether teacher-free training of the construction is possible in principle.

References

1. Laiqiao Qin, Tianqing Zhu, Linlin Wang, Wanlei Zhou, 'Machine Unlearning on Pre-trained Models by Residual Feature Alignment Using LoRA', IEEE Transactions on Dependable and Secure Computing 23(3):5797–5813, 2026. DOI: 10.1109/TDSC.2026.3658545; arXiv:2411.08443. https://arxiv.org/abs/2411.08443
2. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, 'LoRA: Low-Rank Adaptation of Large Language Models', arXiv:2106.09685, 2021. https://arxiv.org/abs/2106.09685
3. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot, 'Machine Unlearning', arXiv:1912.03817, 2019. https://arxiv.org/abs/1912.03817
4. Yichen Gao, Altay Unal, Akshay Rangamani, Zhihui Zhu, 'An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations', arXiv:2604.08271, 2026. https://arxiv.org/abs/2604.08271
5. Anjie Le, Can Peng, Hongcheng Guo, J. Alison Noble, 'DECAF: De-Clustering for Adaptive Representational Unlearning', arXiv:2607.23934, 2026. https://arxiv.org/abs/2607.23934
6. Weidong Zheng, Kongyang Chen, Yuanwei Guo, Yatie Xiao, 'Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluation', arXiv:2605.08730, 2026. https://arxiv.org/abs/2605.08730
7. Chongyang Gao, Lixu Wang, Kaize Ding, Chenkai Weng, Xiao Wang, Qi Zhu, 'On Large Language Model Continual Unlearning', arXiv:2407.10223, 2024. https://arxiv.org/abs/2407.10223
8. Yogachandran Rahulamathavan, Nasir Iqbal, Juncheng Hu, Sangarapillai Lambotharan, 'Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA', arXiv:2604.12526, 2026. https://arxiv.org/abs/2604.12526
9. Qian Feng, JiaHang Tu, Mintong Kang, Hanbin Zhao, Chao Zhang, Hui Qian, 'FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning', arXiv:2601.13578, 2026. https://arxiv.org/abs/2601.13578
10. Yijun Quan, Zushu Li, Giovanni Montana, 'Efficient Verified Machine Unlearning For Distillation', arXiv:2503.22539, 2025. https://arxiv.org/abs/2503.22539
11. Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, Min Lin, 'A Closer Look at Machine Unlearning for Large Language Models', arXiv:2410.08109, 2024. https://arxiv.org/abs/2410.08109
12. Atharv Mittal, 'LoRA Unlearns More and Retains More (Student Abstract)', arXiv:2411.11907, 2024. https://arxiv.org/abs/2411.11907