Approximate machine unlearning removes the influence of specific training samples from a trained model without retraining from scratch. The ManiF-SMC framework (arXiv:2605.22871, 2026) reformulates approximate unlearning as a purely representation-space relocation: push each erased sample's representation away from its original class-manifold centroid toward its nearest semantic neighbors among retained samples, with a self-mode-connectivity module generating adaptive triplet-loss margins. This reformulation removes dependence on labels and task gradients, but the source paper states one important open challenge: purely representation-space methods suffer greater utility loss than methods that fine-tune on the remaining data, especially at large unlearning sample sizes, and it remains open how to mitigate this degradation without reintroducing label dependence.

Background

Machine unlearning is the problem of removing the influence of particular training data from a learned model without full retraining, motivated by privacy regulation such as the right to be forgotten (Bourtoule et al., 2021). The gold standard is retraining from scratch on the retained data; approximate methods trade fidelity to that reference for efficiency. Common approximate approaches use label manipulation (e.g., relabeling erased samples) or gradient ascent on the loss of erased samples, and methods such as SalUn restrict the forgetting update to saliency-selected weights (Fan et al., 2023). Evaluation typically uses membership inference attack success (MIA, an attacker's ability to distinguish erased samples from non-training samples), remaining accuracy (RA) on retained data, and test accuracy (TA). A model learned on data whose classes occupy low-dimensional manifolds in representation space (the space of hidden activations, e.g., penultimate-layer embeddings) has class-manifold centroids; the manifold capacity framework (Yerxa et al., 2023) studies how manifold geometry in representation space controls separability, and Maximum Manifold Capacity Representations (MMCRs) are self-supervised embeddings trained to maximize manifold separability. Mode connectivity is the empirical phenomenon that independently trained networks can be connected by low-loss paths in parameter space; ManiF-SMC exploits a learned such path (self mode connectivity, SMC) to rapidly reconstruct local manifold geometry and generate adaptive margins for a triplet loss (a loss over an anchor, a positive, and a negative sample that enforces a margin between their distances). ManiF-SMC (Wang et al., 2026) observes that a model retrained on retained data classifies erased samples by their semantic similarity to retained data, and recasts unlearning as pushing each erased sample's representation away from its original centroid toward the centroid of its top-kk nearest retained samples, operating purely in representation space with no labels and no task gradients. It reports unlearning effectiveness comparable to state-of-the-art approximate methods on four image datasets, and its ablations show MMCR-trained backbones improve nearly all unlearning methods. However, three 2026 studies show output-level evaluation is misleading: hidden representations often remain discriminative after apparent forgetting (feature-classifier misalignment, Gao et al., 2026), retraining-inconsistent representation residuals persist and grow with forget fraction (Cosma and Finke, 2026), and output forgetting can coexist with structured representation mismatch relative to retraining (Yong et al., 2026). Against this background, the source paper's own limitations are sharp: because ManiF-SMC uses only manifold representations without task information, it suffers greater utility loss than fine-tuning-based competitors, especially when the unlearning sample size (USS) is large, and its fine-tuning remedy reintroduces label dependence. Mode-connectivity studies of unlearning (Cheng and Amiri, 2025) further indicate most approximate methods are mechanistically distinct from retraining. The relationship between representation geometry and unlearning fidelity is therefore both scientifically uncharacterized and practically the bottleneck for label-free unlearning.

Problem Statement

For approximate machine unlearning reformulated as representation-space relocation (the ManiF-SMC setting: push each erased sample's representation away from its original class-manifold centroid toward the centroid of its kk nearest retained semantic neighbors, with margins generated by a self-mode-connectivity module), address the open challenge stated by the source paper: make label-free, representation-space unlearning a faithful and practical substitute for retraining by mitigating the utility loss (degradation of retained and test accuracy) that purely representation-space methods suffer relative to methods fine-tuning on the remaining data as the unlearning sample size grows, without reintroducing dependence on labels or task-specific gradients. Understanding how learned representation geometry — the position of erased-sample representations relative to retained-data manifolds, and the degree of manifold separability, e.g., as shaped by Maximum Manifold Capacity Representations — relates to unlearning fidelity, meaning closeness to the behavior of a model retrained from scratch on the retained data, is relevant explanatory context that such a mechanism may exploit; the source paper's open question is the utility-preserving, label-free mitigation itself.

References

  1. Weiqi Wang, Zhiyi Tian, Chenhan Zhang, Luoyu Chen, Shui Yu (2026). "Approximate Machine Unlearning through Manifold Representation Forgetting Guided by Self Mode Connectivity." arXiv:2605.22871 (v1, 20 May 2026; author version states "To appear at KDD 2026"), DOI 10.48550/arxiv.2605.22871, https://arxiv.org/abs/2605.22871
  2. Thomas Yerxa, Yilun Kuang, Eero Simoncelli, SueYeon Chung (2023). "Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations." Advances in Neural Information Processing Systems 36, arXiv:2303.03307, DOI 10.48550/arxiv.2303.03307, https://arxiv.org/abs/2303.03307
  3. Georgina Cosma, Axel Finke (2026). "RULER: Representation-Level Verification of Machine Unlearning." arXiv:2605.27569, DOI 10.48550/arxiv.2605.27569, https://arxiv.org/abs/2605.27569
  4. Teresa Pui Yee Yong, Win Kent Ong, Chee Seng Chan (2026). "Erased, but Not Gone: Output Forgetting Is Not True Forgetting." arXiv:2606.25001, DOI 10.48550/arxiv.2606.25001, https://arxiv.org/abs/2606.25001
  5. Yichen Gao, Altay Unal, Akshay Rangamani, Zhihui Zhu (2026). "An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations." arXiv:2604.08271, DOI 10.48550/arxiv.2604.08271, https://arxiv.org/abs/2604.08271
  6. Jiali Cheng, Hadi Amiri (2025). "Understanding Machine Unlearning Through the Lens of Mode Connectivity." arXiv:2504.06407, DOI 10.48550/arxiv.2504.06407, https://arxiv.org/abs/2504.06407
  7. Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai, Andres Carranza, Berivan Isik, Alyssa Unell, Mikail Khona, Thomas Yerxa, Yann LeCun, SueYeon Chung, Andrey Gromov, Ravid Shwartz-Ziv, Sanmi Koyejo (2024). "Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations." arXiv:2406.09366, DOI 10.48550/arxiv.2406.09366, https://arxiv.org/abs/2406.09366
  8. Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, Sijia Liu (2023). "SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation." arXiv:2310.12508, DOI 10.48550/arxiv.2310.12508, https://arxiv.org/abs/2310.12508
  9. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot (2021). "Machine Unlearning." 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159, DOI 10.1109/SP40001.2021.00019, https://doi.org/10.1109/SP40001.2021.00019