When do representation-space attacks transfer between neural networks? Alignment conditions beyond linear misalignment

Adversarial attacks optimized against one machine-learning model often transfer to other models, but Gupta et al. (2025) drew a fundamental distinction: attacks perturbing the shared input data-space (adversarial examples, text jailbreak suffixes) can transfer between models, whereas attacks perturbing a model's internal representation-space (soft-prompt jailbreaks, latent image perturbations) jailbreak the attacked model yet fail to transfer to independently trained models — unless the two models' latent geometries are sufficiently aligned. This data-space/representation-space dichotomy was proven only in an idealized setting where two functionally equivalent networks' representations differ by an invertible linear map (worst case a Haar-random rotation), in which representation-space transfer degenerates to a coin toss. The open problem is a predictive characterization of representation-space attack transfer for realistic models: extend the theory to the nonlinear, structured misalignment produced by different initializations, optimization trajectories, and data ordering; replace the heuristic similarity diagnostics (instance average cosine similarity, centered kernel alignment) with alignment measures provably sufficient (or necessary) for transfer; determine how model scale and representation convergence modulate transferability; and establish the corresponding empirical behavior in early-fusion vision-language architectures, with attack constructions stable enough that non-transfer is attributable to misalignment rather than optimization noise.

Background

An adversarial example is a small input perturbation that causes a machine-learning model to misbehave; when the target is a safety-aligned language model (LM) and the misbehavior is harmful output, it is called a jailbreak. A transfer attack is optimized on a surrogate model the attacker can access and then applied to a different target model — long known to work for adversarial examples on image classifiers and for text jailbreaks on LMs. Vision-language models (VLMs) generate text conditioned on images and text; in adapter-style VLMs a frozen vision encoder's features are mapped into the LM's embedding space by a trained projector, whereas early-fusion (native multimodal) models process images without a separate adapter. Schaeffer et al. (2024) and Rando et al. (2024) independently found that gradient-based universal image jailbreaks — optimized on one VLM or an ensemble — jailbreak the attacked model(s) but show little or no transfer to other VLMs (adapter-style and early-fusion Chameleon models, respectively). Gupta et al. (2025) explained this by distinguishing the operational domain of an attack. A data-space attack perturbs the input (pixels, tokens), which all models share; a representation-space attack instead adds a perturbation $\delta$ directly to a model's internal activations, e.g. a soft prompt (a continuous, non-token embedding vector prepended to the input) or an optimized latent image vector injected after the VLM projector. Their theoretical setting models each network as a linear readout $f(x)=w\cdot\phi(x)$ over a feature map $\phi$ (as in kernel regression, where predictions are linear in learned representation features); a second, functionally equivalent network uses $\tilde\phi(x)=Q^{-1}\phi(x)$, $\tilde w=Q^{T}w$ for an invertible linear map $Q$. Data-space attacks then transfer perfectly, while a representation-space attack harms both models equally only if $w^{T}Q=w$, i.e. only under geometric alignment of the two representation spaces; if $Q$ is a Haar-random orthogonal matrix (uniform over rotations, independent of the attack), the sign of the induced harm on the target matches the attacked model's with probability exactly $1/2$ and exact-equal harm never occurs — no better than chance. Empirically they confirm the dichotomy: soft-prompt attacks jailbreak attacked LMs but do not transfer across independently trained LMs, while transferring between fine-tunes of a shared base model whose latent geometries are highly similar; textual (data-space) jailbreak suffixes optimized by the GCG method of Zou et al. (2023) — gradient-guided discrete token search — do transfer between VLMs; and latent image jailbreaks optimized post-projector transfer across VLMs sharing a language backbone. Similarity was measured with heuristic diagnostics: instance-wise average cosine similarity (AvgCosine) between matched representations, and centered kernel alignment (CKA) (Kornblith et al. 2019), a popular second-order statistic of representation similarity. The authors flag the limits of these results: the theory covers only invertible-linear (worst case random-rotation) misalignment, whereas real networks trained from different initializations, optimization trajectories, and data orderings differ by more complex nonlinear or structured transforms; the similarity metrics are heuristic and not proven sufficient to capture attack transfer; and, since emerging work (Huh et al. 2024; Jha et al. 2025) argues that internal representations converge as models scale (the Platonic Representation Hypothesis), whether one-size-fits-all representation-space attacks become broadly viable at scale is undetermined.

Problem Statement

The problem is to characterize, theoretically and empirically, the conditions under which a representation-space attack optimized against one model transfers to another model — replacing the proven-but-idealized linear-misalignment result of Gupta et al. (2025) with a predictive account of realistic model pairs. Concretely, with the common objective of a validated transferability criterion for representation-space attacks, the following linked questions are open. (1) Theory beyond linear misalignment: the known result covers two functionally equivalent networks whose representations differ by an invertible linear map $Q$ (transfer of equal harm occurs only under the alignment condition $w^{T}Q=w$, and is chance-level for Haar-random orthogonal $Q$); characterize transfer for the nonlinear and structured misalignment that arises between real networks trained from different initializations, optimization trajectories, or data orderings — for which model or misalignment classes does representation-space attack transfer fail, and when does it succeed? (2) Provable alignment conditions: the diagnostics used (instance AvgCosine, CKA) are heuristic and not proven sufficient; identify similarity or alignment measures, defined on accessible internal representations, that provably predict (sufficiently, and ideally necessarily) whether representation-space attacks transfer between a given model pair, including minimal sufficient conditions beyond geometric alignment (e.g. shared training artifacts, matched fine-tuning objectives, constrained projector architectures). (3) Scale dependence: determine whether representation convergence with model scale (per the Platonic Representation Hypothesis) makes one-size-fits-all representation-space attacks broadly transferable, or whether it does not. (4) Empirical boundary in architecture families: establish whether the observed transfer properties (post-projector non-transfer, shared-backbone re-alignment, fine-tune transfer) hold for early-fusion or native-multimodal VLMs — with attack constructions stable enough across seeds and optimization hyperparameters that non-transfer can be attributed to representation misalignment rather than optimization failure. The characterization sought is a predictive account of when representation-space attacks do and do not transfer between a given pair of models, resting on theorems for the misalignment or model classes it treats and on empirically validated predictive criteria. The four linked questions (1)–(4) above are facets of that single characterization.

References

1. Isha Gupta, Rylan Schaeffer, Joshua Kazdan, Ken Ziyu Liu, Sanmi Koyejo. Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed. arXiv:2510.01494 (2025). DOI: 10.48550/arXiv.2510.01494. https://arxiv.org/abs/2510.01494
2. Rylan Schaeffer, Dan Valentine, Luke Bailey, James Chua, Cristóbal Eyzaguirre, Zane Durante, Joe Benton, Brando Miranda, Henry Sleight, John Hughes, Rajashree Agrawal, Mrinank Sharma, Scott Emmons, Sanmi Koyejo, Ethan Perez. Failures to Find Transferable Image Jailbreaks Between Vision-Language Models. arXiv:2407.15211 (2024). DOI: 10.48550/arXiv.2407.15211. https://arxiv.org/abs/2407.15211
3. Javier Rando, Hannah Korevaar, Erik Brinkman, Ivan Evtimov, Florian Tramèr. Gradient-based Jailbreak Images for Multimodal Fusion Models. arXiv:2410.03489 (2024). DOI: 10.48550/arXiv.2410.03489. https://arxiv.org/abs/2410.03489
4. Rico Angell, Jannik Brinkmann, He He. Jailbreak Transferability Emerges from Shared Representations. arXiv:2506.12913 (2025). DOI: 10.48550/arXiv.2506.12913. https://arxiv.org/abs/2506.12913
5. Mengqi He, Xinyu Tian, Xin Shen, Shu Zou, Jinhong Ni, Zhaoyuan Yang, Weikang Li, Xuesong Li, Jing Zhang. Break the Brake, Not the Wheel: Untargeted Jailbreak via Entropy Maximization. arXiv:2605.10764 (2026). DOI: 10.48550/arXiv.2605.10764. https://arxiv.org/abs/2605.10764
6. Ruofan Wang, Xin Wang, Yang Yao, Juncheng Li, Xuan Tong, Xingjun Ma. Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models. arXiv:2508.01741 (2025). DOI: 10.48550/arXiv.2508.01741. https://arxiv.org/abs/2508.01741
7. Minyoung Huh, Brian Cheung, Tongzhou Wang, Phillip Isola. The Platonic Representation Hypothesis. arXiv:2405.07987 (2024). DOI: 10.48550/arXiv.2405.07987. https://arxiv.org/abs/2405.07987
8. Rishi Jha, Collin Zhang, Vitaly Shmatikov, John X. Morris. Harnessing the Universal Geometry of Embeddings. arXiv:2505.12540 (2025). DOI: 10.48550/arXiv.2505.12540. https://arxiv.org/abs/2505.12540
9. Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton. Similarity of Neural Network Representations Revisited. arXiv:1905.00414 (2019). DOI: 10.48550/arXiv.1905.00414. https://arxiv.org/abs/1905.00414
10. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043 (2023). DOI: 10.48550/arXiv.2307.15043. https://arxiv.org/abs/2307.15043
11. Wenpeng Xing, Bohan Yang, Mohan Li, Chunqiang Hu, Haitao Xu, Ningyu Zhang, Bo Lin, Meng Han. Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs. arXiv:2508.10029 (2025). DOI: 10.48550/arXiv.2508.10029. https://arxiv.org/abs/2508.10029
12. Ege Çakar, Hannah Guan, Kayden Kehe. Optimizing Against Safety Representations: Activation-Guided Adversarial Suffixes and the Geometry of Refusal. arXiv:2607.08883 (2026). DOI: 10.48550/arXiv.2607.08883. https://arxiv.org/abs/2607.08883
13. Wenyun Li, Guiping Cao, Xiangyuan Lan, Zheng Zhang. Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models. arXiv:2608.22312 (2026). DOI: 10.48550/arXiv.2608.22312. https://arxiv.org/abs/2608.22312