Mukherjee, Ibrayev, and Roy (TMLR 2025; arXiv:2404.00185) reported that two active vision classifiers, GFNet and FALcon, which infer from small downsampled glimpses taken at adaptively chosen fixation points, are 2-3 times more robust than a passive convolutional network under state-of-the-art black-box transfer attacks, and framed this as 'inherent robustness' of active sensing. The authors explicitly left open whether this robustness survives white-box (adaptive) threat models tailored to glimpse-and-fixation pipelines, and whether adversarial training and other established defenses can be combined with active architectures. To date, adjacent foveation and active-vision efforts (FocL, FDT, SAFER-AiD, Embodied Active Defense) address different threat models, architectures, or defense families. The problem remains open: characterize the adversarial robustness of glimpse-and-fixation classifiers under white-box adaptive attacks, and determine whether adversarial training can be adapted to such active vision pipelines.

Background

Deep neural network image classifiers are vulnerable to adversarial examples: small, carefully crafted perturbations of the input that flip the predicted label. Robustness conclusions depend critically on the threat model. In a black-box transfer attack, the adversary cannot query or inspect the target model and instead crafts perturbations against a surrogate network, then transfers them. In a white-box (adaptive) attack, the adversary knows the target's architecture and parameters and computes gradients through the full inference procedure; a defense evaluated only under transfer attacks is routinely shown to be far weaker once a genuinely adaptive attack is applied.

Active vision systems borrow two mechanisms from primate vision: foveation, i.e., processing the scene with high resolution only near the current gaze point and progressively lower resolution toward the periphery, and saccades, i.e., discrete gaze shifts between fixation points. GFNet (Wang et al., NeurIPS 2020) is a glance-and-focus classification network that replaces full-resolution input with a small downsampled glimpse, recurrently selects where to look next, and accumulates evidence over several glimpses. FALcon (Ibrayev et al., Gaze Meets ML Workshop 2023) is a foveation-and-saccade architecture for weakly-supervised localization from a sequence of downsampled glimpses, with independent models for localization (fLf_L) and classification (fCf_C). Both perform inference from a few low-resolution patches rather than from every pixel at once.

Mukherjee, Ibrayev, and Roy studied the adversarial robustness of GFNet and FALcon under a vanilla black-box transfer setting and found 2-3 times greater robustness, measured as accuracy under transferred attacks, than a passive ResNet baseline, with visualizations suggesting that inference from distinct fixation points makes the models less vulnerable. They also included an adversarially trained ResNet50 as an oracle (Adv-T*): its accuracy under transferred attacks stays nearly flat, around 47%, across all attack types and strengths, whereas every other target, including the active models, degrades as attacks strengthen — the active models beat the passive baseline everywhere and even exceed the oracle under the weaker attacks, but fall to or below it under the stronger ones, while Adv-T* pays for its flat robustness with much lower clean accuracy (47.9% versus 72.97-75.88% for FALcon and GFNet). Observing this, the authors suggest "adapting adversarial training methods into active vision methods", i.e., whether adversarial training, the standard defense of training on adversarially perturbed examples, could be combined with active architectures. Precursor and adjacent lines had examined related but distinct questions: foveation-based preprocessing defenses (Luo et al. 2015), training passive classifiers on foveated images (RBlur, Shah and Raj 2023), and the robustness of retina-like recurrent models (Choi et al. 2022) — none of which studies white-box attacks on, or adversarial training of, glimpse-and-fixation classification pipelines. The unresolved question is whether the reported inherent robustness of active vision is a genuine property of active sensing or an artifact of a weak (non-adaptive) threat model, and how trained defenses interact with active sensing.

Problem Statement

For glimpse-and-fixation active vision classifiers such as GFNet and FALcon, whose inference consists of a sequence of small downsampled glimpses taken at adaptively chosen fixation points, the source paper reported 2-3 times greater robustness than a passive convolutional network, but only under a vanilla black-box transfer threat model. Two coupled questions remain open:

  1. White-box adaptive robustness. Does the reported inherent robustness of glimpse-and-fixation classifiers survive a genuinely adaptive white-box threat model in which the attacker has full knowledge of the active pipeline, including the glimpse-selection (fixation) policy, and can tailor the attack to the architecture? Addressing it requires designing and evaluating specialized white-box attacks against the full active inference procedure — including attacks that manipulate or exploit the fixation-selection mechanism itself, not only the per-glimpse classification — and comparing the resulting robustness against the black-box transfer baseline, establishing how much of the reported 2-3x robustness is genuine and how much is an artifact of attacker ignorance.

  2. Adapting trained defenses to active sensing. Can adversarial training or comparable defense strategies be adapted to glimpse-and-fixation architectures, i.e., can the glimpse classifier and/or the fixation policy be trained adversarially, and how does the resulting active-plus-trained-defense combination compare, in robustness and clean accuracy, both to the undefended active models of the source paper and to adversarially trained passive baselines such as the Adv-T* oracle, whose transfer-attack accuracy stays nearly flat across attack strengths at the cost of much lower clean accuracy?

The auxiliary directions the authors list — fine-tuning FALcon's classifier on downsampled glimpses, inference-time strategies for suggesting safe fixation points or rectifying predictions from fixation diagnostics, and broader transferability analysis across surrogate/target families, tasks, and scales — are design and protocol considerations within this objective, not separate deliverables.

References

  1. Amitangshu Mukherjee, Timur Ibrayev, Kaushik Roy. On Inherent Adversarial Robustness of Active Vision Systems. Transactions on Machine Learning Research, 2025; arXiv:2404.00185, DOI 10.48550/arXiv.2404.00185, https://arxiv.org/abs/2404.00185 (OpenReview: https://openreview.net/forum?id=iVV7IzI55V)
  2. Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, Gao Huang. Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification. NeurIPS 2020; arXiv:2010.05300, DOI 10.48550/arXiv.2010.05300, https://arxiv.org/abs/2010.05300
  3. Timur Ibrayev, Manish Nagaraj, Amitangshu Mukherjee, Kaushik Roy. Exploring Foveation and Saccade for Improved Weakly-Supervised Localization. Proceedings of the 2nd Gaze Meets ML Workshop (NeurIPS 2023), PMLR 226:61-89, published 2024, https://proceedings.mlr.press/v226/ibrayev24a.html
  4. Yian Li, Xiaoyu Guo, Hao Zhang, Shuiwang Li, Xiaowei Dai. Active Visual Perception: Opportunities and Challenges. arXiv:2512.03687, 2025, DOI 10.48550/arXiv.2512.03687, https://arxiv.org/abs/2512.03687
  5. Amitangshu Mukherjee, Deepak Ravikumar, Kaushik Roy. From Clutter to Clarity: Visual Recognition through Foveated Object-Centric Learning (FocL). Transactions on Machine Learning Research, 2026, OpenReview forum kVS7sMlv7P, https://openreview.net/forum?id=kVS7sMlv7P (DBLP record: https://dblp.org/rec/journals/tmlr/MukherjeeRR26)
  6. Ibrahim Batuhan Akkaya, Kishaan Jeeveswaran, Bahram Zonooz, Elahe Arani. Foveation-Guided Dynamic Token Selection for Robust and Efficient Vision Transformers. arXiv:2607.09480, 2026, DOI 10.48550/arXiv.2607.09480, https://arxiv.org/abs/2607.09480
  7. Jiayang Liu, Daniel Tso, Yiming Bu, Qinru Qiu. SAFER-AiD: Saccade-Assisted Foveal-peripheral vision Enhanced Reconstruction for Adversarial Defense. arXiv:2510.08761, 2025, DOI 10.48550/arXiv.2510.08761, https://arxiv.org/abs/2510.08761
  8. Lingxuan Wu, Xiao Yang, Yinpeng Dong, Liuwei Xie, Hang Su, Jun Zhu. Embodied Active Defense: Leveraging Recurrent Feedback to Counter Adversarial Patches. arXiv:2404.00540, 2024, DOI 10.48550/arXiv.2404.00540, https://arxiv.org/abs/2404.00540