Unresolved adversarial-robustness questions for dense associative memory classifiers: role of the full training rule, adversarial training, convolutional extension, and interaction-power regime

Deep neural networks built from rectified linear units (ReLU) are vulnerable to adversarial examples — inputs misclassified after small, human-imperceptible perturbations — and to rubbish images, unrecognizable patterns classified with high confidence. Krotov and Hopfield (Neural Computation 30(12):3151–3167, 2018) showed that dense associative memory (DAM) classifiers, attractor networks whose energy function contains higher-than-quadratic interactions, largely resist both pathologies on MNIST in a large-but-finite interaction-power regime, but their paper explicitly leaves the robustness program incomplete. Its Discussion leaves open the training-rule observation, its two untested extension proposals, and a linked regime question grounded in the paper's closing remark that the promising computational regime it identified remains little investigated: (1) a comprehensive account of which components of the full DAM training algorithm and energy-based update rule, beyond the mere replacement of ReLU activations by higher rectified polynomials, are responsible for the observed robustness; (2) whether complementing DAM training with adversarial training simultaneously improves adversarial robustness and clean accuracy, as the authors conjectured; (3) the behavior of higher-order DAMs trained by this algorithm in convolutional architectures under adversarial and rubbish inputs; and (4) how the favorable large-but-finite interaction-power regime identified on MNIST behaves on at least one task more complex than MNIST, including the observed accuracy–robustness trade-off across $n$ and the dependence of the network's reported confidence on $n$.

Background

Two failure modes of supervised deep neural networks (DNNs) motivated this line of work. An adversarial example is an input formed by adding a small, typically human-imperceptible perturbation chosen by gradient-based search on the input so that a correctly classified image is misclassified; the phenomenon was given both a linearity-based explanation and an effective mitigation — adversarial training, i.e., augmenting the training set with adversarial examples generated during training — by Goodfellow, Shlens and Szegedy (arXiv:1412.6572). A rubbish image (fooling image) is an unrecognizable noise pattern obtained by minimizing the training objective directly in pixel space, which the network nevertheless classifies with high confidence. Both pathologies transfer across models with different architectures and training sets, enabling black-box attacks, and both were argued to reflect the behavior of networks built from rectified linear units (ReLU, the activation returning $\max(0,x)$).

Dense associative memory (DAM, also called a modern Hopfield network) offers a different starting point. A classical Hopfield network stores patterns as attractors of a quadratic energy function over binary neurons and holds only about $0.14d$ patterns for $d$ neurons. DAM generalizes the energy to higher-order (higher-than-quadratic) interactions: with $K$ memory vectors $\xi^\mu$ stored in the weights and neuron activities $V_i$, the energy is $E = -\sum_{\mu=1}^{K} F_n(\sum_i \xi_i^\mu V_i)$, where the interaction function is a fast-growing nonlinearity — in the works studied here the rectified polynomial $F_n(x) = x^n$ for $x \ge 0$ and 0 otherwise — and $n$ is called the power of the interaction vertex. Storage capacity then grows as a power law $d^{n-1}$ in the number of feature neurons, and the attractor dynamics performing pattern completion doubles as a classifier. Krotov and Hopfield (arXiv:1606.01164) established a duality between these attractor models and feedforward networks: a DAM with power $n$ corresponds to a layered network whose activation functions are higher rectified polynomials, with $n = 2$ recovering ReLU networks. The same work showed that as $n$ increases the learned representation shifts from feature-based (each class represented through many local features) to prototype-based (each class represented by few class-typical patterns) — the two extreme regimes of pattern recognition from cognitive psychology — and that a broad range of intermediate powers is already in the prototype regime while retaining classification accuracy competitive with ReLU DNNs on MNIST, the standard benchmark of 70,000 handwritten digit images. A companion line (Ramsauer et al., arXiv:2008.02217) connected the exponential-interaction limit of these models to the attention mechanism of transformer networks, renewing broad interest in this model class.

Krotov and Hopfield's Neural Computation paper (DOI 10.1162/neco_a_01143) then showed, on MNIST classifiers trained by their energy-based algorithm, that in the large-but-finite-$n$ regime: minima of the training objective are free of rubbish images; artificial patterns generated at the decision boundary look genuinely ambiguous to human subjects; and adversarial images crafted against small-$n$ (ReLU-equivalent) models fail to transfer to large-$n$ models, suggesting higher-order models as detectors of adversarial attacks.

The paper's Discussion, however, records that the program is incomplete. The duality above suggests the robustness might come simply from the nonlinearity of the activation; the authors tested this by swapping ReLUs for higher rectified polynomials in ordinary DNNs and found these networks better than ReLU DNNs from the adversarial perspective but worse than full DAMs using their energy-based update rule, concluding that other aspects of the training algorithm are important and that these observations need further comprehensive investigation. They further proposed, but did not test, two extensions: complementing the training procedure with adversarial training on a mixture of clean and adversarial images, which they expected to increase both robustness and clean accuracy; and combining the training algorithm with convolutional layers, noting they had deliberately used fully connected architectures because adversarial problems are naively expected to be more severe there. Finally, they stress that more work is required to fully resolve the problem of adversarial and rubbish images, and that the promising computational regime they identified on MNIST remains little investigated — with the observed trade-off that larger $n$ slightly reduces clean accuracy while increasing robustness, and with the network's reported confidence — defined there as the softmax probability of the most probable label, averaged over test images — varying with $n$.

Problem Statement

For dense associative memory (DAM) classifiers with energy $E = -\sum_{\mu=1}^{K} F_n(\sum_i \xi_i^\mu V_i)$ — where $V_i$ is the activity of the $i$-th neuron, the vectors $\xi^\mu$ ($\mu = 1,\dots,K$, with components $\xi_i^\mu$) are the memory patterns stored in the weights, and the interaction function $F_n$ is the rectified polynomial $F_n(x) = x^n$ for $x \ge 0$ and $0$ otherwise — characterize and complete the adversarial-robustness program that Krotov and Hopfield (2018) opened, by answering the following four linked questions: items 1–3 restate the open observation and the two untested proposals of the source's Discussion, and item 4 develops the Discussion's closing remark that the promising computational regime identified there remains little investigated. In their classifier each memory additionally carries a recognition part coupled to the classification neurons, the class is read out through a one-step update rule whose argument is the difference of two DAM energies, and the weights are trained by backpropagation on a classification objective $\sum (Y_\alpha - t_\alpha)^{2m}$ whose even exponent $2m$ emphasizes the largest training errors, with the effective temperature of the update nonlinearity annealed during training.

1. Mechanism. Which components of the full training algorithm and energy-based update rule — as opposed to the higher rectified polynomial activation alone — are responsible for robustness to adversarial and rubbish inputs, and why? A comprehensive account must explain the reported empirical gap whereby DNNs whose ReLUs are replaced by higher rectified polynomials improve adversarial behavior yet remain worse than full DAMs, and must identify whether the load-bearing ingredients are the energy-based update rule itself, the error-emphasizing training objective with its exponent and temperature schedule, the prototype-based representation, or other aspects of the algorithm.

2. Adversarial training. Does complementing the DAM training procedure with adversarial training — training on a mixture of clean and adversarially perturbed images in the manner of Goodfellow, Shlens and Szegedy — further increase robustness to adversarial examples and simultaneously increase clean-image classification accuracy, as the source conjectured?

3. Convolutional architectures. How do higher-order DAMs trained by this algorithm behave in the convolutional setting, where dense connections are replaced by locally connected, weight-shared layers, with respect to adversarial and rubbish inputs (the source deliberately used fully connected networks and left this untested)?

4. Operating regime. The source closed by observing that more work is required to fully resolve the problem of adversarial and rubbish images, and that the promising computational regime it identified — large-but-finite $n$ on MNIST, where larger $n$ slightly reduces clean accuracy while increasing robustness, and where the network's reported confidence (the softmax probability of the most probable label) varies with $n$ — remains little investigated. How does this regime extend to at least one task more complex than MNIST, and how do the accuracy–robustness trade-off and the reported confidence depend on $n$ there?

The four questions above are linked: together they address when and why higher-order associative memories resist adversarial and rubbish inputs, and how this behavior depends on the interaction power $n$, the architecture, and the training scheme.

References

1. Dmitry Krotov and John Hopfield, Dense Associative Memory Is Robust to Adversarial Inputs, Neural Computation 30(12):3151–3167, 2018. DOI: 10.1162/neco_a_01143, https://doi.org/10.1162/neco_a_01143 (preprint arXiv:1701.00939, https://arxiv.org/abs/1701.00939)
2. Dmitry Krotov and John J. Hopfield, Dense Associative Memory for Pattern Recognition, Advances in Neural Information Processing Systems 29 (NIPS 2016). arXiv:1606.01164, https://arxiv.org/abs/1606.01164
3. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and Harnessing Adversarial Examples, ICLR 2015. arXiv:1412.6572, https://arxiv.org/abs/1412.6572
4. Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter, Hopfield Networks is All You Need, ICLR 2021. arXiv:2008.02217, https://arxiv.org/abs/2008.02217
5. Dmitry Krotov and John Hopfield, Large Associative Memory Problem in Neurobiology and Machine Learning, ICML 2020. arXiv:2008.06996, https://arxiv.org/abs/2008.06996
6. Dmitry Krotov, Hierarchical Associative Memory, 2021. arXiv:2107.06446, https://arxiv.org/abs/2107.06446
7. Vijendran G. Venkoparao, Saurav Musunuru, and Rahul Kumar Dubey, Impacts of adversarial inputs in associative memory models and its iterative learning variants, Proceedings of the 1st International Conference on Advanced Information Science and System (AISS 2019), ACM, 2019. DOI: 10.1145/3373477.3373698, https://doi.org/10.1145/3373477.3373698
8. Robin Thériault and Daniele Tantari, Dense Hopfield Networks in the Teacher-Student Setting, SciPost Physics 17.2.040, 2024. DOI: 10.21468/SciPostPhys.17.2.040, https://doi.org/10.21468/SciPostPhys.17.2.040 (preprint arXiv:2401.04191, https://arxiv.org/abs/2401.04191)
9. Robin Theriault, Explaining Machine Learning and Memorization with Statistical Mechanics (PhD thesis, defended January 2026), 2026. arXiv:2606.31110, https://arxiv.org/abs/2606.31110
10. Madhava Gaikwad, Algorithmic Analysis of Dense Associative Memory: Finite-Size Guarantees and Adversarial Robustness, 2026. arXiv:2604.12811, https://arxiv.org/abs/2604.12811
11. Robin Thériault and Daniele Tantari, Saddle Hierarchy in Dense Associative Memory, Machine Learning: Science and Technology 7, 015001, 2026. DOI: 10.1088/2632-2153/ae3051, https://doi.org/10.1088/2632-2153/ae3051 (preprint arXiv:2508.19151, https://arxiv.org/abs/2508.19151)
12. Ci Lin, Tet Yeap, and Iluju Kiringa, Robust Auto-associative Memory via Convolutional Restricted Hopfield Networks, 2026. arXiv:2606.20666, https://arxiv.org/abs/2606.20666
13. Ci Lin, Tet Yeap, Iluju Kiringa, and Biwei Zhang, Restricted Hopfield Networks are Robust to Adversarial Attack, TechRxiv preprint, 2025. DOI: 10.36227/techrxiv.173610613.34874972/v1, https://doi.org/10.36227/techrxiv.173610613.34874972/v1
14. Saleh Sargolzaei and Luis Rueda, Improving Out-of-Distribution Data Handling and Corruption Resistance via Modern Hopfield Networks, 2024. arXiv:2408.11309, https://arxiv.org/abs/2408.11309
15. Benjamin Hoover, Zhaoyang Shi, Krishnakumar Balasubramanian, Dmitry Krotov, and Parikshit Ram, Dense Associative Memory with Epanechnikov Energy, Proceedings of NeurIPS 2025. arXiv:2506.10801, https://arxiv.org/abs/2506.10801; DOI: 10.52202/085713-3947, https://doi.org/10.52202/085713-3947
16. Chandan Tankala and Krishnakumar Balasubramanian, Dense associative memory for Gaussian distributions, 2025. arXiv:2509.23162, https://arxiv.org/abs/2509.23162
17. Hayden McAlister, Anthony Robins, and Lech Szymanski, Improved Robustness and Hyperparameter Selection in the Dense Associative Memory, 2024. arXiv:2407.08742, https://arxiv.org/abs/2407.08742
18. Lorenzo Chicchi, Duccio Fanelli, Diego Febbe, Lorenzo Buffoni, Francesca Di Patti, Lorenzo Giambagli, and Raffaele Marino, Deterministic versus stochastic dynamical classifiers: opposing random adversarial attacks with noise, Machine Learning: Science and Technology, 2025. DOI: 10.1088/2632-2153/ae0244, https://doi.org/10.1088/2632-2153/ae0244 (preprint arXiv:2409.13470, https://arxiv.org/abs/2409.13470)
19. Elena Agliari, Francesco Alemanno, Adriano Barra, Martino Centonze, and Alberto Fachechi, Neural networks with redundant representation: detecting the undetectable, Physical Review Letters 124, 028301, 2020. DOI: 10.1103/PhysRevLett.124.028301, https://doi.org/10.1103/PhysRevLett.124.028301 (preprint arXiv:1911.12689, https://arxiv.org/abs/1911.12689)
20. Linda Albanese, Andrea Alessandrelli, Alessia Annibale, and Adriano Barra, Unsupervised and Supervised learning by Dense Associative Memory under replica symmetry breaking, 2023. arXiv:2312.09638, https://arxiv.org/abs/2312.09638
21. Pin-Yu Chen and Cho-Jui Hsieh, Adversarial robustness of beyond neural network models, chapter in Adversarial Robustness for Machine Learning, Elsevier, 2023. DOI: 10.1016/B978-0-12-824020-5.00027-2, https://doi.org/10.1016/B978-0-12-824020-5.00027-2