Extending unsupervised learning by competing hidden units beyond one fully connected layer: depth iteration, scale, biological realism, and theory of the fast heuristic

Krotov and Hopfield (PNAS, 2019) introduced an unsupervised, purely bottom-up, local learning algorithm in which hidden units compete through global inhibition and a Hebbian/anti-Hebbian plasticity rule with a Lebesgue-p-norm homeostatic constraint; trained on one fully connected hidden layer, it matched end-to-end backpropagation on MNIST and stayed close on CIFAR-10. The paper explicitly left open whether this algorithm can be extended along five linked directions: iterating it to stacked deep architectures, scaling it to convolutional structures and much larger datasets, making it more biologically realistic without performance loss, providing a theoretical account of its fast rank-based heuristic implementation, and combining it with standard training enhancements. Later literature has partially answered the depth and scale questions for the wider family of competitive Hebbian rules (convolutional local learning at scale by the same group; multilayer SoftHebb, a stacked soft winner-take-all Hebbian rule, reaching five unsupervised layers and ImageNet; ImageNet-scale Hebbian training), but no known work directly iterates the exact hard-winner-take-all anti-Hebbian rule into deep stacked architectures, none analyzes the fast ranking heuristic theoretically, and the biological-realism and training-enhancement questions for this specific algorithm remain untreated. The problem is therefore open.

Background

Error backpropagation trains artificial neural networks by propagating a global error signal backwards, which is widely viewed as biologically implausible: the update of a synapse depends on the states of other, distant neurons (nonlocality) and on labels, whereas biological synaptic plasticity is local, depending mainly on pre- and postsynaptic activity. Hebbian plasticity (Donald Hebb's proposal that coactive pre- and postsynaptic neurons strengthen their synapse) is local and unsupervised, but by itself does not produce a diverse set of useful feature detectors. Winner-take-all (WTA) competition, implemented biologically by lateral inhibition within a layer, makes neurons specialize to different patterns. Classical frameworks include Oja's rule (a Hebbian rule with weight normalization that extracts principal components) and the Bienenstock–Cooper–Munro (BCM) theory of receptive-field development, which uses a floating modification threshold to alternate synaptic strengthening (long-term potentiation, LTP) and weakening (long-term depression, LTD).

Krotov and Hopfield (2019) combined these ingredients into a single unsupervised algorithm for the first hidden layer of a feedforward network. For a hidden unit with weights $W_i$, inputs $v_i$, and inner product $\langle W, v\rangle$ weighted by a metric set by the Lebesgue $p$-norm ($p \ge 2$), the plasticity rule is $\tau_L \, dW_i/dt = g(Q)\bigl(R^p v_i - \langle W, v\rangle W_i\bigr)$ with $Q = \langle W, v\rangle / \langle W, W\rangle^{(p-1)/p}$. The first term is Hebbian or anti-Hebbian depending on the sign of the learning activation function $g$; the second term drives the weight vector onto an $L_p$ sphere of radius $R$ (a homeostatic constraint on synaptic strength). Competition is supplied by global inhibition between the $K$ hidden units, whose activities relax to a steady state of inhibitory dynamics before each update. The key design choice is that the most strongly driven units receive a Hebbian update ($g = 1$) while moderately driven units receive an anti-Hebbian update ($g = -\Delta$, $\Delta > 0$), which pushes units away from patterns already claimed by others and produces diverse feature detectors; in the limit $\Delta = 0$ the rule reduces to classical competitive learning. After this unsupervised phase, the top (readout) layer is trained by ordinary supervised stochastic gradient descent on the frozen hidden representation. On MNIST (a standard benchmark dataset of 70,000 grayscale images of handwritten digits, here flattened to 784 input pixels; network architecture 784 $\to$ 2000 $\to$ 10) this reached 1.46% test error, equal to the same-size network trained end to end; on fully connected CIFAR-10 (a benchmark dataset of 60,000 small color photographs in ten object classes) it reached 49.25% test error versus 44.74% for end-to-end training. The authors also introduced a fast implementation that replaces the costly numerical relaxation of the inhibitory dynamics by a heuristic: the input currents are used to rank the units, the top-ranked unit is updated with $g = 1$, the $k$-th ranked unit with $g = -\Delta$, and all others are left unchanged, which also permits minibatch training (updating weights from batches of examples rather than one example at a time). The authors state that this heuristic 'works extremely well in practice' but give no theoretical justification for it.

The paper's Discussion states that all experiments used a single hidden layer, that the algorithm is 'iterative in nature' (layer 1 can be learned, frozen, used to encode the inputs, and the same unsupervised algorithm applied again to train layer 2), and that 'it is unclear whether or not the proposed algorithm can lead to improvements on the single hidden-layer network, if applied iteratively in deeper architectures.' The Discussion further notes that all experiments were on MNIST and CIFAR-10 in the fully connected setting and that many biologically plausible approaches that work on these datasets 'fail on more complicated datasets, like ImageNet' (a large-scale benchmark of over a million labeled natural images spanning roughly a thousand object categories). The paper's Introduction lists known aspects of neurobiology that the algorithm omits: neurons make either excitatory or inhibitory outgoing synapses but not both (Dale's law), a constraint the algorithm explicitly violates, along with spike-timing-dependent plasticity (STDP), quantized synaptic changes, and detailed homeostatic mechanisms. Finally, observing that backpropagation networks gain substantially from dropout, noise injection, and data augmentation, the authors state that 'additional work is required' to see whether such techniques affect the proposed algorithm. These five limitations jointly define the question of the algorithm's capability envelope beyond the demonstrated regime.

Problem Statement

Determine, by systematic experiment and/or theory, the capability envelope of the unsupervised local learning algorithm of Krotov and Hopfield (2019) — the competing-hidden-units rule with global inhibition, a Hebbian/anti-Hebbian learning activation function, and an $L_p$-norm homeostatic weight constraint — beyond the regime demonstrated in the source paper (one fully connected hidden layer, MNIST and CIFAR-10, labels used only to train the readout layer). The common objective uniting the following linked questions, posed by the source paper itself, is to establish whether this label-free, purely bottom-up, local plasticity scheme extends to deeper and larger-scale learning and to a principled understanding of its own heuristics without losing its competitive performance:

(1) Depth iteration: can the algorithm be applied iteratively — learn the first layer of weights unsupervised, freeze it, encode the inputs with it, and apply the same unsupervised algorithm to train a second and further layers — such that the resulting networks improve measurably over the single-hidden-layer network of equal or matched total capacity, rather than degrading or stagnating?

(2) Architecture and scale: does the algorithm remain competitive in architectures with spatial weight sharing or convolutional structure, and on substantially larger and more complex datasets (up to ImageNet-scale), given that many biologically plausible approaches that succeed on MNIST/CIFAR-10 fail on such datasets?

(3) Biological realism: how far can the algorithm be made more biologically realistic — respecting Dale's law (a neuron's outgoing synapses are all excitatory or all inhibitory), incorporating spiking neurons and spike-timing-dependent plasticity, quantized synaptic changes, and detailed homeostatic mechanisms — before its demonstrated empirical performance degrades, and which of these constraints can be absorbed with compensating design changes?

(4) Theory of the fast implementation: the practical speed of the algorithm rests on a heuristic that replaces the numerically solved steady state of the inhibitory dynamics by a ranking of the input currents and updates only the top-ranked unit (Hebbian, $g=1$) and the $k$-th ranked unit (anti-Hebbian, $g=-\Delta$); what is the theoretical justification for this approximation, its convergence properties, the limits of its approximation error relative to the full dynamical algorithm, and can it be improved or replaced by a principled alternative?

(5) Standard training enhancements: what is the quantified effect of the standard deep-learning training improvements — data augmentation, dropout, injected noise, denoising, architectural depth and regularization — when applied to this unsupervised-first procedure, and can they close the remaining gaps on harder datasets?

These questions concern extensions of the algorithm that retain its defining constraints: the hidden-layer training phase uses no labels and no top-down propagation of information, and synaptic updates remain local in the pre- and postsynaptic activities. The five linked questions are independent and each is separately resolvable, positively or negatively. In sub-questions (3), (4) and (5), the listed items enumerate the conditions to be covered rather than forming a sample.

References

1. Krotov, D., Hopfield, J. J. (2019). Unsupervised learning by competing hidden units. Proceedings of the National Academy of Sciences 116(16):7723-7731. DOI: 10.1073/pnas.1820458116. https://doi.org/10.1073/pnas.1820458116 (arXiv preprint: arXiv:1806.10181, https://arxiv.org/abs/1806.10181)
2. Grinberg, L., Hopfield, J., Krotov, D. (2019). Local Unsupervised Learning for Image Analysis. arXiv:1908.08993. https://arxiv.org/abs/1908.08993
3. Moraitis, T., Toichkin, D., Journé, A., Chua, Y., Guo, Q. (2021). SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks. arXiv:2107.05747. https://arxiv.org/abs/2107.05747 (journal version: Neuromorphic Computing and Engineering 2:044017, 2022, DOI 10.1088/2634-4386/aca710, https://doi.org/10.1088/2634-4386/aca710)
4. Journé, A., Garcia Rodriguez, H., Guo, Q., Moraitis, T. (2022). Hebbian Deep Learning Without Feedback. Published at the Eleventh International Conference on Learning Representations (ICLR) 2023. arXiv:2209.11883. https://arxiv.org/abs/2209.11883 (OpenReview: https://openreview.net/forum?id=8gd4M-_Rj1)
5. Shinozaki, T. (2021). Biologically motivated learning method for deep neural networks using hierarchical competitive learning. Neural Networks 144:271-278. DOI: 10.1016/j.neunet.2021.08.027. https://doi.org/10.1016/j.neunet.2021.08.027 (preprint, titled 'Biologically-Motivated Deep Learning Method using Hierarchical Competitive Learning': arXiv:2001.01121, https://arxiv.org/abs/2001.01121)
6. Stuhr, B., Brauer, J. (2019). CSNNs: Unsupervised, Backpropagation-free Convolutional Neural Networks for Representation Learning. Proceedings of the International Conference on Machine Learning and Applications (ICMLA) 2019. DOI: 10.1109/ICMLA.2019.00265. https://doi.org/10.1109/ICMLA.2019.00265 (extended version: arXiv:2001.10388, https://arxiv.org/abs/2001.10388)
7. Lagani, G., Falchi, F., Gennaro, C., Fassold, H., Amato, G. (2024). Scalable bio-inspired training of Deep Neural Networks with FastHebb. Neurocomputing 595:127867. DOI: 10.1016/j.neucom.2024.127867. https://doi.org/10.1016/j.neucom.2024.127867 (preprint, titled 'FastHebb: Scaling Hebbian Training of Deep Neural Networks to ImageNet Level': arXiv:2207.03172, https://arxiv.org/abs/2207.03172)
8. Hinton, G. E. (2022). The Forward-Forward algorithm: Some Preliminary Investigations. arXiv:2212.13345. https://arxiv.org/abs/2212.13345
9. Gupta, M., Modi, S. K., Zhang, H., Lee, J. H., Lim, J. H. (2022). Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backprop. arXiv:2212.04614. https://arxiv.org/abs/2212.04614
10. Illing, B., Gerstner, W., Brea, J. (2019). Biologically plausible deep learning — but how far can we go with shallow networks? Neural Networks 118:90-101. DOI: 10.1016/j.neunet.2019.06.001. https://doi.org/10.1016/j.neunet.2019.06.001 (preprint: arXiv:1905.04101, https://arxiv.org/abs/1905.04101)
11. Lagani, G., Falchi, F., Gennaro, C., Amato, G. (2023). Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Survey. arXiv:2307.16236. https://arxiv.org/abs/2307.16236
12. Stricker, P., Röhrbein, F., Knoblauch, A. (2024). Weight Perturbation and Competitive Hebbian Plasticity for Training Sparse Excitatory Neural Networks. Proceedings of the 2024 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. DOI: 10.1109/IJCNN60899.2024.10650478. https://doi.org/10.1109/IJCNN60899.2024.10650478
13. Jiménez Nimmo, J., Mondragón, E. (2025). Advancing the Biological Plausibility and Efficacy of Hebbian Convolutional Neural Networks. Neural Networks 190:107628. DOI: 10.1016/j.neunet.2025.107628. https://doi.org/10.1016/j.neunet.2025.107628 (preprint: arXiv:2501.17266, https://arxiv.org/abs/2501.17266)
14. Inoue, P., Röhrbein, F., Knoblauch, A. (2026). Guiding Sparse Neural Networks with Neurobiological Principles to Elicit Biologically Plausible Representations. arXiv:2603.03234. https://arxiv.org/abs/2603.03234
15. Lin, W., Piao, Z., Fung, C. C. A. (2025). Delving into Unsupervised Hebbian Learning from Artificial Intelligence Perspectives. Machine Learning and Knowledge Extraction 7(4):143. DOI: 10.3390/make7040143. https://doi.org/10.3390/make7040143