Realistic stereo statistics and multi-component PCA in unsupervised models of ocular dominance and disparity tuning (questions left open by Shouval, Intrator, Law & Cooper, 1996)

Shouval, Intrator, Law & Cooper (1996) showed, in a single-neuron model fed two-eye input from preprocessed natural images, that a fixed cortical misalignment of the left- and right-eye synaptic arbors produces graded ocular dominance under BCM (Bienenstock-Cooper-Munro) learning while single-component PCA (principal component analysis) learning always yields binocular oriented receptive fields; a parity-invariance argument on the two-eye correlation operator proves that every single principal component must be binocular. The study used single-camera 2D images without stereo statistics, and its symmetry argument covers only single components. Two questions are left open by this analysis and remain unaddressed for these learning rules: (i) how depth-dependent stereo image statistics change the learned distribution of best disparities and of ocular dominance under BCM and PCA, and whether PCA's single-component binocularity persists when inter-eye correlations are no longer even shifted functions; (ii) whether combinations of several principal components can yield monocular or imbalanced ocular dominance in a realistic binocular environment. Later literature approached the same territory with other rules (independent component analysis (ICA), sparse coding, active efficient coding) and found mostly binocular solutions under normal rearing, with monocularity increasing under simulated strabismus (a persistent misalignment of the two eyes), appearing in an eccentricity-dependent way for ICA on natural stereo patches, and remaining narrow in models that lack ocular-dominance-band input structure; but no work answers the original questions for BCM or PCA.

Background

Ocular dominance - the degree to which a neuron in primary visual cortex (V1) responds to one eye rather than to both - and binocular disparity - the small difference between the two eyes' retinal images of the same object, the cue for stereoscopic depth perception - develop together in V1 under normal visual experience. A long line of theoretical work asks which statistics of the two-eye visual input, combined with which synaptic plasticity rule, suffice to reproduce this development. Two families of unsupervised rules recur in that literature. The BCM rule (Bienenstock, Cooper and Munro, 1982) modifies a synapse according to the postsynaptic activity $c$ and a sliding modification threshold $\Theta_M=E[c^2]$ (expectation over the visual environment): the modification function $c(c-\Theta_M)$ potentiates synapses when activity exceeds the threshold and depresses them below it, so the converged solution depends on higher-than-second-order statistics of the input. Hebbian rules of the principal-component type, such as Oja's rule (Oja, 1982) $\Delta m_i=\eta(d_i c-c^2 m_i)$ (with $d_i$ the presynaptic input, $m_i$ the synaptic weight, and the $-c^2 m_i$ term providing normalization), instead converge to the leading eigenvector of the input correlation matrix and therefore depend only on second-order statistics. Earlier analyses had studied ocular-dominance development from the two-eye correlation structure itself: Li (1995) derived how ocular dominance can be understood from binocular input statistics, and Erwin and Miller (1999) characterized the inter-eye correlation structure that correlation-based Hebbian development requires to produce ocularly matched binocular simple cells.

Shouval, Intrator, Law and Cooper (1996) placed these rules in a realistic visual environment. A single model neuron received left- and right-eye inputs sampled from a corpus of natural images preprocessed by a difference-of-Gaussians filter (the difference of two spatial Gaussian filters, a standard model of retinal center-surround preprocessing), with the left- and right-eye synaptic arbors - the circular patches of input connections through which each eye feeds the neuron - displaced by a fixed cortical misalignment $\mathbf{s}$, parameterized by an overlap fraction. Simulations showed that misalignment affects the two rules differently: under BCM, misalignment alone produces the full graded range of ocular dominance while orientation selectivity is retained, whereas under PCA learning the dominant single component is always a binocular oriented receptive field (the receptive field is the retinal stimulus pattern to which the neuron responds most strongly; binocular means it weights the two eyes comparably), typically horizontal, reflecting a slight horizontal bias in the empirical two-dimensional correlation function. The paper then gave an analytic parity-invariance result for the PCA case. Writing the two-eye correlation operator in block form

$\mathbf{Q}=\begin{pmatrix}Q_{\mathrm{ll}}&Q_{\mathrm{lr}}\\Q_{\mathrm{rl}}&Q_{\mathrm{rr}}\end{pmatrix}$

translational invariance of single-eye statistics gives $Q_{\mathrm{ll}}(\mathbf r,\mathbf r')=Q(\mathbf r-\mathbf r')$ and between-eye entries $Q_{\mathrm{lr}}(\mathbf r,\mathbf r')=Q'(\mathbf r-\mathbf r'+\mathbf s)$; if the scalar functions $Q$ and $Q'$ are even - as holds for the two-eye statistics generated by sampling one 2D image with shifted patches - the two-eye parity operator $\mathbf P=\begin{pmatrix}0&P\\P&0\end{pmatrix}$ (with $P$ the coordinate inversion $\mathbf r\mapsto-\mathbf r$) satisfies $\mathbf{PQP}=\mathbf Q$, so every eigenvector $\mathbf m=(\mathbf m^{\mathrm l},\mathbf m^{\mathrm r})$ of $\mathbf Q$ is also an eigenvector of $\mathbf P$. This forces the left- and right-eye receptive-field profiles to be inverted copies of one another, i.e. perfectly binocular. Because the corpus consisted of single-camera 2D images, the learned receptive fields peaked at zero disparity (the focal plane), with BCM cells that became more monocular acquiring small nonzero best disparities (the best disparity of a receptive field is the interocular displacement at which its binocular response is maximal).

Two limitations of this analysis define the open questions. First, the environment contained no stereo statistics: real two-eye input has objects at many depths, whose images shift in opposite directions in the two eyes, so the inter-eye correlation is not an even shifted function and the parity argument's hypothesis fails; what BCM and PCA learning produce under such statistics was left unresolved. Second, the parity argument constrains each eigenvector separately, and models that combine several learned components are not covered by it; whether such multi-component solutions can be monocular or imbalanced in ocular dominance was likewise left unresolved.

Problem Statement

Determine how unsupervised synaptic plasticity rules of different statistical order develop ocular dominance and disparity tuning when the two-eye visual environment has realistic stereo statistics, in the two linked forms left open by Shouval, Intrator, Law & Cooper (1996):

(1) Stereo statistics: with two-eye input drawn from natural stereo images or a calibrated stereo simulator in an environment whose depth distribution is asymmetric about the fixation plane - objects in front of and behind the fixation plane shift in opposite directions in the two eyes, but with unequal aggregate weight, so that the between-eye correlation is not an even shifted function - what distribution of best disparities do BCM and PCA learning produce, and what happens to the ocular-dominance distributions - in particular, does the graded ocular dominance that BCM derives from cortical misalignment in 2D images survive, and does the parity-invariance guarantee that single principal components are binocular persist, or is it modified by stereo correlations?

(2) Multi-component PCA: the parity-invariance result applies only to single principal components; can combinations of several principal components - networks that learn or pool multiple components in the same realistic binocular environment - realize monocular or imbalanced ocular-dominance outcomes, or are they also constrained to binocular solutions?

Both enumerated forms are required, and answering one does not resolve the other. Part (1)'s best-disparity distribution, its ocular-dominance outcome, and its single-component parity question, and part (2) each count as separate sub-questions that must all be answered. Ocular dominance is measured throughout by the index $I=(L-R)/(L+R)$, where $L$ and $R$ are the total left- and right-eye response magnitudes of the readout in question (for a linear readout, of its weighted sum); $I$ lies in $[-1,1]$, equals 0 at two-eye balance, $+1$ at exclusive left-eye dominance, and $-1$ at exclusive right-eye dominance, and an output counts as binocular when $|I|\le 0.2$ and as monocular or imbalanced when $|I|>0.2$. Because the part (1) distributions depend on the depth statistics, the best-disparity and ocular-dominance answers must be quantitative and must either hold for every environment in the stated class or state explicitly how they depend on the depth distribution within that class, while the single-component parity question concerns every principal component in the class. For part (2), a multi-component model is a network that learns or pools two or more principal components from the same two-eye input, and its output is the fixed readout the model defines over those components: any fixed function of the component responses, linear or nonlinear (for a linear readout, the weighted sum, with weights of either sign). The output must depend jointly on at least two components, and the reported imbalance must require that combination: with the other components removed, no single constituent component, read out alone, may itself attain the reported imbalance. Part (2) is settled affirmatively by a single such multi-component model, in the specified environment, whose output is monocular or imbalanced, and otherwise requires a general statement, over all such multi-component models, that their outputs are constrained to be binocular.

The two parts serve one scientific objective: establishing which input statistics (two-eye correlations with and without depth structure) and which class of plasticity rule (single- vs. multi-component; second- vs. higher-order) can jointly account for the observed diversity of ocular dominance and the development of disparity tuning.

References

1. Shouval H, Intrator N, Law CC, Cooper LN (1996). Effect of Binocular Cortical Misalignment on Ocular Dominance and Orientation Selectivity. Neural Computation 8(5):1021-1040. DOI: 10.1162/neco.1996.8.5.1021. https://doi.org/10.1162/neco.1996.8.5.1021
2. Bienenstock EL, Cooper LN, Munro PW (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience 2(1):32-48. DOI: 10.1523/JNEUROSCI.02-01-00032.1982. https://doi.org/10.1523/JNEUROSCI.02-01-00032.1982
3. Oja E (1982). Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology 15(3):267-273. DOI: 10.1007/BF00275687. https://doi.org/10.1007/BF00275687
4. Li Z (1995). Understanding Ocular Dominance Development from Binocular Input Statistics. In: The Neurobiology of Computation. Springer. DOI: 10.1007/978-1-4615-2235-5_65. https://doi.org/10.1007/978-1-4615-2235-5_65
5. Erwin E, Miller KD (1999). The subregion correspondence model of binocular simple cells. Journal of Neuroscience 19(16):7212-7229. DOI: 10.1523/JNEUROSCI.19-16-07212.1999. https://doi.org/10.1523/JNEUROSCI.19-16-07212.1999
6. Hoyer PO, Hyvärinen A (2000). Independent component analysis applied to feature extraction from colour and stereo images. Network: Computation in Neural Systems 11(3):191-210. DOI: 10.1088/0954-898X/11/3/302. https://doi.org/10.1088/0954-898X/11/3/302
7. Blais B, Cooper LN, Shouval H (2000). Formation of Direction Selectivity in Natural Scene Environments. Neural Computation 12(5):1057-1066. DOI: 10.1162/089976600300015501. https://doi.org/10.1162/089976600300015501
8. Lee AB, Blais B, Shouval H, Cooper LN (2000). Statistics of lateral geniculate nucleus (LGN) activity determine the segregation of ON/OFF subfields for simple cells in visual cortex. Proceedings of the National Academy of Sciences 97(23):12875-12879. DOI: 10.1073/PNAS.97.23.12875. https://doi.org/10.1073/PNAS.97.23.12875
9. Siddiqui MSM, Bhaumik B (2011). A Reaction-Diffusion Model to Capture Disparity Selectivity in Primary Visual Cortex. PLoS ONE 6(10):e24997. DOI: 10.1371/journal.pone.0024997. https://doi.org/10.1371/journal.pone.0024997
10. Hunt JJ, Dayan P, Goodhill GJ (2013). Sparse Coding Can Predict Primary Visual Cortex Receptive Field Changes Induced by Abnormal Visual Input. PLoS Computational Biology 9(5):e1003005. DOI: 10.1371/journal.pcbi.1003005. https://doi.org/10.1371/journal.pcbi.1003005
11. Hunter DW, Hibbard PB (2018). The effect of image position on the Independent Components of natural binocular images. Scientific Reports 8:449. DOI: 10.1038/s41598-017-18460-1. https://doi.org/10.1038/s41598-017-18460-1
12. Zhang C, Triesch J, Shi BE (2016). An active-efficient-coding model of optokinetic nystagmus. Journal of Vision 16(14):10. DOI: 10.1167/16.14.10. https://doi.org/10.1167/16.14.10 (arXiv: 1606.06443)
13. Klimmasch L, Schneider J, Lelais A, Fronius M, Shi BE, Triesch J (2021). The development of active binocular vision under normal and alternate rearing conditions. eLife 10:e56212. DOI: 10.7554/eLife.56212. https://doi.org/10.7554/eLife.56212
14. Fumarola F, Hein B, Miller KD (2022). Mechanisms for Spontaneous Symmetry Breaking in Developing Visual Cortex. Physical Review X 12:031024. DOI: 10.1103/PhysRevX.12.031024. https://doi.org/10.1103/PhysRevX.12.031024
15. Herrera-Esposito D, Burge J (2025). Optimal Estimation of Local Motion-in-Depth with Naturalistic Stimuli. Journal of Neuroscience 45(8):e0490242024. DOI: 10.1523/jneurosci.0490-24.2024. https://doi.org/10.1523/jneurosci.0490-24.2024 (preprint: Herrera-Esposito D, Burge J (2024). Optimal motion-in-depth estimation with natural stimuli. bioRxiv. DOI: 10.1101/2024.03.14.585059. https://doi.org/10.1101/2024.03.14.585059)
16. Shouval H, Intrator N, Cooper LN (1997a). Effect of Binocular Cortical Misalignment on Networks of BCM and Oja Neurons. In: Computational Neuroscience (CNS*96). Springer, pp. 491-495. DOI: 10.1007/978-1-4757-9800-5_78. https://doi.org/10.1007/978-1-4757-9800-5_78
17. Shouval H, Intrator N, Cooper LN (1997b). BCM network develops orientation selectivity and ocular dominance in natural scene environment. Vision Research 37(23):3339-3342. DOI: 10.1016/S0042-6989(97)00087-4. https://doi.org/10.1016/S0042-6989(97)00087-4