Flexible, high-capacity models such as artificial neural networks (ANNs) are increasingly fitted to scientific data to discover mechanistic hypotheses, and the fitted model is conventionally selected by predictive generalization. Genkin and Engel (2020) demonstrated, in an intrinsically interpretable framework — one-dimensional Langevin latent dynamics (a stochastic differential equation combining deterministic drift with additive random noise) observed through spiking — that many models discovered during optimization lie on a generalization plateau: they predict held-out data equally well yet differ in their features, so validation-based selection can return spurious interpretations. They introduced an alternative selection principle, feature consistency, which compares features of matched complexity — complexity measured by the negative trajectory entropy of the inferred potential — across independent data samples, and they showed it recovers the correct interpretation where generalization-based selection fails. They explicitly acknowledged that developing analogous feature-complexity measures and reproducibility-based model-selection procedures for ANNs and other high-dimensional flexible models remains a significant outstanding issue. This extension remains open: follow-up work by the authors and others extends the intrinsically interpretable Langevin family itself, while on the ANN side the very substrate such a measure needs — a canonical, reproducible feature decomposition — has not been established, and no principled reproducibility-based selection criterion has been proposed.

Background

In data-driven science, a flexible model is a single high-capacity architecture whose parameters span a broad, often continuous space of hypotheses, so that fitting data can in principle discover hypotheses rather than only test a small discrete set chosen in advance. Artificial neural networks (ANNs) are the canonical example: smooth changes in their weights interpolate between qualitatively different input–output behaviors. Because classical model-selection criteria that penalize a count of parameters (such as the Akaike information criterion) are ill-suited to architectures whose capacity is fixed and enormous, flexible models are in practice fitted by gradient descent and selected by predictive performance on held-out data (generalization), using validation loss, early stopping, cross-validation, or regularization. This practice assumes the classical bias–variance picture in which the best-generalizing model is the right complexity; yet modern work shows that assumption can fail — for highly flexible models the test error can exhibit a 'double descent' or long flat plateaus in which many distinct fitted models generalize essentially equally well (Belkin et al., 2019).

Genkin and Engel (2020) asked what this means when flexible models are used to derive scientific hypotheses. They worked in a setting that is flexible yet intrinsically interpretable: a one-dimensional latent variable x(t)x(t) follows Langevin dynamics — a stochastic differential equation combining deterministic drift with additive random noise — dx/dt=F(x)+noisedx/dt=F(x)+\text{noise}, with driving force F(x)=−dΦ(x)/dxF(x)=-d\Phi(x)/dx derived from a potential Φ(x)\Phi(x), and spikes are emitted as an inhomogeneous Poisson process (a point process whose event rate varies over time) with rate f(x(t))f(x(t)); the shape of Φ\Phi is directly interpretable in dynamical-systems terms (its minima are attractors, multiple wells mean metastability). An exact gradient-descent maximum-likelihood fit of Φ\Phi from spike times discovers a sequence of models along optimization. The authors showed that on the resulting generalization plateau these models differ in features (e.g., the number of wells) while their validated likelihood is statistically indistinguishable, and that spurious extra wells have almost no effect on predictive performance — so validation-based selection, cross-validation, and trajectory-entropy regularization all frequently return models that do not match the ground-truth hypothesis.

Their alternative, feature consistency, rests on two ingredients. First, a scalar feature complexity: the negative trajectory entropy M=−S[Φ]\mathcal{M}=-S[\Phi], where the trajectory entropy S[Φ]S[\Phi] is itself written with a leading minus sign, as minus the Kullback–Leibler divergence (a standard information-theoretic measure of the difference between two probability distributions) between the distribution of trajectories generated by the model and that of free diffusion (a constant potential), so that M\mathcal{M} is non-negative and increases as the fitted dynamics depart from free diffusion; M\mathcal{M} is a data-independent measure of how much structure the fitted model contains, providing the common axis on which models fitted from different data samples can be aligned. Second, a reproducibility test: the data are split into two halves, a model is fitted on each, and at each matched complexity the symmetric Kullback–Leibler divergence between the two fitted models' equilibrium distributions is compared with a threshold DKLthreshD_{\mathrm{KL}}^{\text{thresh}}; the selected model is the most complex one whose features are reproducible across the independent halves. On synthetic data with known ground truth this recovers the correct potential far more reliably than generalization-based selection, and a double-blind application to V4 cortical recordings (electrode recordings from visual area V4 of the primate brain) showed concordant potentials on most channels. The method has acknowledged open methodological issues itself: the KL threshold trades underfitting (too conservative) against overfitting (too liberal) with no principled data-adaptive setting; with little data the procedure tends to underfit; and the quantitative data requirements for recovering features of a given complexity are uncharacterized.

Crucially, the entire construction leans on the intrinsic interpretability of the Langevin potential: the 'features' (wells) and their complexity axis are defined by the model's physical dynamics. For ANNs and other high-dimensional flexible models used in practice, there is no such intrinsic feature space, no canonical decomposition of a trained network into features, and no analogue of the trajectory-entropy complexity measure. The authors state in the Discussion that developing similar feature-complexity measures and model-selection procedures for ANNs 'is a significant outstanding issue, the solution of which would provide the necessary theoretical foundation for interpretable machine learning'.

Problem Statement

The problem is to generalize the feature-consistency principle of model selection — selection by reproducibility of interpretable features across independent data samples at matched feature complexity, rather than by predictive generalization — from the intrinsically interpretable one-dimensional Langevin framework in which it was demonstrated to the high-dimensional flexible models actually used in practice, above all ANNs. Concretely, this requires, as one coherent program with a single objective (reliable interpretation-first model selection):

  1. A principled, data-independent scalar measure of feature complexity for fitted high-dimensional flexible models — the analogue of the trajectory-entropy-based M=−S[Φ]\mathcal{M}=-S[\Phi], the negative trajectory entropy, a data-independent scalar quantifying how far the fitted dynamics are from free diffusion, which orders discovered features along a common axis so that models fitted from different data samples can be aligned and compared — together with a definition of the features themselves in model classes where no intrinsic feature space exists.
  2. A reproducibility-based model-selection procedure built on that measure (e.g., comparing features of matched complexity across independent data samples and selecting the most complex reproducible structure), with a tolerance/threshold hyperparameter — the analogue of the KL-threshold (the divergence tolerance above which features fitted from independent data samples are declared irreproducible), whose conservative settings underfit and liberal settings overfit in the source's one-dimensional framework.

References

  1. Mikhail Genkin, Tatiana A. Engel, 'Moving beyond generalization to accurate interpretation of flexible models', Nature Machine Intelligence 2, 674–683 (2020). DOI: 10.1038/s42256-020-00242-6, https://doi.org/10.1038/s42256-020-00242-6
  2. Mikhail Genkin, Owen Hughes, Tatiana A. Engel, 'Learning non-stationary Langevin dynamics from stochastic observations of latent trajectories', Nature Communications 12 (2021). DOI: 10.1038/s41467-021-26202-1, https://doi.org/10.1038/s41467-021-26202-1
  3. Mikhail Genkin, Krishna V. Shenoy, Chandramouli Chandrasekaran, Tatiana A. Engel, 'The dynamics and geometry of choice in the premotor cortex', Nature 645, 168–176 (2025). DOI: 10.1038/s41586-025-09199-1, https://doi.org/10.1038/s41586-025-09199-1
  4. Julia Huiming Wang, Dexter Tsin, Tatiana Engel, 'Predictive variational autoencoder for learning robust representations of time-series data', arXiv:2312.06932 (2023), https://arxiv.org/abs/2312.06932
  5. Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, David Klindt, 'Cross-Entropy Is All You Need To Invert the Data Generating Process', arXiv:2410.21869 (2024), https://arxiv.org/abs/2410.21869
  6. Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, Tom Goldstein, 'Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective', arXiv:2203.08124 (2022), https://arxiv.org/abs/2203.08124
  7. Gonçalo Paulo, Nora Belrose, 'Sparse Autoencoders Trained on the Same Data Learn Different Features', arXiv:2501.16615 (2025), https://arxiv.org/abs/2501.16615
  8. Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, Neel Nanda, 'Sparse Autoencoders Do Not Find Canonical Units of Analysis', arXiv:2502.04878 (2025), https://arxiv.org/abs/2502.04878
  9. Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal, 'Reconciling modern machine-learning practice and the classical bias-variance trade-off', Proceedings of the National Academy of Sciences 116, 15849–15854 (2019). DOI: 10.1073/pnas.1903070116, https://doi.org/10.1073/pnas.1903070116
  10. N. M. Mangan, J. N. Kutz, S. L. Brunton, J. L. Proctor, 'Model selection for dynamical systems via sparse regression and information criteria', Proceedings of the Royal Society A 473, 20170009 (2017). DOI: 10.1098/rspa.2017.0009, https://doi.org/10.1098/rspa.2017.0009
  11. Bálint Király, Balázs Hangya, 'Navigating the Statistical Minefield of Model Selection and Clustering in Neuroscience', eNeuro 9(4), ENEURO.0066-22.2022 (2022). DOI: 10.1523/ENEURO.0066-22.2022, https://doi.org/10.1523/ENEURO.0066-22.2022
  12. Xinyue Xu, Julian Wang, 'Information-theoretic and Bayesian model selection for physics-based modeling: Balancing fit, complexity, and generalization', Information Sciences 726, 122743 (2026). DOI: 10.1016/j.ins.2025.122743, https://doi.org/10.1016/j.ins.2025.122743