Open questions from Perrone and Cooper's Generalized Ensemble Method: finite-sample weight estimation, principled member selection, and ensemble saturation

The Generalized Ensemble Method (GEM) of Perrone and Cooper (1995) combines $p$ trained neural networks by a weighted sum whose optimal weights, expressed through the correlation matrix $C$ of the networks' misfits, provably improve on the simple average (the Basic Ensemble Method) and on the best single network — but only under the idealized assumption that the true, full-rank $C$ is known. In practice $C$ is estimated from a finite cross-validation set, the chapter's own OCR (optical character recognition) experiments used a diagonal approximation, near-duplicate networks were pruned heuristically to keep $C$ invertible, and ensemble gains saturated after roughly six to eight members. The question is what survives when the idealization is dropped. Two strands — nonlinear combiners incorporating confidence measures, and empirical testing of the full GEM with estimated $C$ — have been substantially absorbed by the stacked-generalization and deep-ensemble literature and are out of scope here. The open core of the problem is the finite-sample theory of optimally weighted ensembles of learned estimators: (i) conditions, in terms of member count, validation-set size, and the spectrum of the misfit correlations, under which weights estimated from a sample covariance matrix still provably improve on simple averaging and on the best member, including ill-conditioned regimes; (ii) principled member-subset selection with guarantees balancing statistical stability against information retention; and (iii) a characterization of when ensemble performance saturates, tied to an operational notion of the number of functionally distinct minima reachable by training. Review-level work as late as 2023 still describes the diversity-theory heart of this question as an open research issue of over 30 years standing.

Background

An ensemble predictor is a combination — classically a weighted average — of several predictors trained for the same task, whose risk can be lower than that of any single member. Perrone and Cooper's 1995 chapter studied ensembles of neural networks trained from different random initializations on the same training data, combined under squared-error loss. Writing $f_i(x)$ for the $i$-th network's output on input $x$, $d(x)$ for the target, and $p$ for the number of networks, the misfit of network $i$ at $x$ is $d(x)-f_i(x)$; $C$ is the $p\times p$ matrix of correlations between members' misfits, in practice estimated on a held-out cross-validation set (a split of the data used to measure out-of-sample behavior rather than to train). The Basic Ensemble Method (BEM) is the simple average $\frac{1}{p}\sum_i f_i(x)$; the Generalized Ensemble Method (GEM) chooses weights $\alpha$ with $\sum_i\alpha_i=1$ minimizing the expected squared error of $\sum_i\alpha_i f_i(x)$, which yields $\alpha_i=(\sum_j C^{-1}_{ij})/(\sum_k\sum_j C^{-1}_{kj})$ and, with the true full-rank $C$, is provably never worse than the BEM and the best single member. The chapter's experiments — networks with 10 hidden units on handprinted characters from the NIST (U.S. National Institute of Standards and Technology) upper-case dataset, an OCR (optical character recognition) task — found ensemble error flattening after roughly six to eight members, and required heuristic pruning of near-duplicate networks (identified by large dot products of rows of the estimated $\hat{C}$) to keep $C$ invertible.

The open problem is the gap between this idealized guarantee and practice: only $\hat{C}$ is available; it is noisy when the cross-validation set is small relative to $p$; near-degeneracies must be handled; and the experiments used a diagonal approximation of $C$ rather than the full inverse. Three later threads define the current landscape. First, stacked generalization (Wolpert 1992) — training a higher-level combiner on the cross-validated predictions of the members, the nonlinear-combiner generalization the source anticipated — and Breiman's stacked regressions (1996), which found unconstrained stacking weights unstable on real data and controlled them with non-negativity constraints on the weights (ridge-type constraints were tried but gave inconsistent gains); Ting and Witten (1999) reported that 'best results are obtained when the higher-level model combines the confidence (and not just the predictions) of the lower-level ones'. Second, statistical forecast combination, where the 'forecast combination puzzle' is the empirical finding that simple averages often outperform weights estimated to be optimal; Smith and Wallis (2009) traced it to finite-sample weight-estimation error, and Ledoit and Wolf (2004) supplied shrinkage estimators (estimators that shrink the sample covariance matrix toward a structured target) that keep covariance matrices well-conditioned exactly when the number of variables is comparable to the sample size. Third, the diversity thread: Krogh and Vedelsby (1994) decomposed expected ensemble error into average bias, average variance, and a negative ambiguity term (the member disagreement that ensembling harvests); Kuncheva and Whitaker (2003) showed that existing diversity measures relate to ensemble accuracy only weakly and non-monotonically; and the loss-landscape literature (the geometry of the training loss as a function of the network's weights) — Garipov et al. (2018) and Draxler et al. (2018) on mode connectivity (low-loss paths connecting independently found solutions), Fort, Hu, and Lakshminarayanan (2019) on the distinct basins (separate low-loss regions of the landscape) occupied by independently trained deep networks — supplies the modern language for the source's 'distinct functional minima', while Lakshminarayanan, Pritzel, and Blundell (2017) made randomly initialized deep ensembles a standard tool, Allen-Zhu and Li (2020) gave a first theoretical account of why ensembling helps deep networks, and Abe et al. (2023) showed that diversity-promoting interventions help small-network ensembles but harm large-network ones. Wood et al. (2023) unified diversity theory and still describe diversity as having been 'referred to as the holy grail of ensemble learning, an open research issue for over 30 years'.

Problem Statement

The problem is to close the gap between the idealized Perrone-Cooper Generalized Ensemble Method and ensemble practice, for predictors trained by a randomized learning procedure (originally, neural networks trained from different initializations on the same data) and combined under squared-error loss. The source raises three distinct but related open concerns, which share the GEM framework and the misfit correlation matrix $C$ as common background.

1. Finite-sample weight estimation. Characterize — in terms of the number of members $p$, the validation-set size $n$ on which the misfit correlation matrix is estimated, and the spectrum and conditioning of the true matrix $C$ — when the ensemble weights computed from the estimate $\hat{C}$ (or a stated regularization of it) still improve on the simple average and on the best single member, in expectation or with controlled probability over the validation draw, and when they do not. The characterization must cover, not silently exclude, the near-degenerate and ill-conditioned regime that the source handled by heuristically pruning near-duplicate members.

2. Principled member selection. Replace heuristic near-duplicate removal and greedy pruning with a member-subset selection rule that carries a guarantee — relating the ensemble it returns to the ideal GEM with the true $C$, or to the simple-average baseline — and that makes the stability-versus-information trade-off explicit rather than emergent.

3. Saturation and the number of distinct solutions. Characterize the conditions (task, model class, capacity, training procedure) under which ensemble performance saturates as members are added, and make the source's hypothesis precise: give an operational definition of the number of functionally distinct minima (or solutions) reachable by training — the chapter's own informal criterion was that a set of networks counts as distinct when the correlation matrix of their misfits has a 'robust inverse', a notion it left unoperationalized — and connect it to where and why the ensemble risk curve flattens.

The three concerns may be addressed individually or in combination; an answer need not be a single theorem. Nonlinear combiners incorporating confidence measures and empirical testing of the full GEM with estimated $C$ on real problems are out of scope for this task: the stacking and deep-ensemble literature has substantially settled those strands (Wolpert 1992; Breiman 1996; Ting and Witten 1999; Caruana et al. 2004; Lakshminarayanan et al. 2017).

References

1. Michael P. Perrone and Leon N. Cooper, 'When networks disagree: Ensemble methods for hybrid neural networks', in How We Learn; How We Remember: Toward an Understanding of Brain and Neural Systems, World Scientific Series in 20th Century Physics, pp. 342-358, World Scientific (1995). DOI: 10.1142/9789812795885_0025, https://doi.org/10.1142/9789812795885_0025
2. David H. Wolpert, 'Stacked generalization', Neural Networks 5(2), 241-259 (1992). DOI: 10.1016/S0893-6080(05)80023-1, https://doi.org/10.1016/S0893-6080(05)80023-1
3. Leo Breiman, 'Stacked regressions', Machine Learning 24(1), 49-64 (1996). DOI: 10.1007/BF00117832, https://doi.org/10.1007/BF00117832
4. K. M. Ting and I. H. Witten, 'Issues in Stacked Generalization', Journal of Artificial Intelligence Research 10, 271-289 (1999). DOI: 10.1613/jair.594, https://doi.org/10.1613/jair.594
5. Sherif Hashem, 'Optimal Linear Combinations of Neural Networks', Neural Networks 10(4), 599-614 (1997). DOI: 10.1016/S0893-6080(96)00098-6, https://doi.org/10.1016/S0893-6080(96)00098-6
6. Anders Krogh and Jesper Vedelsby, 'Neural Network Ensembles, Cross Validation, and Active Learning', Advances in Neural Information Processing Systems 7 (NIPS 1994). https://proceedings.neurips.cc/paper_files/paper/1994/hash/b8c37e33defde51cf91e1e03e51657da-Abstract.html
7. Kagan Tumer and Joydeep Ghosh, 'Error Correlation and Error Reduction in Ensemble Classifiers', Connection Science 8(3-4), 385-404 (1996). DOI: 10.1080/095400996116839, https://doi.org/10.1080/095400996116839
8. Thomas G. Dietterich, 'Ensemble Methods in Machine Learning', in Multiple Classifier Systems (MCS 2000), Lecture Notes in Computer Science, pp. 1-15 (2000). DOI: 10.1007/3-540-45014-9_1, https://doi.org/10.1007/3-540-45014-9_1
9. Ludmila I. Kuncheva and Christopher J. Whitaker, 'Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy', Machine Learning 51(2), 181-207 (2003). DOI: 10.1023/A:1022859003006, https://doi.org/10.1023/A:1022859003006
10. Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes, 'Ensemble selection from libraries of models', Proceedings of the 21st International Conference on Machine Learning (ICML 2004), Article 18 (2004). DOI: 10.1145/1015330.1015432, https://doi.org/10.1145/1015330.1015432
11. Olivier Ledoit and Michael Wolf, 'A well-conditioned estimator for large-dimensional covariance matrices', Journal of Multivariate Analysis 88(2), 365-411 (2004). DOI: 10.1016/S0047-259X(03)00096-4, https://doi.org/10.1016/S0047-259X(03)00096-4
12. Jeremy Smith and Kenneth F. Wallis, 'A Simple Explanation of the Forecast Combination Puzzle', Oxford Bulletin of Economics and Statistics 71(3), 331-355 (2009). DOI: 10.1111/j.1468-0084.2008.00541.x, https://doi.org/10.1111/j.1468-0084.2008.00541.x
13. Mark J. van der Laan, Eric C. Polley, and Alan E. Hubbard, 'Super Learner', Statistical Applications in Genetics and Molecular Biology 6(1) (2007). DOI: 10.2202/1544-6115.1309, https://doi.org/10.2202/1544-6115.1309
14. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell, 'Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles', Advances in Neural Information Processing Systems 30 (NIPS 2017); arXiv:1612.01474 (submitted December 2016). https://arxiv.org/abs/1612.01474
15. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson, 'Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs', Advances in Neural Information Processing Systems 31 (NeurIPS 2018); arXiv:1802.10026. https://arxiv.org/abs/1802.10026
16. Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht, 'Essentially No Barriers in Neural Network Energy Landscape', Proceedings of the 35th International Conference on Machine Learning (ICML 2018), PMLR 80, 1308-1317; arXiv:1803.00885. https://arxiv.org/abs/1803.00885
17. Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan, 'Deep Ensembles: A Loss Landscape Perspective', arXiv:1912.02757 (2019). https://arxiv.org/abs/1912.02757
18. Zeyuan Allen-Zhu and Yuanzhi Li, 'Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning', arXiv:2012.09816 (2020). https://arxiv.org/abs/2012.09816
19. Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luján, and Gavin Brown, 'A Unified Theory of Diversity in Ensemble Learning', Journal of Machine Learning Research 24(359), 1-49 (2023); arXiv:2301.03962. https://www.jmlr.org/papers/v24/23-0041.html, https://arxiv.org/abs/2301.03962
20. Taiga Abe, E. Kelly Buchanan, Geoff Pleiss, and John P. Cunningham, 'Pathologies of Predictive Diversity in Deep Ensembles', arXiv:2302.00704 (2023); published in Transactions on Machine Learning Research (2024). https://arxiv.org/abs/2302.00704, https://openreview.net/forum?id=TQfQUksaC8