Mechanism of the non-monotonic dependence of reservoir-computing predictive skill on training-data length in North Atlantic SST prediction

Nadiga (2021, Journal of Advances in Modeling Earth Systems) showed that a reservoir computer (echo-state network) matches or beats Linear Inverse Modeling (fitting a linear stochastic model to the same data) for predicting North Atlantic sea-surface-temperature anomalies in the pre-industrial control simulation of the Community Earth System Model version 2 (CESM2), and observed a puzzling signature: while the linear model's skill degrades monotonically as training data are reduced, reservoir-computing skill varies non-monotonically with training-data length — worst at intermediate lengths (about 30 years with 30 retained empirical-orthogonal-function, i.e. principal-component, coefficients; about 10 years with 4) and better on both sides. The author speculated this reflects a transition from a strongly nonlinear to a more linear reservoir response as data grow, and deferred the investigation, but never published the promised follow-up. This problem asks for the identification and validation of the mechanism behind this non-monotonic dependence.

Background

Reduced-order statistical models of climate variability are used to study predictability when large ensembles of a comprehensive climate model are too expensive to run. A classic linear approach is Linear Inverse Modeling (LIM), which fits a linear stochastic dynamical system — a drift matrix plus additive white noise — to the leading Empirical Orthogonal Function (EOF; equivalently principal-component) time coefficients of a variable such as sea-surface temperature (SST), and predicts by evolving that linear model. Reservoir computing (RC) is a nonlinear alternative: an echo-state network (ESN) is a recurrent neural network whose internal (reservoir) weights are drawn at random and kept fixed, so that training reduces to a single ridge-regularized linear regression for the readout layer mapping reservoir states to predicted states. The echo-state property — the requirement that reservoir states asymptotically forget past inputs — is heuristically controlled by the spectral radius of the random weight matrix, and the degree of nonlinearity of the reservoir response is governed by the input scaling and the saturating tanh activation; a leakage parameter can additionally low-pass filter the reservoir update. Nadiga (2021) compared RC against LIM for predicting North Atlantic SST anomalies in the 800-year pre-industrial control (piControl) simulation of the Community Earth System Model version 2 (CESM2), in a truncated EOF basis (4 coefficients explaining about 50% of the variance, 30 coefficients about 95%), scoring year-one-averaged predictions by normalized root-mean-square error (NRMSE) and anomaly correlation coefficient (ACC) and averaging over 32 random reservoir realizations ($N_r = 200$ neurons, spectral radius 1, ridge coefficient 0.1, no leakage). RC matched or beat LIM essentially everywhere, and — surprisingly — especially in the limited-data regime, even though its readout must estimate far more parameters ($\sim N_y(N_y+N_r)$) than a LIM ($\sim N_y(N_y+1)/2$). The puzzle targeted here: whereas LIM skill degrades monotonically as training data are reduced, as expected from standard statistical reasoning, RC skill varies non-monotonically with training-data length — worst at intermediate lengths (around 30 years with 30 retained EOF coefficients, around 10 years with 4) and better on both sides. The LIM baseline in that comparison follows the source's instability-flagging convention: LIM predictions flagged as unrealistic (if the predicted amplitude of any EOF coefficient exceeded ten times the climatological standard deviation over the prediction period) are eliminated from the LIM skill measures, training-length cases in which greater than 50% of the predictions were flagged are not reported at all (the leftmost LIM points of the training-length sweep figure are omitted for this reason), and the author notes that the presented LIM skill is thereby (artificially) inflated. The author speculated this reflects a transition from a strongly nonlinear reservoir response when data are limited to a more linear response when data are plentiful, and explicitly deferred the investigation. Adjacent results sharpen the puzzle rather than settling it: systematic RC design explorations on chaotic toy systems report only monotone diminishing returns with more training data (Platt et al., 2022), and in statistical-learning theory the test error of ridge-type estimators is known to peak near the interpolation threshold where the number of samples approaches the number of fitted parameters (the double-descent phenomenon); a recent random-matrix analysis of idealized linear teacher–student echo-state networks (Moahker et al., 2026) concludes that leakage generally suppresses such error peaks, except in degenerate regimes with no leakage and vanishing regularization — a notable caveat because the CESM2 SST configuration used no leakage. No existing result explains the observed climate-data behavior.

Problem Statement

Identify and validate the mechanism responsible for the non-monotonic dependence of reservoir-computing predictive skill on training-data length observed by Nadiga (2021) in the CESM2 piControl North Atlantic SST setting: with the published echo-state-network configuration ($N_r = 200$ reservoir neurons, spectral radius $\rho = 1$, input scale $\alpha_{\mathrm{in}}^2 = 10^{-2}$, ridge regularization $\alpha^2 = 0.1$, symmetric tanh activation, no leakage, averaging over 32 random reservoir realizations), the NRMSE and ACC of year-one-averaged predictions are worst at intermediate training lengths — around 30 years with 30 retained EOF coefficients and around 10 years with 4 — while improving both with shorter and longer training records, in contrast to the approximately monotonic behavior of the Linear Inverse Model on the same data (computed under the source's instability-flagging convention, which the author notes artificially inflates LIM skill; see Background). The research target has three parts: (i) reproduction of the qualitative phenomenon on the public CESM2 piControl North Atlantic SST data (12-month-averaged SST anomalies, leading EOF/PC coefficients with $N_y = 4$ and $N_y = 30$) under the fully documented source configuration and case protocols — sparsity 0, washout period $T_{wo} = \min(100,\,0.1\,T_{tr})$ months, monthly time step, year-1-averaged NRMSE and ACC; prediction starts every $T_{skip} = 18$ months in the plentiful-data case; six-year stepping with 60:20:20 training:validation:test splits and new LIM and RC models per segment in the limited-data case — exhibiting a non-monotonic RC skill-versus-training-length dependence with minima near 30 years for $N_y = 30$ and near 10 years for $N_y = 4$, together with approximately monotone LIM degradation under the documented filtering convention; where the source leaves the design of the training-length sweep unspecified (the swept training lengths, the placement of the short segments, and the number and spacing of prediction starts at each length), the design choices are stated and justified, and exact numerical replication of the source figure is not required; (ii) identification of a precise mechanism — for example an interpolation-threshold (double-descent-type) effect of the ridge-trained readout, a data-length-dependent effective nonlinearity of the reservoir response as the author speculated, a regime-sampling property of short versus long climate records, or another cause; (iii) demonstration of that mechanism by derivation or controlled numerical experiment, discriminating it from the other candidate causes and accounting for how the LIM instability-flag filtering affects the RC-versus-LIM contrast at short training lengths.

References

1. Nadiga, B. T. (2021). Reservoir Computing as a Tool for Climate Predictability Studies. Journal of Advances in Modeling Earth Systems, 13(4), e2020MS002290. DOI: 10.1029/2020MS002290. https://doi.org/10.1029/2020MS002290 (also arXiv:2103.06206, https://arxiv.org/abs/2103.06206)
2. Platt, J. A., Penny, S. G., Smith, T. A., Chen, T.-C., & Abarbanel, H. D. I. (2022). A systematic exploration of reservoir computing for forecasting complex spatiotemporal dynamics. Neural Networks, 153. DOI: 10.1016/j.neunet.2022.06.025. https://doi.org/10.1016/j.neunet.2022.06.025 (arXiv:2201.08910, https://arxiv.org/abs/2201.08910)
3. Moahker, Y. (the arXiv version lists this first author as Yessin Moakher), Tiomoko, M., Louart, C., & Liao, Z. (2026). A Random Matrix Perspective of Echo State Networks: From Precise Bias–Variance Characterization to Optimal Regularization. ICASSP 2026 – 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 20716–20720. DOI: 10.1109/ICASSP55912.2026.11464627. https://doi.org/10.1109/ICASSP55912.2026.11464627 (arXiv:2509.22011, https://arxiv.org/abs/2509.22011)
4. Guardamagna, F., Wieners, C., Fang, X., & Dijkstra, H. A. (2024). Detection of limit cycle signatures of El Niño in models and observations using reservoir computing. Journal of Physics: Complexity, 5. DOI: 10.1088/2632-072X/ad2699. https://doi.org/10.1088/2632-072X/ad2699
5. Koyama, S., Inoue, D., Yoshida, H., Aihara, K., & Tanaka, G. (2026). Assessing the predictability of meteorological variables via spatial correlations using echo state networks. Chaos, 36. DOI: 10.1063/5.0324305. https://doi.org/10.1063/5.0324305
6. Smith, T. A., Penny, S. G., Platt, J. A., & Chen, T.-C. (2023). Temporal Subsampling Diminishes Small Spatial Scales in Recurrent Neural Network Emulators of Geophysical Turbulence. Journal of Advances in Modeling Earth Systems, 15. DOI: 10.1029/2023MS003792. https://doi.org/10.1029/2023MS003792
7. Yao, K., Forgoston, E., & Yecko, P. (2022). Learning ocean circulation models with reservoir computing. Physics of Fluids, 34. DOI: 10.1063/5.0119061. https://doi.org/10.1063/5.0119061
8. Agarwal, N., Kondrashov, D., Dueben, P., Ryzhov, E., & Berloff, P. (2021). A Comparison of Data-Driven Approaches to Build Low-Dimensional Ocean Models. Journal of Advances in Modeling Earth Systems, 13. DOI: 10.1029/2021MS002537. https://doi.org/10.1029/2021MS002537
9. Chen, X., Nadiga, B. T., & Timofeyev, I. (2022). Predicting shallow water dynamics using echo-state networks with transfer learning. GEM – International Journal on Geomathematics, 13. DOI: 10.1007/s13137-022-00210-9. https://doi.org/10.1007/s13137-022-00210-9
10. Luo, X., Nadiga, B. T., Park, J. H., Ren, Y., Xu, W., & Yoo, S. (2022). A Bayesian Deep Learning Approach to Near-Term Climate Prediction. Journal of Advances in Modeling Earth Systems, 14. DOI: 10.1029/2022MS003058. https://doi.org/10.1029/2022MS003058
11. Erofeev, A., Nadiga, B. T., & Timofeyev, I. (2025). Using Echo-State Networks to Reproduce Rare Events in Chaotic Systems. arXiv:2505.16208. https://arxiv.org/abs/2505.16208