Extending the Kullback–Leibler divergence measure of ensemble-forecast predictability limits from the Lorenz model to realistic high-dimensional weather and climate prediction systems

Ding et al. (2019) proposed measuring the local predictability limit $T_p$ of ensemble forecasting by the Kullback–Leibler (KL) divergence between a local reference distribution of attractor states and the ensemble forecast distribution, and demonstrated on the three-variable Lorenz model that the lead time of maximum divergence agrees with nonlinear local Lyapunov exponent estimates. The demonstration relies on a perfect model, ensembles of up to $10^5$ members, random initial perturbations, and an unspecified state-space discretization. The open problem, stated by the source paper itself, is the transfer of this procedure to realistic high-dimensional numerical weather and climate ensemble prediction systems. Research challenges implied by the demonstration include: practical construction of the local reference distribution $P$ in a state space of millions of dimensions, where direct attractor counting is infeasible; reliable estimation of the ensemble forecast distribution $Q$ from operationally feasible ensemble sizes and the sampling bias this induces in $D_{\mathrm{KL}}$ and $T_p$; treatment of model error when the true-state distribution is unknown; the behavior of $T_p$ under operational initial-perturbation schemes (bred vectors, singular vectors, ensemble transform Kalman filter); and the choice among practical estimation algorithms (binning, kernel density estimation, dimensionality reduction). No published application of the KL procedure beyond low-dimensional chaotic models is known to date — the source authors' own 2026 work on ECMWF ensembles uses their older NLLE method instead — so the problem remains open.

Background

Ensemble forecasting is the operational practice of running a numerical weather prediction model many times from perturbed initial conditions, so that the spread of the resulting forecasts samples the probability distribution of future atmospheric states. A basic question in dynamical meteorology is the predictability limit: the forecast lead time beyond which the forecast distribution becomes statistically indistinguishable from the climatological distribution of the system, so that no useful information about the specific future state remains (Lorenz 1963; Lorenz 1969). For a chaotic system this limit is finite, and quantifying it — for the atmosphere, the ocean, and for specific variables and regions — is a long-standing problem.

Ding et al. (2019) introduced an information-theoretic way to measure it for ensemble forecasts. The Kullback–Leibler (KL) divergence, also called relative entropy,

$D_{\mathrm{KL}}(P\|Q)=\sum_i P(i)\,\log\frac{P(i)}{Q(i)}$

quantifies how different a probability distribution $Q$ is from a reference distribution $P$, without assuming that either distribution is Gaussian or unimodal. This matters because the common practical proxy for forecast uncertainty, the ensemble spread (the variance of the members around the ensemble mean), is interpretable mainly when the forecast distribution is approximately normal; ensemble forecast distributions can be skewed or multimodal, in which case spread can misrepresent uncertainty. Ding et al. define, for a reference state $x_i$ of a chaotic model, a local reference distribution $P$: the empirical distribution of all attractor states lying within the local attractor radius $R_L(x_i)$ (the square root of the mean squared distance of attractor states from $x_i$), which plays the role of the local truth climatology; and an ensemble forecast distribution $Q$ built from $N$ forecasts started from perturbed initial states. They define the local predictability limit $T_p$ as the forecast lead time at which $D_{\mathrm{KL}}(P\|Q)$ attains its maximum, after which the divergence falls back to stochastic fluctuation, meaning the forecasts have lost their information about the initial state. In the three-variable Lorenz (1963) model at standard parameters, this $T_p$ agrees closely with the predictability limit obtained from the nonlinear local Lyapunov exponent (NLLE) criterion — an independent error-growth diagnostic in which the limit is the time when the ensemble-mean error growth reaches its saturation level (correlation 0.92 between the two estimates over 600 initial states; see Huai et al. 2017 for the NLLE method on the Lorenz system). They further showed that $T_p$ varies strongly across the attractor (roughly 3.6 to 16 time units at perturbation amplitude $\varepsilon=10^{-3}$), decreases approximately linearly with $\log\varepsilon$, and is only mildly affected when the ensemble size $N$ is reduced from $10^5$ to 200 in the Lorenz model.

The demonstration, however, is confined to an idealized setting. (i) The Lorenz model has a three-dimensional state space, whereas realistic weather and climate models have state spaces of $10^7$–$10^9$ dimensions, in which neither $P$ nor $Q$ can be estimated by direct counting; the paper itself notes that estimating probability distributions in a higher-dimensional space from a small number of ensemble forecasts may behave very differently. (ii) Constructing $P$ from the model attractor assumes a perfect model; in an imperfect model the true-state distribution is unknown, and substituting model states or observational samples may introduce systematic errors into $D_{\mathrm{KL}}$. (iii) The experiments used random initial perturbations, whereas operational ensemble prediction systems generate initial perturbations with structured schemes — bred vectors (fast-growing perturbations recycled from successive analysis cycles; Toth & Kalnay 1997), singular vectors (fastest-growing perturbations over a finite time interval), or ensemble transform Kalman filter (ETKF) perturbations (Wei et al. 2006) — and it is unknown how $T_p$ behaves under such optimized initializations. (iv) The discretization of the continuous variables used to compute the discrete KL divergence (the binning scheme) is not specified, and no comparison of binning, kernel density estimation, or dimensionality-reduction estimators is given.

Information theory has a longer history in forecast evaluation and predictability theory: Roulston & Smith (2002) used relative entropy to evaluate probabilistic forecasts, DelSole (2004) developed information-theoretic measures of predictability, Weijs et al. (2010) showed the KL divergence can serve as a forecast skill score with a reliability–resolution decomposition, and Tang et al. (2008) compared information-based measures of forecast uncertainty in ensemble ENSO prediction. What remains open is exactly what Ding et al. (2019) asked: whether their KL-divergence procedure for measuring the predictability limit of ensemble forecasting can be transferred to realistic, high-dimensional operational prediction systems — the setting in which predictability limits are otherwise estimated by error-growth methods (e.g., Zhang et al. 2019, who estimated the predictability limit of midlatitude weather from operational ensemble forecasts).

Problem Statement

Can the KL-divergence procedure of Ding et al. (2019) for quantifying the local predictability limit $T_p$ of ensemble forecasting be extended from the three-variable Lorenz model to realistic, high-dimensional numerical weather and climate ensemble prediction systems, so as to yield a practical, working and validated KL-divergence-based estimate of predictability limits for real-world ensemble forecasts? Research challenges implied by the source's demonstrated setting include: (a) practical construction of the local reference distribution $P$ in a high-dimensional state space, where the direct attractor-neighborhood construction of the source is infeasible; (b) estimation of the ensemble forecast distribution $Q$ from operationally feasible ensemble sizes (tens to a few hundred members), and the sampling uncertainty and bias this induces in $D_{\mathrm{KL}}$ and $T_p$; (c) the effect of model error — how to define or substitute the true-state distribution $P$ when the model is imperfect, and the systematic errors this may introduce in the estimated predictability limit; (d) the sensitivity of the KL-based $T_p$ to operationally realistic initial-perturbation generation schemes (bred vectors, singular vectors, ensemble transform Kalman filter) compared with the random perturbations used in the source study; and (e) practical computational algorithms — binning, kernel density estimation, dimensionality reduction, or other estimators — and which implementations yield robust $T_p$ estimates for real-world ensemble forecasts. How these challenges are addressed is open. The transfer question concerns a weather or climate prediction system whose state space is of order $10^3$ dimensions or larger, far beyond the three-variable Lorenz model (realistic models have $10^7$-$10^9$ dimensions), with operationally realistic initial perturbations, for which a local reference distribution $P$ and an ensemble forecast distribution $Q$ would have to be defined and computed and a $T_p$ exhibited that an independent error-growth or NLLE predictability-limit estimate reproduces within a stated tolerance (the comparison that validates the Lorenz-model demonstration itself). A reasoned negative answer — showing, with evidence, that the extension is impractical or ill-posed for a stated reason — resolves the task on the same footing as a working positive estimate.

References

1. Ding, Ruiqiang; Liu, Baojia; Gu, Bin; Li, Jianping; Li, Xuan (2019). Predictability of Ensemble Forecasting Estimated Using the Kullback-Leibler Divergence in the Lorenz Model. Advances in Atmospheric Sciences 36(8): 837–846. DOI: 10.1007/s00376-019-9034-9. https://doi.org/10.1007/s00376-019-9034-9
2. Lorenz, E. N. (1963). Deterministic Nonperiodic Flow. Journal of the Atmospheric Sciences 20(2): 130–141. DOI: 10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2. https://doi.org/10.1175/1520-0469(1963)020%3C0130:DNF%3E2.0.CO;2
3. Lorenz, E. N. (1969). The predictability of a flow which possesses many scales of motion. Tellus 21(3): 289–307. DOI: 10.1111/j.2153-3490.1969.tb00444.x. https://doi.org/10.1111/j.2153-3490.1969.tb00444.x
4. Roulston, M. S.; Smith, L. A. (2002). Evaluating probabilistic forecasts using information theory. Monthly Weather Review 130(6): 1653–1660. DOI: 10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2. https://doi.org/10.1175/1520-0493(2002)130%3C1653:EPFUIT%3E2.0.CO;2
5. DelSole, T. (2004). Predictability and Information Theory. Part I: Measures of Predictability. Journal of the Atmospheric Sciences 61(20): 2425–2440. DOI: 10.1175/1520-0469(2004)061<2425:PAITPI>2.0.CO;2. https://doi.org/10.1175/1520-0469(2004)061%3C2425:PAITPI%3E2.0.CO;2
6. Weijs, Steven V.; van Nooijen, Ronald; van de Giesen, Nick (2010). Kullback–Leibler Divergence as a Forecast Skill Score with Classic Reliability–Resolution–Uncertainty Decomposition. Monthly Weather Review 138(9): 3387–3399. DOI: 10.1175/2010MWR3229.1. https://doi.org/10.1175/2010MWR3229.1
7. Tang, Y.; Kleeman, R.; Moore, A. M. (2008). Comparison of information-based measures of forecast uncertainty in ensemble ENSO prediction. Journal of Climate 21(2): 230–247. DOI: 10.1175/2007JCLI1719.1. https://doi.org/10.1175/2007JCLI1719.1
8. Toth, Z.; Kalnay, E. (1997). Ensemble Forecasting at NCEP and the Breeding Method. Monthly Weather Review 125(12): 3297–3319. DOI: 10.1175/1520-0493(1997)125<3297:EFANAT>2.0.CO;2. https://doi.org/10.1175/1520-0493(1997)125%3C3297:EFANAT%3E2.0.CO;2
9. Wei, M.; Toth, Z.; Wobus, R.; Zhu, Y.; Bishop, C. H.; Wang, X. (2006). Ensemble Transform Kalman Filter-based ensemble perturbations in an operational global prediction system at NCEP. Tellus A 58(1): 28–44. DOI: 10.1111/j.1600-0870.2006.00159.x. https://doi.org/10.1111/j.1600-0870.2006.00159.x
10. Huai, X.; Li, J.; Ding, R.; Feng, J.; Liu, D. (2017). Quantifying local predictability of the Lorenz system using the nonlinear local Lyapunov exponent. Atmospheric and Oceanic Science Letters 10(5): 372–378. DOI: 10.1080/16742834.2017.1346457. https://doi.org/10.1080/16742834.2017.1346457
11. Zhang, Fuqing; Sun, Y. Qiang; Magnusson, Linus; Buizza, Roberto; Lin, Shian-Jiann; Chen, Jan-Huey; Emanuel, Kerry (2019). What Is the Predictability Limit of Midlatitude Weather? Journal of the Atmospheric Sciences 76(4): 1077–1091. DOI: 10.1175/JAS-D-18-0269.1. https://doi.org/10.1175/JAS-D-18-0269.1
12. Fang, Xianghui; Chen, Nan (2023). Quantifying the Predictability of ENSO Complexity Using a Statistically Accurate Multiscale Stochastic Model and Information Theory. Journal of Climate 36(8): 2681–2702. DOI: 10.1175/JCLI-D-22-0151.1. https://doi.org/10.1175/JCLI-D-22-0151.1 (also arXiv:2203.02657)
13. Bednář, Hynek; Kantz, Holger (2024). Analysis of model error in forecast errors of extended atmospheric Lorenz 05 systems and the ECMWF system. Geoscientific Model Development 17(16): 6489–6511. DOI: 10.5194/gmd-17-6489-2024. https://doi.org/10.5194/gmd-17-6489-2024
14. Vannitsem, Stéphane (2023). Weather pattern dynamics over Western Europe under climate change: predictability, information entropy and production. Nonlinear Processes in Geophysics 30(1): 1–12. DOI: 10.5194/npg-30-1-2023. https://doi.org/10.5194/npg-30-1-2023
15. Mamgain, Ashu; Sarkar, Abhijit; Dube, Anumeha; Abhijith, V.; George, John P.; Prasad, V. S. (2024). Grand ensemble forecasts verification based on two high resolution (~12 km) global ensemble prediction systems. Atmospheric Research 309: 107585. DOI: 10.1016/j.atmosres.2024.107585. https://doi.org/10.1016/j.atmosres.2024.107585
16. Li, Xuan; Zhang, Xinyue; Ding, Ruiqiang; Li, Jianping; Huai, Xiaowei (2026). Investigating the predictability limit and forecast error dynamics of summer air temperatures over East Asia using the ensemble forecast dataset from ECMWF. Atmospheric Research 335: 108834. DOI: 10.1016/j.atmosres.2026.108834. https://doi.org/10.1016/j.atmosres.2026.108834
17. Lawson, J. R.; Potvin, C.; Skinner, P.; Reinhart, A. (2021). The Vice and Virtue of Increased Horizontal Resolution in Ensemble Forecasts of Tornadic Thunderstorms in Low-CAPE, High-Shear Environments. Monthly Weather Review 149(4): 921–944. DOI: 10.1175/MWR-D-20-0281.1. https://doi.org/10.1175/MWR-D-20-0281.1
18. Arranz, G.; Lozano-Durán, A. (2026). Disentangling informative and non-informative dynamics between time signals in chaotic systems. Chaos, Solitons & Fractals 203: 117600. DOI: 10.1016/j.chaos.2025.117600. https://doi.org/10.1016/j.chaos.2025.117600