Rainfall–runoff model calibration is affected by equifinality — many parameter sets fit observed streamflow comparably well — so predictive uncertainty is widely quantified with informal methods such as GLUE rather than formal statistics. Shafii, Tolson and Matott (2015) reduced some of GLUE's subjective decisions by comparing multiple behavioural-solution identification strategies and by selecting behavioural thresholds with a Distance-to-Ideal (DTI) metric that balances the reliability and sharpness of prediction bounds; whether these ideas transfer beyond GLUE remains open within that research program — the first author's dissertation hypothesizes that the improvements can be applied to other informal uncertainty-estimation methods as well, and the companion comparative study (Shafii, Tolson and Matott 2014) concludes that research to define alternative, more practical convergence criteria for MCMC applications to computationally intensive hydrologic models may be warranted. This record examines three linked open questions in that direction: (1) whether considering multiple behavioural-solution identification strategies within DDS-AU, analogous to the GLUE treatment, improves DDS-AU's predictive-uncertainty performance relative to GLUE; (2) whether the DTI metric is useful as a convergence criterion for stopping the sampling process in other sampling-based uncertainty estimation methods; and (3) how the subjectivity-reduction framework extends to other informal uncertainty methods such as multiple DDS-AU configurations, ParaSOL, and informal MCMC samplers.

Background

Rainfall–runoff models convert observed precipitation (and often temperature) into simulated streamflow at a catchment outlet, and their parameters must be calibrated against measured discharge. Because errors in input and observation data and deficiencies in model structure make many distinct parameter sets fit the observations comparably well — the equifinality problem — calibration is normally accompanied by an uncertainty analysis that reports prediction bounds around the simulated hydrograph rather than a single best simulation.

The oldest and most widely used framework is GLUE (Generalized Likelihood Uncertainty Estimation; Beven and Binley 1992). GLUE is informal in the statistical sense: it samples parameter sets, scores each with a user-chosen likelihood measure (any performance statistic, such as the Nash–Sutcliffe efficiency of the flow series), retains the sampled sets that pass a behavioural threshold as behavioural solutions, and pools their simulations into prediction limits. Every step embeds a subjective decision: which likelihood measure or measures, which threshold, and which identification strategy (for example taking the best 10 percent of runs, or ranking runs by Pareto non-domination across several criteria) define the behavioural set. Different defensible choices produce materially different prediction bounds — a long-standing criticism of informal uncertainty analysis.

Multi-criteria calibration sharpens the problem: when several performance metrics must be satisfied at once there is no single ordering of runs, so how behavioural solutions are identified becomes decisive for what is retained and reported. Two standard diagnostics judge the resulting bounds: reliability (the fraction of observations captured inside the prediction bounds; higher is better) and sharpness (the width of the bounds; narrower is sharper). The two trade off — bounds can be reliable only by being wide. Informal alternatives to GLUE share the same dependence on user choices: DDS-AU (DDS–approximation of uncertainty; Tolson and Shoemaker 2008) builds prediction-uncertainty bounds from the trajectory of the dynamically dimensioned search (DDS) calibration algorithm (Tolson and Shoemaker 2007), a greedy random search designed for computationally expensive watershed models; ParaSOL (van Griensven and Meixner 2007) couples a global multi-objective optimization with a parameter-based uncertainty estimate for water-quality catchment models; and informal MCMC denotes Markov chain Monte Carlo samplers driven by informal likelihoods without formal statistical convergence diagnostics.

Shafii, Tolson and Matott (2015) attacked GLUE's subjectivity in a multi-criteria rainfall–runoff calibration setting. They compared multiple behavioural-solution identification strategies — including a Pareto rank-based strategy in which runs are ranked by non-domination across the criteria — and replaced fixed behavioural thresholds with an optimal criteria-aggregation rule based on the Distance-to-Ideal (DTI) metric from multi-criteria decision analysis: each candidate threshold yields achieved reliability RR and sharpness SS (each normalized so the ideal value is one), and the threshold minimizing the Euclidean distance DTI=(1−R)2+(1−S)2DTI=\sqrt{(1-R)^2+(1-S)^2} to the ideal point (1,1)(1,1) is selected objectively. They found that model output uncertainty varies substantially with the identification strategy, concluding that a robust GLUE implementation should consider multiple strategies and choose the one giving the desired reliability–sharpness balance. Their companion comparative study (Shafii, Tolson and Matott 2014) had earlier shown that formal Bayesian MCMC inference with a multi-criteria likelihood outperforms GLUE at equal computational budget, while non-converged MCMC results can approximate fully converged ones — motivating practical convergence criteria for sampling-based uncertainty estimation.

After 2015, the DTI-based threshold was adopted by later studies, but only inside GLUE: Liu et al. (2021) used the optimal criteria-aggregation threshold of Shafii et al. (2015) for behavioural identification when estimating hydrologic-model parameters from ensemble meteorological forcing, and Schreiner-McGraw et al. (2022) used the optimum-percentage rule via the Pythagorean DTI condition when integrating evapotranspiration data into hydrologic models. The transfer of the multi-strategy and DTI framework beyond GLUE — to DDS-AU, ParaSOL, and informal MCMC, and to sampling termination rules — is the unresolved open direction of the source research program, and is examined here.

Problem Statement

Overarching objective: reduce the subjective decisions inherent in informal (statistically non-rigorous) methods for describing rainfall–runoff model output uncertainty, by extending beyond GLUE the framework validated by Shafii, Tolson and Matott (2015) — multiple behavioural-solution identification strategies combined with Distance-to-Ideal (DTI)-based objective selection. The source research program identifies this transfer beyond GLUE as an open direction; three linked open questions arise:

(a) DDS-AU with multiple behavioural-solution identification strategies. DDS-AU (Tolson and Shoemaker 2008) approximates prediction uncertainty from the trajectory of the DDS calibration algorithm, and its results depend on user-made configuration decisions. Considering multiple behavioural-solution identification strategies within DDS-AU, analogous to the approach taken for GLUE in the source paper — does this improve DDS-AU's performance relative to GLUE? Performance is judged, following the source study's own conventions, by the quality of the predictive uncertainty bounds — their reliability and sharpness (for example the containing ratio and width of prediction bounds over calibration and validation periods) — under comparable computational budgets.

(b) DTI as a convergence criterion. Evaluate the utility of the DTI metric in sampling-based uncertainty estimation methods other than GLUE as a convergence criterion to stop the sampling process: can the distance in criteria space between currently achieved (reliability, sharpness) and the ideal point, at which both equal one, be used during sampling to decide when further model evaluations no longer materially improve the predictive uncertainty bounds?

(c) Extension to other informal methods. Extend the ideas of the source paper to address and reduce the subjective decisions inherent in other informal methods for describing hydrological model output uncertainty — for example multiple DDS-AU configurations, ParaSOL, or informal MCMC samplers.

The three questions are linked facets of the single objective of de-subjectifying informal uncertainty estimation: (a) is a concrete comparative question, (b) concerns an objective stopping rule that such comparisons and practical applications would need, and (c) is the generalization direction.

References

  1. Shafii, M., Tolson, B.A., Matott, L.S. (2015). Addressing subjective decision-making inherent in GLUE-based multi-criteria rainfall–runoff model calibration. Journal of Hydrology 523:693–705. DOI: 10.1016/j.jhydrol.2015.01.051. https://doi.org/10.1016/j.jhydrol.2015.01.051
  2. Beven, K., Binley, A. (1992). The future of distributed models: model calibration and uncertainty prediction. Hydrological Processes 6(3):279–298. DOI: 10.1002/hyp.3360060305. https://doi.org/10.1002/hyp.3360060305
  3. Tolson, B.A., Shoemaker, C.A. (2007). Dynamically dimensioned search algorithm for computationally efficient watershed model calibration. Water Resources Research 43(1). DOI: 10.1029/2005WR004723. https://doi.org/10.1029/2005WR004723
  4. Tolson, B.A., Shoemaker, C.A. (2008). Efficient prediction uncertainty approximation in the calibration of environmental simulation models. Water Resources Research 44(4). DOI: 10.1029/2007WR005869. https://doi.org/10.1029/2007WR005869
  5. van Griensven, A., Meixner, T. (2007). A global and efficient multi-objective auto-calibration and uncertainty estimation method for water quality catchment models. Journal of Hydroinformatics 9(4):277–291. DOI: 10.2166/hydro.2007.104. https://doi.org/10.2166/hydro.2007.104
  6. Shafii, M., Tolson, B.A., Matott, L.S. (2014). Uncertainty-based multi-criteria calibration of rainfall-runoff models: a comparative study. Stochastic Environmental Research and Risk Assessment 28(6):1493–1510. DOI: 10.1007/s00477-014-0855-x. https://doi.org/10.1007/s00477-014-0855-x
  7. Zhou, R., Li, Y., Lu, D., Liu, H., Zhou, H. (2016). An optimization based sampling approach for multiple metrics uncertainty analysis using generalized likelihood uncertainty estimation. Journal of Hydrology 540:274–286. DOI: 10.1016/j.jhydrol.2016.06.030. https://doi.org/10.1016/j.jhydrol.2016.06.030
  8. Liu, H., Tolson, B.A., Newman, A.J., Wood, A.W. (2021). Leveraging ensemble meteorological forcing data to improve parameter estimation of hydrologic models. Hydrological Processes 35(11):e14410. DOI: 10.1002/hyp.14410. https://doi.org/10.1002/hyp.14410
  9. Schreiner-McGraw, A.P., Ajami, H., Anderson, R.G., Wang, D. (2022). Integrating partitioned evapotranspiration data into hydrologic models: vegetation parameterization and uncertainty quantification of simulated plant water use. Hydrological Processes 36(6):e14580. DOI: 10.1002/hyp.14580. https://doi.org/10.1002/hyp.14580
  10. Her, Y., Seong, C. (2018). Responses of hydrological model equifinality, uncertainty, and performance to multi-objective parameter calibration. Journal of Hydroinformatics 20(4):864–885. DOI: 10.2166/hydro.2018.108. https://doi.org/10.2166/hydro.2018.108
  11. Baig, F., Demirel, M.C., Taseer, M.Y.R., Kassem, A., Sherif, M. (2025). Effect of different likelihood measures on parameter conditioning and model uncertainty. Stochastic Environmental Research and Risk Assessment 39(3):1091–1108. DOI: 10.1007/s00477-025-02909-1. https://doi.org/10.1007/s00477-025-02909-1
  12. Yu, H.-L., Chang, K.-H., Chang, T.-J. (2025). An Improved Generalized Likelihood Uncertainty Estimation (GLUE) Approach for Uncertainty Analysis of Capacity Factors in Operational Wind Turbines. International Journal of Energy Research. DOI: 10.1155/er/9235993. https://doi.org/10.1155/er/9235993
  13. Gupta, A., Hantush, M.M., Govindaraju, R.S., Beven, K. (2024). Evaluation of hydrological models at gauged and ungauged basins using machine learning-based limits-of-acceptability and hydrological signatures. Journal of Hydrology 641:131774. DOI: 10.1016/j.jhydrol.2024.131774. https://doi.org/10.1016/j.jhydrol.2024.131774
  14. Shi, P., Yang, T., Yong, B., Li, Z., Xu, C.-Y., Shao, Q., Wang, X., Zhou, X., Qin, Y. (2019). A New Uncertainty Measure for Assessing the Uncertainty Existing in Hydrological Simulation. Water 11(4):812. DOI: 10.3390/w11040812. https://doi.org/10.3390/w11040812