Executives forecasting strategic outcomes — a film's revenue, a project's cost, an investment's return — can take an 'outside view' by consulting the distribution of outcomes of similar past cases rather than reasoning from the focal case alone. Lovallo, Clarke, and Camerer (2012) formalized one such method, similarity-based forecasting (SBF), which combines an objectively sampled reference class with human similarity judgments, aggregated by a robust mean and clustered to choose which analogies enter a regression; on 19 unreleased films, SBF roughly halved forecast error relative to hedonic and interaction regressions. The authors themselves left the method's foundations untested: self-generated reference classes are small and skewed toward successes, no objective sampling rule is known to be optimal across domains, and the sensitivity of the forecasts to the similarity-elicitation population, the aggregation rule, and the clustering algorithm was never measured. Later work has proposed data-driven reference-class selection for corporate sales-growth forecasting (Theising, Wied, and Ziggel, 2023), biclustering-based class selection for IoT value propositions (Garbuio and Gheno, 2023), and weighted reference-class variants for infrastructure costs (Zani and Adey, 2025), while a 2025 review of reference-class forecasting still lists class construction and validity among the field's open questions (Cantarelli et al., 2025). The open problem is to establish, with cross-domain empirical evidence, how reference-class construction choices determine the accuracy and calibration of structured outside-view forecasts, and which objective construction rules are optimal.

Background

Strategic forecasts — the revenue of an unreleased film, the cost of a megaproject, the return on an investment — are notoriously optimistic when the forecaster reasons only about the focal case. Kahneman and Lovallo (1993) called this singular reasoning the 'inside view' and proposed the 'outside view' as a corrective: locate the current case within a reference class of similar past cases and consult the distribution of their outcomes. Reference-class forecasting (RCF) operationalizes the outside view by attaching to a forecast the empirical outcome distribution of comparable past projects; it has been adopted widely in infrastructure and megaproject cost estimation, where an 'uplift' drawn from the class distribution is added to a bottom-up estimate. Case-based decision theory (Gilboa and Schmeidler, 1995) supplies the complementary formal ingredient: a model of decision making under uncertainty in which past cases influence choices in proportion to their similarity to the current problem. Green and Armstrong (2007) showed that a structured protocol in which several experts each retrieve and weight analogous past cases ('structured analogies') can improve forecasts relative to unaided judgment.

Lovallo, Clarke, and Camerer (2012) combined these threads into similarity-based forecasting (SBF), a seven-step procedure: (1) assemble an objectively sampled memory database of past cases (1,751 U.S. theatrical films, 1993-2004); (2) select target cases and the reference class by an objective two-stage rule rather than by memory; (3) elicit similarity judgments between each target and past cases from a panel of respondents (358 moviegoers rating similarity on a seven-point scale); (4) aggregate individual judgments with a robust mean (an averaging rule that downweights outlying raters) and submit them to hierarchical clustering (a method that groups cases by similarity, yielding a nested 'top cluster' of the most similar analogies) so that the number of analogies used is fixed endogenously by the similarity structure; (5-6) build similarity-weighted predictors and combine them with observable controls such as production budget and opening-week theater count, the last serving as a proxy for unobservable marketing effort; (7) regress log revenue on these inputs. On 19 unreleased films, SBF achieved a mean absolute relative error (MARE — the average of |forecast minus actual| divided by actual) of about 30 percent, versus about 62 percent for a hedonic regression (a regression of revenue on observable attributes) and 66 percent for a regression rich in interaction terms; a single-analogy forecast erred by about 135 percent. A companion study of private-equity investment decisions showed that merely prompting investors to form reference classes moved initially optimistic expected-return estimates downward by about six percentage points on average.

The authors, however, flagged the construction of the reference class itself as the unresolved core: professionals left to generate their own classes produced small, success-skewed samples (in their data, only 42 percent of respondents listed the requested five examples per category, and most examples were described as successes); nothing established which objective sampling rule is optimal, in any domain; and the sensitivity of the forecasts to the elicitation population, the aggregation rule, and the clustering algorithm was never measured. The surrounding RCF literature is likewise unsettled: an empirical evaluation in project management reports RCF outperforming traditional forecasting methods (Batselier and Vanhoucke, 2016), a longitudinal megaproject case study reports RCF failing to improve cost estimates (Themsen, 2019), and a 2025 review concludes that questions of validity, applicability, and limitations remain open while adoption keeps expanding (Cantarelli et al., 2025).

Problem Statement

How should the reference class underlying a structured outside-view forecast be constructed, and how much does the answer matter? For similarity-based forecasting (SBF) and cognate structured reference-class methods, the problem is to determine (i) how forecast accuracy and distributional calibration depend on each load-bearing construction choice — the objective rule that samples or selects the reference class (of which Lovallo, Clarke, and Camerer's two-stage selection is one instance), the population from which human similarity judgments are elicited, the rule that aggregates individual similarity judgments into a case-similarity measure, and the clustering algorithm and case-count rule that fix which analogies enter the forecast — and (ii) which objective construction rules yield forecasts that are accurate and well calibrated relative to competing statistical and machine-learning models, in forecasting domains beyond the original 19-film box-office test. These two prongs serve one overarching objective: replacing the paper's demonstration that one particular construction worked once with knowledge of how construction choices drive forecast quality. The problem therefore also includes quantifying the sensitivity of forecasts to these choices — in particular whether the construction avoids the small-sample success skew of self-generated reference classes — and determining which objective construction rules dominate, or, where none does, the conditions (domain, reference-database size, outcome distribution) under which different rules dominate.

References

  1. Dan Lovallo, Carmina Clarke, Colin Camerer (2012). 'Robust analogizing and the outside view: two empirical tests of case-based decision making.' Strategic Management Journal 33(5):496-512. DOI: 10.1002/smj.962. https://doi.org/10.1002/smj.962
  2. Daniel Kahneman, Dan Lovallo (1993). 'Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking.' Management Science 39(1):17-31. DOI: 10.1287/mnsc.39.1.17. https://doi.org/10.1287/mnsc.39.1.17
  3. I. Gilboa, D. Schmeidler (1995). 'Case-Based Decision Theory.' The Quarterly Journal of Economics 110(3):605-639. DOI: 10.2307/2946694. https://doi.org/10.2307/2946694
  4. Kesten C. Green, J. Scott Armstrong (2007). 'Structured analogies for forecasting.' International Journal of Forecasting 23(3):365-376. DOI: 10.1016/j.ijforecast.2007.05.005. https://doi.org/10.1016/j.ijforecast.2007.05.005
  5. Etienne Theising, Dominik Wied, Daniel Ziggel (2023; first published online 2022). 'Reference class selection in similarity-based forecasting of corporate sales growth.' Journal of Forecasting 42(5):1069-1085. DOI: 10.1002/for.2927. https://doi.org/10.1002/for.2927
  6. Massimo Garbuio, Gloria Gheno (2023; early access 2021). 'An Algorithm for Designing Value Propositions in the IoT Space: Addressing the Challenges of Selecting the Initial Class in Reference Class Forecasting.' IEEE Transactions on Engineering Management 70(9):3171-3182. DOI: 10.1109/TEM.2021.3077625. https://doi.org/10.1109/TEM.2021.3077625
  7. Jordy Batselier, Mario Vanhoucke (2016). 'Practical Application and Empirical Evaluation of Reference Class Forecasting for Project Management.' Project Management Journal 47(5):36-51. DOI: 10.1177/875697281604700504. https://doi.org/10.1177/875697281604700504
  8. Tim Neerup Themsen (2019). 'The processes of public megaproject cost estimation: The inaccuracy of reference class forecasting.' Financial Accountability & Management 35(4):337-352. DOI: 10.1111/faam.12210. https://doi.org/10.1111/faam.12210
  9. David Zani, Bryan T. Adey, Simon Carroll (2024). 'An approach to support reference class forecasting when adequate project data are unavailable.' Results in Engineering 22:102333. DOI: 10.1016/j.rineng.2024.102333. https://doi.org/10.1016/j.rineng.2024.102333
  10. David Zani, Bryan T. Adey (2025). 'Empirical comparison of weighted and non-weighted reference class forecasting.' Infrastructure Asset Management 12(3):147-158. DOI: 10.1680/jinam.24.00024. https://doi.org/10.1680/jinam.24.00024
  11. Chantal C. Cantarelli, Kate Davis, Jeffrey K. Pinto, Neil Turner (2025). 'Reference class forecasting: promises, problems, and a research agenda moving forward.' Production Planning & Control 37(7):691-709. DOI: 10.1080/09537287.2025.2578708. https://doi.org/10.1080/09537287.2025.2578708
  12. Costanza Mariani, Francesco Cellerino, Erik Humberto Araya Aliaga, Edison Atencio, Mauro Mancini (2025). 'The C-BA method: enhancing megaproject forecasting through the Fifth Hand principle.' International Journal of Managing Projects in Business 18(8):50-78. DOI: 10.1108/IJMPB-11-2024-0281. https://doi.org/10.1108/IJMPB-11-2024-0281
  13. Chiranjib Paul, Prabir Kumar Das (2022). 'Predicting movie revenue before committing significant investments.' Journal of Media Economics 34(2):63-90. DOI: 10.1080/08997764.2022.2066108. https://doi.org/10.1080/08997764.2022.2066108
  14. Han Bleichrodt, Martin Filko, Amit Kothiyal, Peter P. Wakker (2017). 'Making Case-Based Decision Theory Directly Observable.' American Economic Journal: Microeconomics 9(1):123-151. DOI: 10.1257/mic.20150172. https://doi.org/10.1257/mic.20150172
  15. C. Jansen, G. Schollmeyer, T. Augustin, J. Rodemann (online 2025; volume year 2026). 'Empirical decision theory.' Information Sciences 757:123906. DOI: 10.1016/j.ins.2026.123906. https://doi.org/10.1016/j.ins.2026.123906