In a field study with 31 forecasters from four companies, Alvarado-Valencia, Barrero, Önkal and Dennerlein (2017) showed that judgmental adjustment of statistical (system) demand forecasts improves accuracy over both the unadjusted system forecast and two alternative integration methods (50-50 combination and divide-and-conquer) only under joint conditions: high forecaster expertise, low credibility of the system forecast, and a strong need for correction. They left open whether this deployment rule survives contact with real forecasting organizations: the study rotated integration methods monthly without performance feedback (so learning effects could not be observed), elicited forecasts without decision consequences (so organizational and political pressures were absent), hand-picked a few products per company rather than the thousands of stock-keeping units real planners face, and excluded intermittent demand. The open problem is to establish the conditions under which judgmental adjustment improves accuracy in practice: criteria for selecting the products and aggregation levels for which expertise can most clearly make a difference when expert attention is scarce (together with valid, scalable measurement of forecaster expertise and system-forecast credibility), the robustness of the expertise-credibility-need-for-correction effect to repeated outcome-informed exposure, the influence of accountability and organizational pressures, and generalization across industries, product types, and intermittent-demand series.

Background

Companies that manufacture or retail products typically produce demand forecasts in two steps. A computerized forecasting system first fits statistical models (such as exponential smoothing, an exponentially weighted average of past observations) to the historical sales of each product and generates a 'system forecast'; human demand planners then modify this number using contextual knowledge the models cannot see, such as planned promotions, competitor actions, or supply disruptions. This modification is called judgmental adjustment, and it is usually performed through a forecasting support system (FSS) — software that displays the historical sales series together with the system forecast. A stock-keeping unit (SKU) is the finest level at which inventory of a product is tracked; real planners must forecast thousands of SKUs, at one of several aggregation levels (per SKU, per product family, per location, per region). A demand series is called intermittent when many periods have zero demand, which both complicates forecasting and makes percentage-error accuracy measures ill-behaved. The most influential field evidence on whether judgmental adjustment pays off is Fildes, Goodwin, Lawrence and Nikolopoulos (2009), who analyzed over 60,000 forecasts from four supply-chain companies: adjustments increased accuracy on average in three of the companies, but the benefit came from relatively large adjustments — predominantly downward ones — while the numerous small and upward adjustments often damaged accuracy, indicating a systematic bias toward optimism. Alvarado-Valencia, Barrero, Önkal and Dennerlein (2017) extended this by comparing three ways of integrating expert judgment with system forecasts in a field study with 31 forecasters from four companies: (i) judgmental adjustment, in which the forecaster sees the historical series and the system forecast and issues a final forecast; (ii) the 50-50 combination, in which the forecaster forecasts without seeing the system forecast and the two forecasts are then averaged; and (iii) divide-and-conquer, in which the forecaster sees neither the historical series nor the system forecast value, only a description of how the system forecast is generated, and decides whether and how to modify it. Forecaster expertise was measured with the knowledge subscale of the Germain-Tejeda scale, a peer-rated psychometric instrument for recognized job expertise, and the credibility of system forecasts with an adaptation of Meyer's source-credibility scale, an instrument originally developed for newspaper credibility research. The headline result: judgmental adjustment improved accuracy relative to both the alternative methods and the unadjusted system forecast only when (a) the forecaster's relative expertise was high, (b) the relative credibility of the system forecast was low, and (c) the system forecast was in strong need of correction (substantially inaccurate, leaving room for improvement); higher-expertise forecasters also adjusted more often. In the study, a period was labeled 'correction needed' when the realized demand value for a product fell outside the prediction intervals calculated from the system forecast, and 'correction not needed' when the actual value lay inside the 95% prediction interval, so the need for correction became known only after the demand data were collected. The dependent variable was accuracy improvement, measured as the reduction in average percentage error, APEt=100×∣Yt−Ft∣/YtAPE_t = 100 \times |Y_t - F_t| / Y_t with YtY_t the realized demand and FtF_t the final forecast, that is APEIMP=APEsystem forecast−APEintegrated forecastAPE^{IMP} = APE_{\text{system forecast}} - APE_{\text{integrated forecast}}; because every series in the study was strictly positive, scaled error measures such as the average scaled error entered only as robustness checks. This matters because it suggests a deployment rule: expert adjustment improves accuracy only in a specific region of the expertise-credibility-need-for-correction space, while expert attention is too scarce to be spent on every product and aggregation level. The authors interpreted the low-credibility benefit as healthy skepticism counteracting anchoring, the cognitive bias whereby people stick too closely to an initial value (here the system forecast) and adjust insufficiently from it. The study design, however, could not establish whether this rule survives real forecasting organizations. The forecasters rotated among the three methods each month and received no performance feedback, so possible learning effects could not be observed; the authors noted that earlier experimental work (Lim and O'Connor, 1995) had found only small or no learning from outcome feedback, but that evidence predates the expertise-credibility framework. The forecasts had no consequences for real decisions or performance evaluations, so organizational and political pressures were absent even though some elicited adjustment reasons pointed to goals and perceived control. Only a handful of products per company were studied, with products and aggregation levels hand-picked by the researchers, whereas real planners face thousands of SKUs under time pressure; and intermittent demand was excluded altogether. The published study therefore closes with an explicit call for best practices in selecting the products and aggregation levels for which expertise can most clearly make a difference, for validation of the expertise measure in other contexts, and for experimental work on anchoring under different levels of system-forecast credibility. The authors also observed that, although it is not possible to know in advance whether a modification will be needed, there are occasions on which such tweaks are clearly warranted (supply-chain restrictions, structural changes in the series, inflection points in a product life cycle, sudden environmental changes, anticipated competitor moves).

Problem Statement

The problem is to establish the conditions under which judgmental adjustment of statistical demand forecasts improves forecast accuracy in real demand-planning organizations, along the directions left open by Alvarado-Valencia, Barrero, Önkal and Dennerlein (2017). The question remains open in four related directions: (1) Selection criteria for scale - criteria for selecting, without per-product expert elicitation, the products and aggregation levels for which expertise can most clearly make a difference, in assortments of thousands of SKUs where expert attention is scarce, together with valid and scalable measurement of forecaster expertise and system-forecast credibility in these settings; (2) Dynamics - whether the joint effect of forecaster expertise, system-forecast credibility, and need for correction on adjustment accuracy persists, strengthens, or attenuates under repeated, outcome-informed exposure, that is, when forecasters keep the same integration method and observe the accuracy consequences of their adjustments over time; (3) Organizational context - how real decision consequences, accountability, and political and goal pressures change adjustment behavior and its accuracy relative to the consequence-free elicitation of the original study; and (4) Generalization - whether the deployment conditions hold across industries, product types, and intermittent-demand series, or how they must be modified. The four directions are jointly the components of this single open problem - each is independently open, so settling or refuting any one leaves the others open, and direction (1) itself bundles two separately assessable deliverables (a criterion for selecting products and aggregation levels, and a valid, scalable measurement of forecaster expertise and system-forecast credibility) that must both be established. Throughout, the outcome of interest is the accuracy improvement of the integrated forecast over the unadjusted system forecast, measured as in the source by improvement in average percentage error where demand is strictly positive, and by a scaled error measure - to be chosen and justified - for intermittent series, in which percentage errors are undefined at zero demand. Adjacent support-system design questions the source also raised (whether providing only the system forecast without the full historical track suffices, and how to elicit expert knowledge qualitatively) are recorded here as adjacent open items rather than part of this core.

References

  1. Jorge Alvarado-Valencia, Lope H. Barrero, Dilek Önkal, Jack T. Dennerlein (2017), 'Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting', International Journal of Forecasting 33(1):298-313. DOI: 10.1016/j.ijforecast.2015.12.010. https://doi.org/10.1016/j.ijforecast.2015.12.010 (source paper)
  2. Robert Fildes, Paul Goodwin, Michael Lawrence, Konstantinos Nikolopoulos (2009), 'Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning', International Journal of Forecasting 25(1):3-23. DOI: 10.1016/j.ijforecast.2008.11.010. https://doi.org/10.1016/j.ijforecast.2008.11.010
  3. Joa Sang Lim, Marcus O'Connor (1995), 'Judgemental adjustment of initial forecasts: Its effectiveness and biases', Journal of Behavioral Decision Making 8(3):149-168. DOI: 10.1002/bdm.3960080302. https://doi.org/10.1002/bdm.3960080302
  4. Marie-Line Germain, Manuel J. Tejeda (2012), 'A preliminary exploration on the measurement of expertise: An initial development of a psychometric scale', Human Resource Development Quarterly 23(2):203-232. DOI: 10.1002/hrdq.21134. https://doi.org/10.1002/hrdq.21134
  5. Robert Fildes, Paul Goodwin, Shari De Baets (2025), 'Forecast value added in demand planning', International Journal of Forecasting 41(2):649-669. DOI: 10.1016/j.ijforecast.2024.07.006. https://doi.org/10.1016/j.ijforecast.2024.07.006
  6. Budhi S. Wibowo (2024), 'When and how to adjust statistical forecasts in supply chains? Insight from causal machine learning', Journal of Business Analytics 7(1):25-41. DOI: 10.1080/2573234X.2023.2248203. https://doi.org/10.1080/2573234X.2023.2248203
  7. Maud Van den Broeke, Shari De Baets, Ann Vereecke, Philippe Baecke, Karlien Vanderheyden (2019), 'Judgmental forecast adjustments over different time horizons', Omega 87:34-45. DOI: 10.1016/j.omega.2018.09.008. https://doi.org/10.1016/j.omega.2018.09.008
  8. Larissa Koupriouchina, Jean-Pierre van der Rest, Zvi Schwartz (2023), 'Judgmental adjustments of algorithmic hotel occupancy forecasts: Does user override frequency impact accuracy at different time horizons?', Tourism Economics 29(8):2143-2164. DOI: 10.1177/13548166221126572. https://doi.org/10.1177/13548166221126572
  9. Nikolaos Kourentzes, Robert Fildes (2023), 'The dynamics of judgemental adjustments in demand planning', SSRN working paper. DOI: 10.2139/ssrn.4534744. https://doi.org/10.2139/ssrn.4534744
  10. Behnam Fahimnia, Meysam Arvan, Tarkan Tan, Enno Siemsen (2023), 'A hidden anchor: The influence of service levels on demand forecasts', Journal of Operations Management 69:856-871. DOI: 10.1002/joom.1229. https://doi.org/10.1002/joom.1229
  11. Sarah A. Jessup, Gene M. Alarcon, Sasha M. Willis, Michael A. Lee (2024), 'A closer look at how experience, task domain, and self-confidence influence reliance towards algorithms', Applied Ergonomics 121:104363. DOI: 10.1016/j.apergo.2024.104363. https://doi.org/10.1016/j.apergo.2024.104363
  12. H. Niles Perera, Jason Hurley, Behnam Fahimnia, Mohsen Reisi (2019), 'The human factor in supply chain forecasting: A systematic review', European Journal of Operational Research 274(2):574-600. DOI: 10.1016/j.ejor.2018.10.028. https://doi.org/10.1016/j.ejor.2018.10.028
  13. Meysam Arvan, Behnam Fahimnia, Mohsen Reisi, Enno Siemsen (2019), 'Integrating human judgement into quantitative forecasting methods: A review', Omega 86:237-252. DOI: 10.1016/j.omega.2018.07.012. https://doi.org/10.1016/j.omega.2018.07.012
  14. Fergus Bolger, George Wright (2017), 'Use of expert knowledge to anticipate the future: Issues, analysis and directions', International Journal of Forecasting 33(1):230-243. DOI: 10.1016/j.ijforecast.2016.11.001. https://doi.org/10.1016/j.ijforecast.2016.11.001
  15. Stefanie Mauksch, Heiko A. von der Gracht, Theodore J. Gordon (2020), 'Who is an expert for foresight? A review of identification methods', Technological Forecasting and Social Change 154:119982. DOI: 10.1016/j.techfore.2020.119982. https://doi.org/10.1016/j.techfore.2020.119982
  16. Rebekah Brau, John Aloysius, Enno Siemsen (2023), 'Demand planning for the digital supply chain: How to integrate human judgment and predictive analytics', Journal of Operations Management 69:965-982. DOI: 10.1002/joom.1257. https://doi.org/10.1002/joom.1257
  17. B.B.J.P.J. van der Staak, R.J.I. Basten, P.P.F.M. van de Calseyde, E. Demerouti, A.G. de Kok (2025), 'Light-touch forecasting: A novel method to combine human judgment with statistical algorithms', International Journal of Forecasting 41(2):440-451. DOI: 10.1016/j.ijforecast.2024.04.003. https://doi.org/10.1016/j.ijforecast.2024.04.003
  18. Robert Fildes, Paul Goodwin, Dilek Önkal (2019), 'Use and misuse of information in supply chain forecasting of promotion effects', International Journal of Forecasting 35(1):144-156. DOI: 10.1016/j.ijforecast.2017.12.006. https://doi.org/10.1016/j.ijforecast.2017.12.006
  19. Robert Fildes, Shaohui Ma, Stephan Kolassa (2022), 'Retail forecasting: Research and practice', International Journal of Forecasting 38:1283-1318. DOI: 10.1016/j.ijforecast.2019.06.004. https://doi.org/10.1016/j.ijforecast.2019.06.004
  20. Yacine Rekik, Christoph H. Glock, Aris A. Syntetos (2017), 'Enriching demand forecasts with managerial information to improve inventory replenishment decisions: Exploiting judgment and fostering learning', European Journal of Operational Research 261:182-194. DOI: 10.1016/j.ejor.2017.02.001. https://doi.org/10.1016/j.ejor.2017.02.001
  21. Anna Sroginis, Robert Fildes, Nikolaos Kourentzes (2023), 'Use of contextual and model-based information in adjusting promotional forecasts', European Journal of Operational Research 307(3):1177-1191. DOI: 10.1016/j.ejor.2022.10.005. https://doi.org/10.1016/j.ejor.2022.10.005