Zwaan, Monteiro, Sherbino, Ilgen, Howey and Norman asked physicians to identify cognitive biases in written clinical case workups and found no agreement on the presence or absence of any individual bias (Cohen's kappa 0.000-0.044) and a strong hindsight effect: twice as many biases were identified when the case outcome implied a diagnostic error rather than a correct diagnosis. This left open, in the authors' own interpretation, whether error-reduction strategies built on the retrospective identification of cognitive biases are scientifically viable. The unresolved research program comprises linked questions: (1) whether larger, better-sampled, or specifically trained reviewer cohorts can identify specific biases reliably, and whether validated operational definitions and coding manuals can make such identification reproducible; (2) whether methods that combine retrospective case review with contemporaneous process capture can support valid inference about unconscious (Type 1) contributions such as availability bias; (3) to what extent retrospective attributions labelled 'bias' actually reflect knowledge gaps or system factors rather than bias; and (4) whether interventions that do not depend on accurate retrospective bias recognition (system-level checks, decision support, knowledge-focused training) reduce diagnostic error more effectively than recognition-based debiasing training. The cited literature reports that the hindsight/outcome-knowledge effect has been independently replicated in emergency care, anesthesiology, and supervision settings, while the near-zero inter-rater agreement of bias identification continues to be cited as a standing methodological problem, bias definitions remain overlapping and unvalidated, debiasing-intervention evidence remains weak or equivocal, and the bias-versus-knowledge attribution question is still actively debated. The full original question remains open.

Background

Diagnostic error — a diagnosis that is missed, delayed, or wrong — is a leading cause of preventable harm, and a large body of work attributes a substantial share of it to problems in clinicians' reasoning rather than in tests or data access. The dominant framework is dual-process theory: fast, automatic, associative reasoning (Type 1) operates alongside slow, effortful, analytical reasoning (Type 2). 'Cognitive biases' are named, repeatable deviations from normative reasoning — for example anchoring (over-relying on an initial impression), confirmation bias (seeking evidence that supports a working diagnosis rather than evidence against it), availability bias (judging a diagnosis more likely because similar cases come easily to mind), and premature closure (accepting a diagnosis before it is verified) — that are theorized to arise mainly in Type 1 processing. Since the early 2000s, prominent authors have proposed that teaching clinicians to recognize these named biases, and to 'debias' their reasoning, should reduce diagnostic error; morbidity and mortality conferences (structured case-review meetings in which complications and deaths are discussed) and root-cause analyses (systematic institutional reviews of why an incident happened) are often asked to identify which biases occurred in a case. A prerequisite of this program is that biases can be reliably identified in a clinical workup at all. Zwaan, Monteiro, Sherbino, Ilgen, Howey and Norman (BMJ Quality & Safety, 2017) tested that prerequisite with a vignette study: 37 physicians each read eight written case workups, half with outcomes implying a correct diagnosis and half implying an incorrect one, and checked off which biases from a provided list were present. Agreement between physicians on the presence or absence of each individual bias was essentially nil (kappa 0.000-0.044, where kappa is a chance-corrected agreement statistic with 1.0 meaning perfect agreement), and outcome knowledge doubled the number of biases perceived (mean 1.75 with a correct-outcome implication versus 3.45 with an incorrect one), indicating heavy hindsight bias — the tendency, once an outcome is known, to overestimate how predictable it was. The authors concluded that these results challenge error-reduction strategies based on identifying cognitive biases, and acknowledged the study's own limits: a small sample with modest response rate, the overlap and imprecision of bias definitions when applied to retrospective case descriptions, the inability of record-based review to access clinicians' internal reasoning or unconscious Type 1 processes, and the possibility that what is labelled 'bias' in hindsight reflects knowledge gaps or system factors instead. A companion review by largely the same group (Norman, Monteiro, Sherbino, Ilgen, Schmidt and Mamede, Academic Medicine, 2017) argued that educational strategies directed at recognizing biases are ineffective and that knowledge-focused strategies hold more promise, while earlier reviews (Graber et al., BMJ Quality & Safety, 2012; Ludolph and Schulz, Medical Decision Making, 2018) had already found the tested-intervention evidence base thin, especially for health professionals. The open question is thus a joint measurement-and-intervention question: can the cognitive-bias contribution to diagnostic error be measured validly and reliably at all with case-review methods, and if the answer requires moving beyond retrospective bias recognition, which alternative strategies actually reduce diagnostic error?

Problem Statement

The problem, as left open by Zwaan et al. (2017) and taken up by the subsequent literature, is to determine valid ways to measure and reduce the cognitive contributions to diagnostic error. It comprises four jointly linked questions whose common objective is to replace the unvalidated assumption — that clinicians can retrospectively recognize which cognitive biases occurred in a case workup — with a scientifically validated measurement-and-intervention basis. (1) Reliability and validity of bias identification: establish, empirically, whether physician or other reviewer cohorts (larger samples, better response rates, or raters trained and tested for bias-recognition expertise, with or without structured consensus procedures) can identify the presence or absence of specific cognitive biases in clinical case workups at acceptable, quantified inter-rater agreement; and develop validated operational definitions and coding manuals for named biases (such as anchoring, confirmation bias, availability bias, premature closure) that make such identification reproducible across raters and settings. (2) Access to underlying processes: determine whether methods that combine retrospective record review with contemporaneous process capture (structured clinician interviews, think-aloud or eye-tracking protocols, prospective observation) can yield valid inference about unconscious Type 1 contributions such as availability bias — contributions that written case descriptions cannot reveal. (3) Attribution: establish to what extent judgements labelled 'bias' in retrospective case review actually reflect knowledge gaps, system factors, or other non-bias causes, using designs capable of disentangling these alternatives. (4) Comparative intervention effectiveness — a comparison framed by the companion review of Norman et al. (2017) and the subsequent debiasing literature: determine, in controlled evaluations, whether interventions that do not depend on accurate retrospective bias recognition — system-level checks, clinical decision support, or training focused on knowledge gaps — reduce diagnostic error more effectively than recognition-based debiasing training. Together, these strands operationalize the overarching question as posed by this statement and the cited literature: which measurement methods validly quantify cognitive contributions to diagnostic error, and which intervention strategy reduces those contributions in practice. All four strands are jointly required: resolving any one strand alone does not resolve the task.

References

  1. Zwaan L, Monteiro S, Sherbino J, Ilgen J, Howey B, Norman G. Is bias in the eye of the beholder? A vignette study to assess recognition of cognitive biases in clinical case workups. BMJ Quality & Safety 2017;26(2):104-110. doi:10.1136/bmjqs-2015-005014. https://doi.org/10.1136/bmjqs-2015-005014
  2. Norman GR, Monteiro SD, Sherbino J, Ilgen JS, Schmidt HG, Mamede S. The Causes of Errors in Clinical Reasoning: Cognitive Biases, Knowledge Deficits, and Dual Process Thinking. Academic Medicine 2017;92(1):23-30. doi:10.1097/ACM.0000000000001421. https://doi.org/10.1097/ACM.0000000000001421
  3. Dhaliwal G. Premature closure? Not so fast. BMJ Quality & Safety 2017;26(2):87-89. doi:10.1136/bmjqs-2016-005267. https://doi.org/10.1136/bmjqs-2016-005267
  4. Graber ML, Kissam S, Payne VL, Meyer AN, Sorensen A, Lenfestey N, Tant E, Henriksen K, Labresh K, Singh H. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Quality & Safety 2012;21(7):535-557. doi:10.1136/bmjqs-2011-000149. https://doi.org/10.1136/bmjqs-2011-000149
  5. Ludolph R, Schulz PJ. Debiasing Health-Related Judgments and Decision Making: A Systematic Review. Medical Decision Making 2018;38(1):3-13. doi:10.1177/0272989X17716672. https://doi.org/10.1177/0272989X17716672
  6. Eichbaum Q, Adkins B, Craig-Owens L, Ferguson D, Long D, Shaver A, Stratton C. Mortality and morbidity rounds (MMR) in pathology: relative contribution of cognitive bias vs. systems failures to diagnostic error. Diagnosis 2019;6(3):249-257. doi:10.1515/dx-2018-0089. https://doi.org/10.1515/dx-2018-0089
  7. Sibbald M, Sherbino J, Ilgen JS, Zwaan L, Blissett S, Monteiro S, Norman G. Debiasing versus knowledge retrieval checklists to reduce diagnostic error in ECG interpretation. Advances in Health Sciences Education 2019;24(3):427-440. doi:10.1007/s10459-019-09875-8. https://doi.org/10.1007/s10459-019-09875-8
  8. Mamede S, Goeijenbier M, Schuit SCE, de Carvalho Filho MA, Staal J, Zwaan L, Schmidt HG. Specific Disease Knowledge As Predictor of Susceptibility to Availability Bias in Diagnostic Reasoning: a Randomized Controlled Experiment. Journal of General Internal Medicine 2021;36(3):640-646. doi:10.1007/s11606-020-06182-6. https://doi.org/10.1007/s11606-020-06182-6
  9. Loncharich MF, Robbins RC, Durning SJ, Soh M, Merkebu J. Cognitive biases in internal medicine: a scoping review. Diagnosis 2023;10(3):205-214. doi:10.1515/dx-2022-0120. https://doi.org/10.1515/dx-2022-0120
  10. Thompson J, Bujalka H, McKeever S, Lipscomb A, Moore S, Hill N, Kinney S, Cham KM, Martin J, Bowers P, Gerdtz MF. Educational strategies in the health professions to mitigate cognitive and implicit bias impact on decision making: a scoping review. BMC Medical Education 2023;23(1):455. doi:10.1186/s12909-023-04371-5. https://doi.org/10.1186/s12909-023-04371-5
  11. Plaum P, Visser LN, de Groot B, Morsink MEB, Duijst WLJM, Candel BGJ. Using case vignettes to study the presence of outcome, hindsight, and implicit bias in acute unplanned medical care: a cross-sectional study. European Journal of Emergency Medicine 2024;31(4):260-266. doi:10.1097/MEJ.0000000000001127. https://doi.org/10.1097/MEJ.0000000000001127
  12. Millan PD, Kleiman AM, Friedman JF, Dunn LK, Gui JL, Bechtel AJ, Collins SR, Huffmyer JL, Dwivedi P, Wolpaw JT, Nemergut EC, Tsang S, Forkin KT. The impact of hindsight bias on the diagnosis of perioperative events by anesthesia providers: A multicenter randomized crossover study. Journal of Clinical Anesthesia 2024;97:111549. doi:10.1016/j.jclinane.2024.111549. https://doi.org/10.1016/j.jclinane.2024.111549
  13. Staal J, Alsma J, Van der Geest J, Mamede S, Jansen E, Frens MA, Van den Broek WW, Zwaan L. Selective processing of clinical information related to correct and incorrect diagnoses: An eye-tracking experiment. Medical Education 2025;59(5):540-549. doi:10.1111/medu.15544. https://doi.org/10.1111/medu.15544
  14. van Sassen C, Mamede S, van den Broek W, Bindels P, Zwaan L. Does Knowledge of Clinical Case Outcome Influence Supervisor Evaluation of Resident Clinical Reasoning? Academic Medicine 2025;100(10):1194-1202. doi:10.1097/ACM.0000000000006122. https://doi.org/10.1097/ACM.0000000000006122
  15. Pelaccia T, Sherbino J, Wyer P. From bias to knowledge: A necessary shift to understand diagnostic errors. Academic Emergency Medicine 2025;32(9):1043-1044. doi:10.1111/acem.70056. https://doi.org/10.1111/acem.70056
  16. Martínez JA, Martínez L. Distinguishing anchoring from confirmation bias in diagnostic vignette studies: methodological implications for clinical decision-making. Frontiers in Public Health 2026;14:1770198. doi:10.3389/fpubh.2026.1770198. https://doi.org/10.3389/fpubh.2026.1770198