Literature-based discovery (LBD) automatically infers unstated relations between concepts scattered across disjoint literatures, but its open-discovery output is a flood of candidate associations whose status - genuinely new knowledge versus already-public or trivially inferable knowledge, and valid versus spurious - cannot currently be determined at prediction time. This problem asks for a sound, empirically established validation methodology for open-discovery LBD output: one that scores candidate relations for novelty and for scientific validity or worthiness of follow-up at a scale beyond the classic Swanson-style replications, avoids the documented biases of replication-based and time-sliced evaluation, and validates its own judgments against ground truth independent of the LBD system's input literature.

Background

Literature-based discovery (LBD) is a text-mining paradigm founded on Don Swanson's 1986 observation of "undiscovered public knowledge": scientific findings are scattered across non-communicating specialties, so two published facts may jointly imply a third relation that no single paper states. Swanson's canonical example connected dietary fish oil to Raynaud's syndrome (a painful disorder of the small blood vessels) through their shared associations with blood viscosity; the inferred connection was later supported by a double-blind, controlled, prospective clinical trial (DiGiacomo et al., 1989). The standard formalization is the A-B-C model: if the literature relates concept A to concept B, and B to concept C, then A-C is a candidate hidden connection. In closed discovery both A and C are fixed and only linking terms B are sought; in open discovery only A is fixed and all connections are pursued, which is where the discovery potential - and the validation problem - is largest. Modern pipelines extract subject-predicate-object relation triples from all of PubMed (the U.S. National Library of Medicine's citation database) with tools such as SemRep, which maps text to concepts in the Unified Medical Language System (UMLS) Metathesaurus, and search the resulting concept graph; at this scale open discovery yields millions to billions of candidate hidden knowledge pairs (CHKPs), so filtering and ranking are integral parts of every system (for example HiDE and Serial KinderMiner).

Evaluating and validating these candidates is obstructed by a structural catch: a genuinely new discovery has, by definition, no published gold standard. Two evaluation conventions dominate. Replication replays a known historical discovery (almost always one of Swanson's few cases) on literature predating it; it is statistically fragile - a handful of instances, reused across three decades of papers, invites confirmation bias, data leakage, and conclusions that cannot be generalized. Time-sliced evaluation generates candidates from literature before a cutoff date and treats relations first published after the cutoff as the gold standard; it scales, but its gold standard consists mostly of chance co-occurrences and of "background knowledge" - relations that are trivial or already public - so systems are rewarded for re-deriving what is effectively known rather than for discovering anything. Preiss (2023) demonstrated this concretely: the measured precision of the same LBD system changed qualitatively depending on how much background knowledge the gold standard rewards, and she concluded that general time-sliced gold standards may present misleading results. Moreau, Hardiman, Heverin and O'Sullivan (2024) diagnosed evaluation practice in the field as "methodologically poor": replication is biased and statistically unreliable, co-occurrence-based time-slicing is "not fit for purpose", and the lack of a properly designed, decent-sized benchmark is a principal reason LBD has not been adopted as a practical biomedical research tool. LBD output therefore remains a stream of hypotheses "worthy of further investigation" that require costly manual curation before any expert or experimental follow-up. Recent responses - benchmark construction from literature-impact surges (Moreau et al., 2024), LLM-based hypothesis evaluation (SKiM-GPT, 2025), reproducible pipelines over classic benchmark data (Cestnik et al., 2025), and self-evaluating LLM agents (BioVerge, 2025) - each leave the central question open: none establishes a validation methodology whose novelty and validity judgments are themselves shown sound against ground truth independent of the LBD system's input literature.

Problem Statement

Construct, and empirically establish as sound, a validation methodology for open-discovery literature-based discovery output: a procedure that, applied to candidate hidden-knowledge pairs (or more expressive literature-derived hypotheses) proposed at time t, (i) scores or classifies each candidate's novelty - separating relations that are genuinely new from those that are already public, whether explicitly published before t or trivially inferable as background knowledge from the pre-t literature - and (ii) scores or classifies each candidate's likely scientific validity or worthiness of follow-up, so that the filtered output can be trusted enough to justify expert attention or experimental testing. The methodology must be accompanied by evidence of its own soundness at a scale beyond the traditional handful of Swanson-type replication cases: its novelty judgments must demonstrably avoid the background-knowledge rewarding of time-sliced gold standards (for example through contrastive evaluation against gold standards of differing background-knowledge content), and its validity judgments must be validated against ground truth independent of the LBD system's own input literature - for example prospective outcomes such as later experimental confirmation or independently dated literature emergence, or calibrated agreement with expert judgment gathered at sufficient scale. All four requirements - (i) the novelty scoring or classification procedure, (ii) the validity scoring or classification procedure, (iii) the evidence that the novelty judgments avoid background-knowledge-rewarding evaluation, and (iv) the evidence that the validity judgments agree with independent ground truth - are jointly mandatory. The submission must itself state and defend an operational acceptance criterion for soundness - the evidence scale at which it is established, any agreement statistic it reports, and the baseline or test by which its novelty judgments are shown to separate from background-knowledge-rewarding evaluation - so that its claimed soundness is decidable.

References

  1. Don R. Swanson, "Fish Oil, Raynaud's Syndrome, and Undiscovered Public Knowledge", Perspectives in Biology and Medicine, 1986. DOI: 10.1353/pbm.1986.0087, https://doi.org/10.1353/pbm.1986.0087
  2. Don R. Swanson, "Migraine and Magnesium: Eleven Neglected Connections", Perspectives in Biology and Medicine, 1988. DOI: 10.1353/pbm.1988.0009, https://doi.org/10.1353/pbm.1988.0009
  3. Ralph A. DiGiacomo, Joel M. Kremer, Dhiraj M. Shah, "Fish-oil dietary supplementation in patients with Raynaud's phenomenon: a double-blind, controlled, prospective study", The American Journal of Medicine, 1989. DOI: 10.1016/0002-9343(89)90261-1, https://doi.org/10.1016/0002-9343(89)90261-1
  4. Ronald N. Kostoff, "Validating discovery in literature-based discovery", Journal of Biomedical Informatics, 2007. DOI: 10.1016/j.jbi.2007.05.001, https://doi.org/10.1016/j.jbi.2007.05.001
  5. Wanda Pratt, Meliha Yetisgen-Yildiz, "Response to "Validating discovery in literature-based discovery"", Journal of Biomedical Informatics, 2007. DOI: 10.1016/j.jbi.2007.07.002, https://doi.org/10.1016/j.jbi.2007.07.002
  6. Meliha Yetisgen-Yildiz, Wanda Pratt, "A new evaluation methodology for literature-based discovery systems", Journal of Biomedical Informatics, 2009. DOI: 10.1016/j.jbi.2008.12.001, https://doi.org/10.1016/j.jbi.2008.12.001
  7. Sam Henry, Bridget T. McInnes, "Literature Based Discovery: Models, methods, and trends", Journal of Biomedical Informatics, 2017. DOI: 10.1016/j.jbi.2017.08.011, https://doi.org/10.1016/j.jbi.2017.08.011
  8. Judita Preiss, Mark Stevenson, "Quantifying and filtering knowledge generated by literature based discovery", BMC Bioinformatics, 2017. DOI: 10.1186/s12859-017-1641-9, https://doi.org/10.1186/s12859-017-1641-9
  9. Judita Preiss, Mark Stevenson, "HiDE: A Tool for Unrestricted Literature Based Discovery", Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, 2018, pp. 34-37. ACL Anthology: C18-2008, https://aclanthology.org/C18-2008/
  10. Menasha Thilakaratne, Katrina Falkner, Thushari Atapattu, "A Systematic Review on Literature-based Discovery", ACM Computing Surveys, 2019. DOI: 10.1145/3365756, https://doi.org/10.1145/3365756
  11. Andrej Kastrin, Dimitar Hristovski, "Scientometric analysis and knowledge mapping of literature-based discovery (1986-2020)", Scientometrics, 2021. DOI: 10.1007/s11192-020-03811-z, https://doi.org/10.1007/s11192-020-03811-z
  12. Kalpana Raja, John Steill, Ian Ross, Lam C. Tsoi, Finn Kuusisto, Zijian Ni, Miron Livny, James Thomson, Ron Stewart, "SKiM - A generalized literature-based discovery system for uncovering novel biomedical knowledge from PubMed", bioRxiv, 2020. DOI: 10.1101/2020.10.16.343012, https://doi.org/10.1101/2020.10.16.343012
  13. Judita Preiss, "Avoiding background knowledge: literature based discovery from important information", BMC Bioinformatics 23(Suppl 9):570, 2023. DOI: 10.1186/s12859-022-04892-8, https://doi.org/10.1186/s12859-022-04892-8
  14. Erwan Moreau, "Literature-based discovery: addressing the issue of the subpar evaluation methodology", Bioinformatics (ISMB/ECCB 2023 proceedings), 2023. DOI: 10.1093/bioinformatics/btad090, https://doi.org/10.1093/bioinformatics/btad090
  15. Robert J. Millikin, Kalpana Raja, John Steill, Cannon Lock, Xuancheng Tu, Ian Ross, Lam C. Tsoi, Finn Kuusisto, Zijian Ni, Miron Livny, Brian Bockelman, James Thomson, Ron Stewart, "Serial KinderMiner (SKiM) discovers and annotates biomedical knowledge using co-occurrence and transformer models", BMC Bioinformatics, 2023. DOI: 10.1186/s12859-023-05539-y, https://doi.org/10.1186/s12859-023-05539-y
  16. Qingyun Wang, Doug Downey, Heng Ji, Tom Hope, "SciMON: Scientific Inspiration Machines Optimized for Novelty", 2023. arXiv:2305.14259, https://arxiv.org/abs/2305.14259
  17. Erwan Moreau, Orla Hardiman, Mark Heverin, Declan O'Sullivan, "Mining impactful discoveries from the biomedical literature", BMC Bioinformatics 25:303, 2024. DOI: 10.1186/s12859-024-05881-9, https://doi.org/10.1186/s12859-024-05881-9
  18. Bojan Cestnik, Andrej Kastrin, Boshko Koloski, Nada Lavrač, "Make Literature-Based Discovery Great Again through Reproducible Pipelines", 2025. arXiv:2502.16450, https://arxiv.org/abs/2502.16450
  19. Andrej Kastrin, Bojan Cestnik, Nada Lavrač, "Recent Advances and Future Directions in Literature-Based Discovery", 2025. arXiv:2506.12385, https://arxiv.org/abs/2506.12385
  20. Jack Freeman, Robert J. Millikin, Leo Xu, Ishaan Sharma, Bethany Moore, Cannon Lock, Kevin Shine George, Aviral Bal, Chitrasen Mohanty, Ron Stewart, "SKiM-GPT: Combining Biomedical Literature-Based Discovery with Large Language Model Hypothesis Evaluation", bioRxiv, 2025. DOI: 10.1101/2025.07.28.664797, https://doi.org/10.1101/2025.07.28.664797
  21. Fuyi Yang, Chenchen Ye, Mingyu Derek Ma, Yijia Xiao, Matthew Yang, Wei Wang, "BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation", 2025. arXiv:2511.08866, https://arxiv.org/abs/2511.08866
  22. Nada Lavrač, Bojan Cestnik, Andrej Kastrin, "Bisociative Literature-Based Discovery: Methods with Tutorials in Python", Springer Nature Switzerland, 2025. DOI: 10.1007/978-3-031-96863-1, ISBN 978-3-031-96862-4, https://doi.org/10.1007/978-3-031-96863-1
  23. Dan Ofer, Michal Linial, Dafna Shahaf, "InterFeat: a pipeline for finding interesting scientific features", Scientific Reports, 2026. DOI: 10.1038/s41598-026-43169-5, https://doi.org/10.1038/s41598-026-43169-5