The ACL 2024 Findings paper that introduced the TOMATO task (automated open-domain hypothetical induction) and the MOOSE framework was the first to show that large language models can generate scientific hypotheses that experts judge both novel (not existing in the literature) and valid (reflecting reality), but only in social science, with a handcrafted heuristic standing in for automated corrective feedback. The authors' own follow-up line (MOOSE-Chem, ICLR 2025; MOOSE-Chem2, NeurIPS 2025; MOOSE-Chem3) resolved the generalization question in the rediscovery sense — LLM pipelines can regenerate unseen, Nature/Science-level chemistry hypotheses from open-domain inputs under contamination control — and replaced heuristic feedback with LLM-internal scoring for benchmark evaluation. What survives as a precise open problem is the paper's stated final goal at its genuinely broken point: a system that, in a natural-science discipline, takes only open-domain material as input, proposes hypotheses that are simultaneously novel to the literature and independently confirmed as valid by domain experts, and generates its own corrective feedback in place of the human heuristic, all under data-contamination control. The verification bottleneck this exposes is recognized field-wide as the central open challenge of AI-driven scientific discovery.

Background

Hypothetical induction is the reasoning pattern by which scientists move from observations to hypotheses that explain them. Yang et al. (ACL 2024 Findings; arXiv:2309.02726) formalized an automated version of this, called TOMATO (auTOmated opendoMAin hypoThetical inductiOn): given only raw open-domain corpora (web text and literature) rather than handpicked, close-domain annotations, a system must propose research hypotheses that are simultaneously novel — not existing in the literature, 'new to humanity' — and valid — 'reflecting reality'. They built the first dataset for the task, 50 recent papers from top social-science journals across 7 disciplines, each annotated with background, inspirations, reasoning, and the ground-truth hypothesis, and developed MOOSE, a multi-module pipeline that searches a research background, retrieves inspiration sources, and composes hypotheses, with three feedback mechanisms: present-feedback evaluates a module's output immediately; past-feedback retrospectively corrects an earlier module's output once a later, evaluable output reveals its quality; future-feedback (variants FF1 and FF2) passes information forward to improve later modules. Evaluation combines GPT-4-based scoring with PhD-student expert scoring on validness, novelty, and helpfulness. A key limitation the authors themselves flag: ChatGPT could not produce good enough past-feedback for the inspiration-selection module, so they substituted a human heuristic ('if the inspiration titles are less related to the background, then more novel hypotheses are likely'), noting that more powerful LLMs might one day find better feedback than human heuristics. They also state that whether the framework applies to disciplines other than social science — chosen because social-science data is language-native — is open, and that their 50-paper benchmark invites scaling to reduce collection bias. The paper's stated final goal is systems that 'automatically generate valid, novel, and helpful scientific hypotheses, given only a pile of raw web corpus'. After 2024 the same group extended the program to the natural sciences: MOOSE-Chem (ICLR 2025) showed, via a benchmark of 51 high-impact chemistry papers published after January 2024 and annotated by PhD chemists — contamination-controlled in that the papers appeared only after the language models' training-data cutoff, so the target hypotheses could not have leaked into pretraining — that an agentic LLM framework (the model itself autonomously planning the retrieval, composition, and ranking steps) can rediscover many unseen hypotheses with high similarity to ground truth, instantiating a formal decomposition h=f(b,i1,…,ik)h = f(b, i_1, \ldots, i_k) in which a chemistry hypothesis is composed from a background bb and a small set of inspirations. MOOSE-Chem2 (NeurIPS 2025) pushed toward fine-grained, experimentally actionable hypotheses, using the LLM's own internal scoring as a reward landscape over the hypothesis space; MOOSE-Chem3 studied experiment-guided hypothesis ranking with a simulator standing in for wet-lab feedback; MOOSE-Copilot (ACL 2026 Demonstrations) added a human-steerable web interface. Crucially, all of these evaluate by similarity to held-out ground truth (rediscovery) or simulated feedback — none demonstrates the generation and independent confirmation of genuinely new-to-humanity natural-science hypotheses, and a 2025 position paper by researchers at Samsung AI, IBM Research, and Imperial College London (Cornelio et al.) argues that scalable, reliable verification of machine-generated hypotheses is now the cornerstone open problem for AI-driven scientific discovery, since generation already far outpaces any mechanism for establishing validity. Natural-science hypotheses differ from social-science ones in exactly the way that makes this residue hard: their validity is a claim about an experimentally testable world, not about a text-native literature, and ground truth for a genuinely new hypothesis does not yet exist.

Problem Statement

Can an open-domain automated hypothesis-discovery system of the TOMATO/MOOSE type meet the source paper's stated final goal — automatically generating valid, novel, and helpful scientific hypotheses given only raw open-domain input — in a natural-science discipline? Concretely, construct and rigorously evaluate a system that: (1) takes as its only human-provided input a research question and/or survey material in a natural-science discipline (e.g., chemistry, materials science, biomedicine), retrieving all background and inspiration material itself from open-domain corpora; (2) proposes hypotheses that are simultaneously novel — not present in existing literature — and valid — reflecting reality, as established by independent, blinded domain-expert evaluation under a published rubric (in the spirit of the source paper's validness/novelty/helpfulness protocol); and (3) implements its corrective feedback (the past-feedback role that the source paper had to fill with a human heuristic) with model-generated signals. Evaluation must control for data contamination, e.g., by requiring demonstrated rediscovery performance on ground-truth hypotheses published after the underlying models' knowledge cutoffs (as in the MOOSE-Chem protocol) alongside the novelty claims for new-to-literature hypotheses. A complete answer establishes, for at least one natural-science discipline, whether the full hypothesis-generation-and-confirmation cycle can be automated from open-domain inputs, or provides comparably rigorous evidence that it cannot.

References

  1. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, Erik Cambria. Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. ACL 2024 Findings. arXiv:2309.02726, DOI 10.48550/arXiv.2309.02726. https://arxiv.org/abs/2309.02726
  2. Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, Dongzhan Zhou. MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses. ICLR 2025. arXiv:2410.07076, DOI 10.48550/arXiv.2410.07076. https://arxiv.org/abs/2410.07076
  3. Zonglin Yang, Wanhao Liu, Ben Gao, Yujie Liu, Wei Li, Tong Xie, Lidong Bing, Wanli Ouyang, Erik Cambria, Dongzhan Zhou. MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search. NeurIPS 2025. arXiv:2505.19209, DOI 10.48550/arXiv.2505.19209. https://arxiv.org/abs/2505.19209
  4. Wanhao Liu, Zonglin Yang, Jue Wang, Lidong Bing, Di Zhang, Dongzhan Zhou, Yuqiang Li, Houqiang Li, Erik Cambria, Wanli Ouyang. MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedback. arXiv:2505.17873, DOI 10.48550/arXiv.2505.17873. https://arxiv.org/abs/2505.17873
  5. Cristina Cornelio, Takuya Ito, Ryan Cory-Wright, Sanjeeb Dash, Lior Horesh. The Need for Verification in AI-Driven Scientific Discovery. arXiv:2509.01398, DOI 10.48550/arXiv.2509.01398. https://arxiv.org/abs/2509.01398
  6. Hongran An, Zonglin Yang. MOOSE-Copilot: A Web-Based Interactive Assistant for Unified Exploratory and Fine-Grained Scientific Hypothesis Discovery. ACL 2026 System Demonstrations. arXiv:2605.29475, DOI 10.48550/arXiv.2605.29475. https://arxiv.org/abs/2605.29475