Whether standard large-language-model pretraining objectives — autoregressive next-token prediction or masked-token prediction — are fundamentally sufficient to induce genuine, generalizable logical reasoning over natural language, or whether observed benchmark performance substantially reflects dataset-specific pattern learning, contamination, and recitation of memorized content. The question was posed by Luo et al. (2023) from their LogiGLUE benchmark and LogiT5 experiments, which found near-ceiling in-domain accuracy on synthetic datasets far above cross-dataset averages, an abductive-greater-than-deductive-greater-than-inductive capability ranking, contamination-suspected advantages on public examination datasets, and chain-of-thought or knowledge-distillation gains only under narrow conditions. Answering it centrally requires a contamination-controlled determination of sufficiency across deductive, inductive, and abductive reasoning; separating recitation from inference, characterizing when chain-of-thought transfer, instruction fine-tuning, and few-shot in-context learning genuinely improve out-of-domain logical reasoning, and robustly evaluating free-form outputs are supporting measurement challenges that bear on how far such a determination can be trusted. Later work (counterfactual evaluation, compositionality analyses, contamination-matched twins, mechanistic analyses of internal circuits, and reinforcement-learning-trained reasoning models) documents facets of the concern but leaves the sufficiency question open.

Background

Large language models (LLMs) are neural networks, almost universally based on the Transformer architecture, trained on massive text corpora before being adapted to downstream tasks. Two pretraining objectives dominate: autoregressive next-token prediction (predicting the following token from the preceding context, used by GPT- and LLaMA-style decoder models) and masked-token prediction (recovering randomly hidden tokens using bidirectional context, used by BERT- and Flan-T5-style models). Logical reasoning over natural language — answering questions that follow necessarily or plausibly from stated premises rather than from world knowledge alone — is conventionally divided into deductive reasoning (deriving conclusions that follow necessarily from rules and facts), abductive reasoning (inferring the most plausible explanation for an observation), and inductive reasoning (generalizing an underlying rule from instances).

Luo et al. (2023) assembled LogiGLUE, a benchmark of 24 datasets spanning these three reasoning types, trained an instruction-fine-tuned model called LogiT5 (instruction fine-tuning means supervised training on many tasks phrased as natural-language instructions; chain-of-thought, or CoT, prompting means eliciting explicit intermediate reasoning steps before the final answer), and evaluated GPT-4, GPT-3.5, LLaMA-2-Chat, and Flan-T5 zero-shot on leave-out out-of-domain (OOD) datasets. Their findings motivate the open question. First, fine-tuned models reach near-ceiling accuracy on their own synthetic in-domain test sets while scoring far lower on average, and the top in-domain results are almost all on synthetic datasets, suggesting the models learn heuristic surface patterns rather than the underlying logic. Second, on the OOD evaluation all LLMs rank abductive above deductive above inductive reasoning. Third, multi-task instruction fine-tuning lifts average OOD performance but with large per-dataset variability. Fourth, CoT prompting gave only a modest, 'not markedly significant' advantage, and CoT knowledge distillation (training a smaller student model on reasoning chains generated by a larger model) improved the student only when a large 15K-sample distilled corpus was used, with 3K and 6K samples giving no benefit. Fifth, on ReClor and LogiQA — publicly available examination-derived datasets — large LLMs dramatically outperform the fine-tuned model, which the authors attribute to the possible presence of these datasets in pretraining corpora, i.e. recall or recitation rather than reasoning; free-form model outputs were also hard to score, requiring synonym handling via ConceptNet (a large public knowledge graph of common-sense concepts and relations, used here to match answer-word synonyms), and LLaMA-2 sometimes ignored the provided context. The authors note in their survey that language models 'learn syntax, semantics, and world knowledge through tasks like next-word prediction or masked language modeling, but this doesn't ensure logical reasoning skills', and suggest that incorporating few-shot in-context learning (providing solved examples in the prompt) could potentially enhance inductive reasoning capabilities.

Adjacent results sharpen the worry without settling it. Wu et al. (2023) showed that model performance degrades substantially on 'counterfactual' variants of 11 tasks that deviate from default surface statistics, indicating reliance on narrow, non-transferable procedures. Dziri et al. (2023) found that transformers on compositional tasks such as multi-digit multiplication behave like pattern matchers that effectively compile depth-one lookup tables rather than executing systematic algorithms. Wang et al. (2024) showed that implicit two-hop reasoning over parametric knowledge emerges only after grokking — delayed generalization appearing during extended training long past overfitting — and fails to generalize compositionally out of distribution. Bachmann and Nagarajan (2024) gave a theoretical account of a mechanism (teacher-forcing myopia, where teacher forcing means training the model on ground-truth previous tokens rather than on its own predicted context, so the learned features can be myopic) by which next-token training can fail to learn accurate predictors on planning-like tasks in both Transformer and Mamba (a state-space sequence architecture, an alternative to Transformers) models. Zhang et al. (2024) commissioned GSM1k, a contamination-matched twin of the GSM8k arithmetic benchmark, and measured accuracy drops of up to 8% with systematic overfitting for several model families; Pacchiardi et al. (2024) showed that simple lexical features such as individual n-grams predict answers on many LLM benchmarks, a 'Clever Hans' shortcut effect. Meanwhile the practical frontier moved to large reasoning models trained with reinforcement learning on reasoning traces (DeepSeek-R1, 2025) — an engineering judgment that pretraining alone does not finish the job — yet Shojaee et al. (2025) report that even these models collapse in accuracy beyond a complexity threshold on controllable puzzle environments, so the question of which training signal induces genuine reasoning persists. No published work gives a decisive, contamination-controlled determination of whether standard pretraining objectives can induce generalizable logical reasoning, nor a validated criterion that separates recitation from inference.

Problem Statement

As posed by Luo et al. (2023) from the LogiGLUE benchmark and its experiments: determine whether standard large-model pretraining objectives — autoregressive next-token prediction or masked-token prediction — are fundamentally sufficient to induce genuine, generalizable logical reasoning over natural language, or whether the high benchmark performance of LLMs on logical tasks is substantially explained by dataset-specific pattern learning, spurious cues, and recitation of memorized content from contaminated training corpora.

The scientific objective is a scientifically validated account of whether and when LLMs acquire genuine logical reasoning, centered on the following determination and the supporting measurement challenges on which its trustworthiness depends:

  1. Core determination (sufficiency). A contamination-controlled empirical determination of whether models trained exclusively with standard pretraining objectives — at any scale, with or without instruction fine-tuning that does not itself target logical reasoning — exhibit generalizable logical reasoning across deductive, inductive, and abductive tasks, evaluated on human-authored and freshly constructed tasks that are verifiably absent from training corpora, in contrast with their synthetic in-domain performance.

  2. Recitation-versus-inference criterion. An operational, validated criterion or methodology that distinguishes genuine inferential reasoning from recitation — detecting memorized reasoning chains, training-data contamination, and spurious-cue exploitation — applicable across the three reasoning types.

  3. Intervention characterization. A characterization of the conditions (data volume, filtering, and training or prompting protocol) under which CoT prompting, CoT knowledge distillation, multi-task instruction fine-tuning, and few-shot in-context examples reliably improve generalizable logical reasoning.

  4. Free-form evaluation methodology. A robust evaluation method for free-form logical-reasoning outputs that handles synonymy and paraphrase and detects context-ignoring behavior, so that the measurements underlying the core determination are trustworthy.

Components 2–4 are supporting measurement challenges rather than jointly mandatory deliverables: they condition how far the determination in component 1 can be trusted, and the scientific objective is the determination itself. The determination may come out either way — sufficiency established, insufficiency demonstrated (with the mechanism that explains why), or a precise boundary identified between the reasoning types, task families, or scales where pretraining-induced reasoning does and does not generalize.

References

  1. Man Luo, Shrinidhi Kumbhar, Ming Shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, Chitta Baral. 'Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models', arXiv:2310.00836 (2023), DOI 10.48550/arXiv.2310.00836, https://arxiv.org/abs/2310.00836
  2. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral. 'LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models', arXiv:2404.15522 (2024), DOI 10.48550/arXiv.2404.15522, https://arxiv.org/abs/2404.15522
  3. Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, Chitta Baral. 'Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models', arXiv:2406.17169 (2024), DOI 10.48550/arXiv.2406.17169, https://arxiv.org/abs/2406.17169
  4. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim. 'Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks', arXiv:2307.02477 (2023), DOI 10.48550/arXiv.2307.02477, https://arxiv.org/abs/2307.02477
  5. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, Yejin Choi. 'Faith and Fate: Limits of Transformers on Compositionality', arXiv:2305.18654 (2023), DOI 10.48550/arXiv.2305.18654, https://arxiv.org/abs/2305.18654
  6. Boshi Wang, Xiang Yue, Yu Su, Huan Sun. 'Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization', arXiv:2405.15071 (2024), DOI 10.48550/arXiv.2405.15071, https://arxiv.org/abs/2405.15071
  7. Gregor Bachmann, Vaishnavh Nagarajan. 'The pitfalls of next-token prediction', arXiv:2403.06963 (2024), DOI 10.48550/arXiv.2403.06963, https://arxiv.org/abs/2403.06963
  8. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, Summer Yue. 'A Careful Examination of Large Language Model Performance on Grade School Arithmetic', arXiv:2405.00332 (2024), DOI 10.48550/arXiv.2405.00332, https://arxiv.org/abs/2405.00332
  9. Lorenzo Pacchiardi, Marko Tesic, Lucy G. Cheke, José Hernández-Orallo. 'Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers', arXiv:2410.11672 (2024), DOI 10.48550/arXiv.2410.11672, https://arxiv.org/abs/2410.11672
  10. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, et al. 'DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning', arXiv:2501.12948 (2025), DOI 10.48550/arXiv.2501.12948, https://arxiv.org/abs/2501.12948
  11. Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar. 'The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity', arXiv:2506.06941 (2025), DOI 10.48550/arXiv.2506.06941, https://arxiv.org/abs/2506.06941