Modern reasoning models are trained almost exclusively on questions whose final answers can be mechanically checked (mathematics with known answers, code with unit tests). The source paper (Xiang et al., 2025, Meta Chain-of-Thought) poses the still-open question of how to reward and reliably evaluate the full reasoning chain itself — not only the final answer — in open-ended domains such as natural-language proofs and long scientific derivations that lack deterministic verifiers, while ensuring that emitted chains of thought are faithful (reflecting the computation that actually determines the answer) rather than superficially plausible. It further asks for the missing quantitative scaling laws for learned process verifiers: how verifier quality, data requirements, and downstream search/reinforcement-learning performance scale, and how much of the gap to oracle Monte-Carlo verification can be closed at reasonable cost. Post-2025 evidence (Anthropic's and OpenAI's faithfulness studies, ProcessBench/PRMBench, full-solution USAMO grading) confirms both halves remain open.

Background

A chain of thought (CoT) is the sequence of intermediate reasoning steps a large language model (LLM) emits before its final answer. 'System 2' or deliberate reasoning refers to architectures that make this search-like computation explicit rather than implicit in the network's activations; the source paper's Meta Chain-of-Thought (Meta-CoT) framework extends CoT to also model the underlying search process (exploration, backtracking, verification) that produces a given CoT. Current reasoning models are typically trained with reinforcement learning from verifiable rewards (RLVR), where a reward checks only the final answer — possible only in domains like competition math with known answers or code with executable unit tests. An outcome reward model (ORM) scores a full solution by its final result; a process reward model (PRM) instead scores each intermediate step, promising better credit assignment for multi-step reasoning. Human per-step labels (Lightman et al., 2023, the PRM800K dataset) improved mathematical reasoning over outcome supervision, but are expensive and limited to math; Setlur et al. (2024) showed automatically constructed 'progress'-based process rewards can be trained from outcome labels alone and observed roughly log-linear scaling of verifier performance with training data; Zhang et al. (2024) and Mahan et al. (2024) proposed generative verifiers that evaluate solutions by producing verification chains of thought, gaining accuracy from verifier-side inference-time sampling and voting. A complementary line measures best-of-N selection: a generator produces N candidate solutions and a verifier selects one; the oracle pass@N is the accuracy achievable with a perfect selector that knows the ground truth, and the 'verifier gap' is the shortfall of learned verifiers against this oracle — a gap also characterized by the generation-verification asymmetry, the empirical fact that checking a solution is often easier than producing one. Faithfulness of a CoT means the stated steps are the actual cause of the answer rather than a post-hoc rationalization; it is probed by truncating or perturbing the CoT and measuring the effect on the answer (Lanham et al., 2023), or by inserting hints into the prompt and measuring whether the CoT verbalizes their use (Turpin et al., 2023; Chen et al., 2025). Against this backdrop, the source paper consolidated the pipeline for verifiable-question reasoning and explicitly flagged what remains unanswered: how to reward full CoTs and open-ended problems such as proofs, how to ensure the resulting CoTs are faithful rather than merely plausible, and what scaling laws govern learned verifiers — questions that matter because in scientific applications the reasoning chain can be more important than the final answer, and because unfaithful chains undermine CoT-based monitoring and oversight of reasoning models.

Problem Statement

How can full chain-of-thought reasoning chains — not only final answers — be rewarded and reliably evaluated in domains that lack deterministic verifiers, such as natural-language proofs and long scientific derivations? Concretely, following §8.3.1–8.3.2 of Xiang et al. (2025), the question asks for (1) a verification or reward mechanism (e.g., a generative verifier supplied with reference solutions, proofs, or tools) whose evaluation of a complete reasoning chain reliably discriminates valid from invalid or incomplete reasoning in open-ended domains; (2) evidence that training against such process-level rewards elicits CoTs that are faithful — whose stated steps actually determine the answer — rather than superficially plausible post-hoc rationalizations; and (3) a quantitative characterization of the scaling laws of learned verifiers: how verifier quality and downstream search/reinforcement-learning performance scale with training data, verifier inference-time compute, and model capacity, and how much of the gap between best-of-N verifier selection and oracle pass@N performance can be closed at reasonable data and compute cost. All three deliverables are jointly required for a resolution — the mechanism with the evaluation establishing its discrimination, the evidence that process-level training elicits faithful CoTs, and the scaling characterization with the data and compute cost it assumes — and it is their conjunction that must make process-level verification of reasoning scalable and trustworthy enough to extend verifiable-reward training and chain-level oversight beyond domains with mechanically checkable final answers.

References

  1. Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, Louis Castricato, Jan-Philipp Fränken, Nick Haber, Chelsea Finn (2025), 'Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought', arXiv:2501.04682, DOI: 10.48550/arXiv.2501.04682, https://arxiv.org/abs/2501.04682 (source paper)
  2. Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman (2023), 'Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting', arXiv:2305.04388, DOI: 10.48550/arXiv.2305.04388, https://arxiv.org/abs/2305.04388
  3. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, Ethan Perez (2023), 'Measuring Faithfulness in Chain-of-Thought Reasoning', arXiv:2307.13702, DOI: 10.48550/arXiv.2307.13702, https://arxiv.org/abs/2307.13702
  4. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe (2023), 'Let's Verify Step by Step', arXiv:2305.20050, DOI: 10.48550/arXiv.2305.20050, https://arxiv.org/abs/2305.20050
  5. Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar (2024), 'Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters', arXiv:2408.03314, DOI: 10.48550/arXiv.2408.03314, https://arxiv.org/abs/2408.03314
  6. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, Aviral Kumar (2024), 'Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning', arXiv:2410.08146, DOI: 10.48550/arXiv.2410.08146, https://arxiv.org/abs/2410.08146
  7. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal (2024), 'Generative Verifiers: Reward Modeling as Next-Token Prediction', arXiv:2408.15240, DOI: 10.48550/arXiv.2408.15240, https://arxiv.org/abs/2408.15240
  8. Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, Alon Albalak (2024), 'Generative Reward Models', arXiv:2410.12832, DOI: 10.48550/arXiv.2410.12832, https://arxiv.org/abs/2410.12832
  9. Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng (2024), 'Free Process Rewards without Process Labels', arXiv:2412.01981, DOI: 10.48550/arXiv.2412.01981, https://arxiv.org/abs/2412.01981
  10. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin (2024), 'ProcessBench: Identifying Process Errors in Mathematical Reasoning', arXiv:2412.06559, DOI: 10.48550/arXiv.2412.06559, https://arxiv.org/abs/2412.06559
  11. Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, Yu Cheng (2025), 'PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models', arXiv:2501.03124, DOI: 10.48550/arXiv.2501.03124, https://arxiv.org/abs/2501.03124
  12. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, Ning Ding (2025), 'Process Reinforcement through Implicit Rewards', arXiv:2502.01456, DOI: 10.48550/arXiv.2502.01456, https://arxiv.org/abs/2502.01456
  13. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciek Zaremba, Jakub Pachocki, David Farhi (2025), 'Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation', arXiv:2503.11926, DOI: 10.48550/arXiv.2503.11926, https://arxiv.org/abs/2503.11926
  14. Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, Martin Vechev (2025), 'Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad', arXiv:2503.21934, DOI: 10.48550/arXiv.2503.21934, https://arxiv.org/abs/2503.21934
  15. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, Ethan Perez (2025), 'Reasoning Models Don't Always Say What They Think', arXiv:2505.05410, DOI: 10.48550/arXiv.2505.05410, https://arxiv.org/abs/2505.05410