A 2025 study (Chen, Plaat, and Van Stein, published at AAAI 2026) combined sparse autoencoders (SAEs) with activation patching to give the first feature-level causal study of chain-of-thought (CoT) faithfulness: in Pythia-2.8B solving GSM8K math problems, SAE features extracted under CoT prompting are sparser, more interpretable, and causally transferable — patching a small set of CoT-conditioned features into a no-CoT run raises the log-probability of the correct answer — while none of these effects appear in Pythia-70M, indicating a scale threshold. The evidence, however, was restricted to the residual stream at layer 2 at a single final-token position, one model family up to 2.8B parameters, LLM-generated feature labels validated only by correlation, and an SAE decomposition of unknown completeness. The open problem is to turn this single-point, single-scale description into a causally validated, general account: determine whether the CoT-induced feature phenotype holds across token positions and causal paths through the reasoning trajectory, and across larger or different model families, and separate genuine reasoning-structure changes from lexical and stylistic correlates of CoT text. The source reports the scale threshold descriptively and offers no mechanism for it, but it does not pose a mechanism why-question among its explicit open questions; a mechanistic explanation of why CoT induces structured sparsity therefore remains non-prescriptive background here. Follow-up work through August 2026 has advanced individual strands (SAE analysis at 7B, sequential token-level patching, step-level SAEs, and a falsification-based critique of contrastively selected reasoning features), but no study delivers the general, validated account, which therefore remains open.

Background

Large language models (LLMs) solve multi-step problems far more accurately when prompted to write out intermediate reasoning steps before the final answer — chain-of-thought (CoT) prompting (Wei et al., 2022, arXiv:2201.11903). A long-standing concern is faithfulness: whether the written steps reflect the computation the model actually performs or are plausible post-hoc rationalization. Behavioral faithfulness tests can be gamed, so the question is naturally mechanistic. Mechanistic interpretability approaches it by analyzing internal activations. Two obstacles make raw activations hard to read: polysemanticity, where a single neuron fires in many unrelated contexts, and superposition, where a model represents more concepts than it has neurons by packing them into overlapping directions. Sparse autoencoders (SAEs) address both: an SAE is trained to reconstruct a model's activation vector as a sparse code over a much larger dictionary of features, and the learned features are often individually interpretable (Cunningham et al., 2023, arXiv:2309.08600). Activation patching is the standard causal tool: run the model twice under two conditions, replace an internal activation in one run with its value from the other, and measure the effect on the output, which turns observational correlations into causal evidence. Chen, Plaat, and Van Stein (2025; published in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, 2026, DOI 10.1609/aaai.v40i36.40281) combined the two tools for CoT. They trained separate SAEs on the activations of Pythia-70M and Pythia-2.8B (Pythia is an open family of language models; 70M and 2.8B denote parameter counts) solving GSM8K problems (a benchmark of grade-school math word problems) under CoT and plain no-CoT prompting. All analysis targeted the residual stream — the running sum of a transformer layer's outputs that is passed forward — at layer 2, at the final token position. In the 2.8B model they found that CoT-conditioned features are (i) sparser, with fewer features strongly active; (ii) more interpretable, as scored by a module that labels features using an external LLM; and (iii) causally transferable, in that patching a small — even randomly selected — set of CoT-conditioned features into a no-CoT forward pass significantly raises the log-probability of the correct answer. None of this held in the 70M model, which the authors read as a scale threshold. Patch-curve analyses (gradually patching the top-K features ranked by activation difference, where K is the number patched) and Random-K controls showed that the causal information is not concentrated in the few most-differentiated features but broadly distributed. The authors themselves list what this leaves open. The causal evidence is a single static snapshot at one token position, while CoT reasoning unfolds over many generated tokens; static SAE patching cannot trace effects through the reasoning trajectory. The interpretability scores come from LLM-generated labels validated only by correlation, not by causal tests. The SAE dictionary may miss distributed or entangled representations, so features that look CoT-specific may not capture the reasoning-relevant part of the representation space. And experiments cover one model family up to 2.8B parameters and one task. Separately, the paper offers no mechanism for why CoT induces structured sparsity or why the effect appears only in sufficiently large models, but it does not pose that mechanism question as an explicit open problem.

Problem Statement

The problem is to turn the source study's single-point, single-scale, correlation-labeled description into a causally validated, general account of CoT-induced internal restructuring in language models. Concretely, an answer must establish, jointly: (1) Generalization of the phenotype — measure feature sparsity, feature interpretability, and causal transfer of CoT-conditioned SAE features across token positions and causal paths through the generated reasoning trajectory, and across model families and scales beyond Pythia-2.8B, including models of roughly 7B parameters and larger, and at least one family unrelated to Pythia, and state for which regimes the CoT-induced phenotype holds, weakens, or disappears. (2) Measurement validity — establish that the features and labels used in (1) track reasoning-relevant structure rather than lexical, stylistic, or formatting correlates of CoT text: quantify what the SAE decomposition fails to recover and how that omission affects the causal conclusions, and causally validate (or falsify) the natural-language labels assigned to features, requiring that feature activation be decoupled from reasoning behavior. These components form one scientific objective: deciding, with causal evidence that survives validity controls, whether CoT prompting genuinely reorganizes a language model's internal computation into sparser, more interpretable, causally efficacious features. A mechanistic explanation of why CoT induces structured sparsity, or of the reported scale threshold, is a natural further question that the source does not pose as an open question and that is not part of the completion requirement for this problem. Results in (1) that rest on unvalidated labels or incomplete decompositions do not settle it; improved feature-selection strategies for interventions and scaling feature-level causal discovery into automated circuit-finding pipelines are enabling methodology for (1)–(2), not separate objectives.

References

  1. Xi Chen, Aske Plaat, and Niki van Stein (2026), 'How Does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding', Proceedings of the AAAI Conference on Artificial Intelligence 40, pp. 30297–30305, DOI: 10.1609/aaai.v40i36.40281, https://doi.org/10.1609/aaai.v40i36.40281 (preprint arXiv:2507.22928, https://arxiv.org/abs/2507.22928)
  2. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou (2022), 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models', arXiv:2201.11903, https://arxiv.org/abs/2201.11903
  3. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey (2023), 'Sparse Autoencoders Find Highly Interpretable Features in Language Models', arXiv:2309.08600, https://arxiv.org/abs/2309.08600
  4. George Ma, Zhongyuan Liang, Irene Y. Chen, and Somayeh Sojoudi (2026), 'Do Sparse Autoencoders Identify Reasoning Features in Language Models?', arXiv:2601.05679, https://arxiv.org/abs/2601.05679
  5. Bo Cheng, Qiaolin Lu, Yi Chang, and Yuan Wu (2026), 'Thinking vs. NoThinking: Towards Interpreting Reasoning Mechanisms of Large Language Models via Sparse Autoencoders', arXiv:2608.08168, https://arxiv.org/abs/2608.08168
  6. Murat Dura, Serkan Öztürk, and Selma Tekir (2026), 'Mechanistic Interpretability of Chain-of-Thought Reasoning via Sequential Activation Patching', arXiv:2608.22332, https://arxiv.org/abs/2608.22332
  7. Xuan Yang, Jiayu Liu, Yuhang Lai, Hao Xu, Zhenya Huang, and Ning Miao (2026), 'Step-Level Sparse Autoencoder for Reasoning Process Interpretation', arXiv:2603.03031, https://arxiv.org/abs/2603.03031
  8. Houman Mehrafarin, Amit Parekh, and Ioannis Konstas (2026), 'When Chain-of-Thought Fails, the Solution Hides in the Hidden States', arXiv:2604.23351, https://arxiv.org/abs/2604.23351
  9. Zhenghao He, Guangzhi Xiong, Bohan Liu, Sanchit Sinha, and Aidong Zhang (2026), 'Reasoning Beyond Chain-of-Thought: A Latent Computational Mode in Large Language Models', arXiv:2601.08058, https://arxiv.org/abs/2601.08058
  10. Kyle Cox, Darius Kianersi, and Adrià Garriga-Alonso (2026), 'Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers', arXiv:2603.01437, https://arxiv.org/abs/2603.01437
  11. Mosh Levy, Zohar Elyoseph, Shauli Ravfogel, and Yoav Goldberg (2025), 'State over Tokens: Characterizing the Role of Reasoning Tokens', arXiv:2512.12777, https://arxiv.org/abs/2512.12777