On tasks from the Inverse Scaling Prize, language-model performance worsens with scale (inverse scaling), yet Wei, Kim, Tay, and Le (EMNLP 2023) showed that at larger scales (up to 540B parameters, five times the training compute) six of the eleven prize tasks recover, producing U-shaped scaling. The authors proposed a speculative 'distractor task' explanation - a competing secondary task that mid-sized models solve instead of the true task - but explicitly left its causal validation open; the companion Inverse Scaling Prize report (McKenzie et al., TMLR 2023) explicitly left open when and why scaling trends reverse and how to predict the scale at which a given task's trend reverses. This problem asks for a causal, mechanistic account of why performance falls and then recovers with scale, discriminating the distractor-task account against metric-artifact, random-seed-mixture, and subtask-composition alternatives, and for a validated way to predict which inverse-scaling tasks will remain inverse and which will reverse, at approximately what scale, without full-scale evaluation.

Background

Scaling laws describe how language-model quality improves predictably as model size, training data, and compute increase, and downstream task performance usually improves along with it. 'Inverse scaling' names the surprising exception: tasks on which larger models perform worse. The Inverse Scaling Prize (McKenzie et al., TMLR 2023) ran a public contest that collected eleven tasks showing inverse scaling on models up to 280B parameters, and proposed four informal causes, one of which is a task containing an 'easy distractor task' - a superficially learnable secondary objective that competes with the harder true task the evaluator intends. McKenzie et al. also left open whether a task's trend-reversal regime can be predicted from its structure or from mechanistic analysis of model internals, without requiring evaluation at full scale. The phenomenon was documented before the prize, most prominently by TruthfulQA (Lin, Hilton, and Evans, ACL 2022), a benchmark of questions that some humans answer falsely due to misconceptions, on which the largest evaluated models were generally the least truthful.

Wei, Kim, Tay, and Le ('Inverse Scaling Can Become U-Shaped', EMNLP 2023) re-evaluated these eleven tasks on the PaLM family of dense Transformer language models up to 540B parameters. Only four tasks remained inverse scaling; six exhibited 'U-shaped scaling', where performance first decreases and then increases again with scale (the remaining task showed positive scaling), implying that an observed inverse-scaling trend cannot be extrapolated to larger models. The paper offered a mechanism hypothesis, explicitly labeled speculative: each task decomposes into a true task and a distractor task; small models can do neither (near-chance performance), mid-sized models can do the distractor task (performance drops below chance), and sufficiently large models can ignore the distractor and perform the true task (recovery, which the authors connect to emergent abilities - capabilities present only above some scale). The authors noted that simply measuring performance on the distractor task alone 'would be an imperfect ablation' and wrote: 'We leave further explanation of why U-shaped scaling occurs to future work.' A second finding was that prompting mitigates the effect - a single worked demonstration (1-shot prompting) turns all four persistently inverse tasks into U-shaped or flat scaling, and chain-of-thought prompting (CoT, asking the model to produce intermediate reasoning steps before the answer) turns several U-shaped tasks positive - but the largest models still performed near chance in absolute terms on several tasks, and the mitigations require manually crafted, task-specific demonstrations, which strict zero-shot users do not have.

Later work supplies competing, partially overlapping explanations for non-monotonic scaling, none of which settles the question for these tasks. Schaeffer, Miranda, and Koyejo (NeurIPS 2023) argued that sharp 'emergent' transitions are often artifacts of nonlinear or discontinuous evaluation metrics, with smooth metrics giving smooth curves. Zhao, Qin, Alvarez-Melis, Kakade, and Saphra (2025) showed that scaling curves can flip between smooth and emergent across random training seeds, and that U-shaped mean curves can arise purely from changing mixture weights between failure and success modes even when successful runs improve monotonically. Zhang et al. (Findings of ACL 2023, the NeQA negation dataset) demonstrated a compositional account: when a task combines a linearly scaling subtask with a sigmoid-shaped emergent subtask, the composed trend is inverse, U-shaped, or positive depending on where the emergent transition falls within the evaluated model sizes. Michaelov and Bergen (Findings of EMNLP 2023) found inverse scaling also over the course of pretraining, so 'scale' is multidimensional (parameters, compute, training steps). Douglas, Draguns, and Gavenčiak (2024) gave the first formal treatment of the distractor-task idea, modeling a language model as a product of experts - an output distribution composed from component generating processes - that over-weights a distractor process, and used this to build prior-aware decoding (PAD), a contrastive inference method that counteracts the over-weighted distractor process at decoding time - a formalization and a demonstration-free mitigation, but evaluated on smaller models and not a causal test of the U-shape's origin. On the prediction side, a meta-analysis by Lourie, Hu, and Cho (Findings of EMNLP 2025) found that downstream scaling extrapolation succeeds only about 39% of the time, and Wu and Lo (ICLR 2025), slicing benchmarks by question difficulty, showed that U-shaped trends on hard questions can offset inverted-U trends on easy ones and used this to predict emergence thresholds - without, however, predicting which inverse-scaling tasks recover. The phenomenon also persists in new regimes: inverse scaling reappears with longer test-time reasoning (inference-time computation spent on reasoning steps; Gema et al., 2025), in vision-language models (Zhang et al., NegVQA, 2025; Bae et al., 2025), in robustness to distractor instructions in instruction-following (Su et al., 2026), and in tool discovery by agents (Klein-Seetharaman, Wang, and Xu, 2026), where a prompt that motivates tool use makes large models recover U-shaped recognition accuracy - convergent, but again not a controlled test on the original prize tasks.

Problem Statement

For tasks that exhibit inverse scaling in language models - in particular the Inverse Scaling Prize tasks, of which six of eleven become U-shaped (performance falling, then recovering) at scales up to 540B parameters while four remain inverse (the eleventh task already scales positively) - establish the causal mechanism by which performance falls and then recovers with scale, and characterize the predictive boundary of the phenomenon.

Specifically: (1) causally establish, refine, or refute the distractor-task hypothesis - that each such task contains a competing secondary task which mid-sized models solve at the expense of the true task and sufficiently large models suppress - by measurements or interventions that genuinely isolate the mechanism, while discriminating this account against the competing explanations already in the literature (metric nonlinearity, random-seed mixture effects, and composition of subtask scaling trends). The source authors noted that a naive distractor-only evaluation would be an imperfect ablation because the distractor and true tasks can have joint as well as competing effects, so the design must address this confound. (2) Characterize which inverse-scaling tasks remain inverse at all scales versus recover with scale, and approximately at what model scale or training compute the reversal occurs, with the characterization validated out-of-sample - that is, predicting scaling-trend class and reversal location for tasks or model families not used to derive it, without requiring evaluation at full scale.

These two components form one scientific objective: turning the empirical observation that inverse scaling can become U-shaped into an explained and predictable regularity. A complete resolution requires both (1) and (2) together; satisfying either component alone - a mechanism result without the predictive characterization, or a predictor without the causal account - does not resolve the task. Progress on the source paper's further flagged directions - mitigation without task-specific demonstrations under strict zero-shot setups, and improving the near-chance absolute accuracy of recovered tasks - is related but separable and is not required for this problem.

References

  1. Jason Wei, Najoung Kim, Yi Tay, Quoc V. Le. Inverse Scaling Can Become U-Shaped. Proceedings of EMNLP 2023, pp. 15580-15591. DOI: 10.18653/v1/2023.emnlp-main.963 (arXiv:2211.02011). https://aclanthology.org/2023.emnlp-main.963/ ; https://arxiv.org/abs/2211.02011
  2. Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman, Ethan Perez. Inverse Scaling: When Bigger Isn't Better. Transactions on Machine Learning Research, 2023. arXiv:2306.09479. https://arxiv.org/abs/2306.09479 ; https://openreview.net/forum?id=DwgRm72GQF
  3. Stephanie Lin, Jacob Hilton, Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022. arXiv:2109.07958. https://arxiv.org/abs/2109.07958
  4. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research, 2022. arXiv:2206.07682. https://arxiv.org/abs/2206.07682
  5. Rylan Schaeffer, Brando Miranda, Sanmi Koyejo. Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023. arXiv:2304.15004. https://arxiv.org/abs/2304.15004
  6. Raymond Douglas, Andis Draguns, Tomáš Gavenčiak. Mitigating the Influence of Distractor Tasks in LMs with Prior-Aware Decoding. arXiv:2401.17692, 2024. https://arxiv.org/abs/2401.17692
  7. Tung-Yu Wu, Pei-Yu Lo. U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models. ICLR 2025. arXiv:2410.01692. https://arxiv.org/abs/2410.01692
  8. Rosie Zhao, Tian Qin, David Alvarez-Melis, Sham Kakade, Naomi Saphra. Random Scaling of Emergent Capabilities. arXiv:2502.17356, 2025. https://arxiv.org/abs/2502.17356
  9. Yuhui Zhang, Michihiro Yasunaga, Zhengping Zhou, Jeff Z. HaoChen, James Zou, Percy Liang, Serena Yeung-Levy. Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models. Findings of ACL 2023. arXiv:2305.17311. https://arxiv.org/abs/2305.17311
  10. James A. Michaelov, Benjamin K. Bergen. Emergent inabilities? Inverse scaling over the course of pretraining. Findings of EMNLP 2023. arXiv:2305.14681. https://arxiv.org/abs/2305.14681
  11. Nicholas Lourie, Michael Y. Hu, Kyunghyun Cho. Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check. Findings of EMNLP 2025. arXiv:2507.00885. https://arxiv.org/abs/2507.00885
  12. Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, Bernhard Schölkopf. Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals. arXiv:2402.11655, 2024. https://arxiv.org/abs/2402.11655
  13. Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, Ethan Perez. Inverse Scaling in Test-Time Compute. arXiv:2507.14417, 2025. https://arxiv.org/abs/2507.14417
  14. Yuhui Zhang, Yuchang Su, Yiming Liu, Serena Yeung-Levy. NegVQA: Can Vision Language Models Understand Negation? Findings of ACL 2025. arXiv:2505.22946. https://arxiv.org/abs/2505.22946
  15. Jiyun Bae, Hyunjong Ok, Sangwoo Mo, Jaeho Lee. Understanding the Effects of Distractors on Reasoning Vision-Language Models. arXiv:2511.21397, 2025. https://arxiv.org/abs/2511.21397
  16. Zeli Su, Zhankai Xu, Tianlei Chen, Longfei Zheng, Xiaolu Zhang, Jun Zhou, Wentao Zhang. The Curse of Helpfulness: Inverse Scaling Law in Robustness to Distractor Instructions via DistractionIF. arXiv:2605.29491, 2026. https://arxiv.org/abs/2605.29491
  17. Roshan Klein-Seetharaman, Daniel Wang, Andrew Xu. Lomekwi: Resource-Bounded Tool Discovery in LLM Agents. Conference on Learning Theory Workshop on Learning in an Agentic World, 2026. arXiv:2607.16961. https://arxiv.org/abs/2607.16961