Du et al. (2024) proposed studying emergent abilities of large language models through pre-training loss rather than model size or compute: with a fixed corpus, tokenizer, and architecture, models with equal loss show equal downstream performance regardless of size, and abilities such as MMLU and GSM8K remain near random guessing until loss falls below a threshold that is shared across several unrelated tasks, even under continuous metrics. The authors themselves flag much of what remains unresolved: whether these performance-versus-loss relations and thresholds survive fundamentally different architectures (routed/mixture-of-experts Transformers, state-space models) and training algorithms beyond AdamW; how to make loss comparable across tokenizers and corpora; and whether new tipping points will appear at larger scales. Their noted caveat that instruction tuning can improve zero-shot performance further raises the question whether abilities can be acquired without crossing the same thresholds, and the reported but unexplained coincidence of task thresholds motivates asking what internal dynamics cause sudden capability onset and why unrelated tasks share one threshold. This problem asks for the establishment — or refutation — of the loss account as a general law of language-model training, and for a causal explanation of its thresholds. Prior work shows partial progress on each thread (emergence prediction via finetuning, cross-dataset loss mappings, phase-transition analyses) but no resolution of any of them; the problem remains open.

Background

Emergent abilities of large language models were introduced by Wei et al. (2022) as abilities that are absent in smaller models but appear in larger ones, with sharp jumps rather than gradual improvement. Schaeffer, Miranda and Koyejo (2023) challenged this picture, arguing that the apparent jumps are artifacts of nonlinear or discontinuous evaluation metrics (for example, exact-match accuracy on multiple-choice questions, which scores an answer 1 or 0) rather than genuine discontinuities in model capability. Du et al. (2024) entered this debate by proposing to scale, and study, language models along pre-training loss instead of model size or compute. Pre-training loss is the average cross-entropy the model assigns to the next token over the training corpus; its exponentiated form is the perplexity. Because loss depends on the tokenizer (the rule that splits text into discrete tokens) and on the corpus distribution, loss values are only directly comparable within one fixed tokenizer/corpus setting. The authors pre-trained more than 30 Transformer language models of varied model and data sizes from scratch with a fixed corpus, tokenizer, and architecture, evaluated them on 12 tasks covering multiple-choice knowledge benchmarks (MMLU, English undergraduate-level questions across subjects; C-Eval, its Chinese counterpart) and grade-school math word problems (GSM8K), and made three observations: (1) performance-versus-loss curves of differently sized models collapse onto a single curve, so loss, not size, predicts downstream performance; (2) on tasks such as MMLU and GSM8K, performance stays at the random-guess level until loss falls below a specific threshold, after which it rises sharply — and this persists under continuous metrics (the Brier score, which measures the squared error of predicted answer probabilities, and the probability assigned to the correct choice), countering the pure-metric-artifact explanation; (3) the loss thresholds for these tasks coincide at roughly the same value, even though the tasks differ greatly in content. Based on this, emergent abilities are redefined: an ability is emergent if it is absent in models with higher pre-training loss and present in models with lower pre-training loss, which makes the phenomenon in-principle tied to training progress rather than parameter count. The evidence, however, comes from one Transformer family trained with the AdamW optimizer (a standard adaptive gradient-descent method) on one corpus and tokenizer, plus open models (LLaMA, Pythia) with only slightly different architectures. Whether the loss account is a general law of language-model training or a property of this specific setting is untested: routed Transformers (mixture-of-experts models, which activate only a subset of parameter specialists per token) and state-space models (sequence architectures such as Mamba that replace attention with recurrent state updates) have never been examined for loss-threshold emergence, losses across different tokenizers and corpora cannot currently be compared under any canonical normalization, and the analysis is correlational — it does not say what changes inside the model at the threshold, nor why tasks as different as English multiple-choice knowledge and arithmetic word problems would share one threshold. Instruction tuning (post-training a model on instruction–response data) is also known to lift zero-shot benchmark scores, raising the question of whether crossing the pre-training threshold is the only route to these abilities.

Problem Statement

Determine whether Du et al.'s pre-training-loss account of emergent abilities is a general law of language-model training or a property of one architecture/tokenizer/corpus family, and explain its thresholds. Concretely, with the performance-versus-loss curve collapse (equal loss implies equal downstream performance, independent of model and data size) and the task thresholds (near-random performance until loss falls below a task-specific, partially shared threshold, persisting under continuous metrics) as the claims under test, answer: (1) do curve collapse and emergent loss thresholds hold for fundamentally different model architectures — routed/mixture-of-experts Transformers and non-Transformer architectures such as state-space models — and for training algorithms and optimizers other than AdamW; (2) can pre-training loss be normalized so that loss values, and hence thresholds, are comparable across tokenizers and pre-training corpora; (3) can the existence and location of emergent thresholds be predicted for tasks and scales beyond those already exhibiting them — in particular, whether new tipping points appear at substantially larger scales — and can the abilities in question instead be acquired by other procedures such as instruction tuning without crossing the same pre-training thresholds; and (4) what internal model dynamics or representation changes cause the sudden capability onset at a threshold, and in particular why multiple unrelated tasks such as MMLU and GSM8K share approximately the same loss threshold. Each of the four questions is independently attackable and independently refutable, including the compound parts within them (new architectures versus new optimizers; new-scale threshold prediction versus alternative acquisition routes): a positive or refuting result on any one is genuine partial progress on the stated problem. A complete resolution, however, requires accounting for all four together, since they jointly test the single claim that the pre-training-loss account is a general law of language-model training — converting the correlational, single-family observation into a general, comparable, and causally explained account of emergent abilities.

References

  1. Zhengxiao Du, Aohan Zeng, Yuxiao Dong, Jie Tang. Understanding Emergent Abilities of Language Models from the Loss Perspective. NeurIPS 2024. DOI: 10.48550/arXiv.2403.15796, https://arxiv.org/abs/2403.15796
  2. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus. Emergent Abilities of Large Language Models. TMLR 2022. DOI: 10.48550/arXiv.2206.07682, https://arxiv.org/abs/2206.07682
  3. Rylan Schaeffer, Brando Miranda, Sanmi Koyejo. Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023. DOI: 10.48550/arXiv.2304.15004, https://arxiv.org/abs/2304.15004
  4. Leonardo Berti, Flavio Giorgi, Gjergji Kasneci. Emergent Abilities in Large Language Models: A Survey. arXiv preprint, 2025. DOI: 10.48550/arXiv.2503.05788, https://arxiv.org/abs/2503.05788
  5. Charlie Snell, Eric Wallace, Dan Klein, Sergey Levine. Predicting Emergent Capabilities by Finetuning. 2024. DOI: 10.48550/arXiv.2411.16035, https://arxiv.org/abs/2411.16035
  6. Bo-Wen Zhang, Yan Yan, Boxiang Yang, Yifei Xue, Guang Liu. Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need. 2024. DOI: 10.48550/arXiv.2412.07111, https://arxiv.org/abs/2412.07111
  7. James A. Michaelov, Roger P. Levy, Benjamin K. Bergen. Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale. 2025. DOI: 10.48550/arXiv.2510.24963, https://arxiv.org/abs/2510.24963
  8. David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, Sham Kakade. Loss-to-Loss Prediction: Scaling Laws for All Datasets. 2024. DOI: 10.48550/arXiv.2411.12925, https://arxiv.org/abs/2411.12925
  9. Nicholas Lourie, Michael Y. Hu, Kyunghyun Cho. Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check. 2025. DOI: 10.48550/arXiv.2507.00885, https://arxiv.org/abs/2507.00885
  10. Yuko Nakagi, Keigo Tada, Sota Yoshino, Shinji Nishimoto, Yu Takagi. Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspective. 2025. DOI: 10.48550/arXiv.2502.20779, https://arxiv.org/abs/2502.20779
  11. Ihor Kendiukhov. A Review of Developmental Interpretability in Large Language Models. 2025. DOI: 10.48550/arXiv.2508.15841, https://arxiv.org/abs/2508.15841
  12. Ahmed Elhady, Eneko Agirre, Mikel Artetxe. Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation. 2025. DOI: 10.48550/arXiv.2506.00288, https://arxiv.org/abs/2506.00288
  13. Jing Huang, Daniel Wurgaft, Rachit Bansal, Laura Ruis, Naomi Saphra, David Alvarez-Melis, Andrew Kyle Lampinen, Christopher Potts, Ekdeep Singh Lubana. Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention. 2026. DOI: 10.48550/arXiv.2605.29548, https://arxiv.org/abs/2605.29548
  14. Jakub Krajewski, Amitis Shidani, Dan Busbridge, Sam Wiseman, Jason Ramapuram. Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training. 2025. DOI: 10.48550/arXiv.2512.08894, https://arxiv.org/abs/2512.08894