Large language models sometimes show emergent abilities: performance on a task stays near chance over a hundredfold increase in training compute and then jumps sharply at an unpredictable threshold. Wu and Lo (ICLR 2025) explained this pattern on six multiple-choice benchmarks by splitting questions into difficulty groups: hard groups show U-shaped scaling (performance first declines, then improves) while easy groups show inverted-U scaling followed by steady improvement, the two trends offsetting each other until the easy-group trend reverts to standard scaling. On this basis they built Slice-and-Sandwich, a pipeline that uses only pre-threshold data to forecast the emergence threshold and post-threshold performance. The method, however, is confined to multiple-choice tasks, because its continuous performance measure (the Target-Conditioned Brier Score) is defined from predicted probabilities over answer choices. Extending the framework to string-match, exact-match, and generative tasks requires a continuous metric that both stratifies example difficulty from pre-threshold data and correlates strongly enough with the traditional metric (e.g., exact match) that a projected value converts into an accurate prediction of it. Wu and Lo's preliminary analysis shows token edit distance fails both requirements and a modified cosine similarity satisfies difficulty stratification but correlates too poorly with exact match to be useful. The open problem is to identify or design such metric(s) and thereby extend pre-threshold emergence forecasting to non-multiple-choice task families, together with principled functional forms for the per-group fits and guidelines for the pre-threshold training-data cutoff.

Background

Emergent abilities were catalogued by Wei et al. (2022): on certain downstream tasks, language-model performance stagnates near random even as training compute grows by orders of magnitude, then improves sharply beyond a critical scale, in a way not predicted by extrapolating scaling laws from smaller models. Schaeffer, Miranda and Koyejo (2023) countered that many claimed emergent abilities are a 'mirage' created by discontinuous evaluation metrics: they arise mainly on string-match and multiple-choice tasks, where the traditional measures (exact match, accuracy) jump from wrong to right, and they argued that continuous metrics — for multiple-choice tasks the Brier Score (a strictly proper scoring rule averaging squared error between the model's predicted class probabilities and the true class indicator), and for string-match tasks token edit distance (TED, the minimal number of token insertions, deletions and substitutions turning the model output into the reference answer) — reveal smooth, predictable improvement with scale.

Wu and Lo (2024, ICLR 2025) sharpened this measurement debate into a forecasting question. Grouping questions of a benchmark by difficulty level (measured with small models, i.e., pre-threshold), they observed two scaling regimes on six multiple-choice tasks with emergent abilities: hard question groups follow U-shaped scaling, where performance first declines with log compute before improving, and easy groups follow inverted-U scaling with steady improvement (a deep-double-descent-like shape), where performance first improves, then dips, then climbs. The two patterns initially offset each other, producing stagnant overall accuracy; overall performance soars when the easy-group trend reverts to standard scaling, giving emergence. They introduced the Target-Conditioned (TC) Brier Score, a variant of the Brier Score that scores only the probability assigned to the target answer conditioned on the available answer choices, so that higher means better, and used it as the continuous, difficulty-discriminating metric. Their Slice-and-Sandwich pipeline fits low-order polynomials (odd degree for the easy group, even for the hard group, chosen to match the qualitative shape) to pre-threshold data of each difficulty group, forecasts the compute scale at which group trends revert, and projects performance beyond the emergence threshold.

The framework is structurally tied to multiple-choice tasks: the TC Brier Score needs predicted probabilities over answer choices, which string-match and generative tasks do not have. The authors' preliminary analysis of a string-match task (word unscramble from BIG-bench, a large crowdsourced benchmark suite of diverse language tasks, evaluated by exact match) shows the difficulty: TED, the continuous metric suggested by the mirage line of work, does not separate easy from hard question groups pre-threshold and correlates poorly with exact match, while their own candidate, a modified cosine similarity (embedding the model output and the reference answer with a text encoder and thresholding subset containment), does stratify difficulty but again correlates too poorly with exact match, so that even a precisely forecast continuous value would convert into exact-match accuracy too noisily to be useful. Adjacent forecasting routes bypass this metric question rather than answer it: PASSUNTIL (Hu et al., 2023) defines an infinite-resolution continuous evaluation for generative tasks but forecasts the continuous measure itself rather than converting to the traditional metric; Du et al. (2024) predict emergence from pre-training loss on multiple-choice-style benchmarks; and Snell et al. (2024) predict capabilities of large models by finetuning smaller ones, without a difficulty-stratified metric. The remaining question is thus which measurement scale can carry difficulty-stratified, pre-threshold forecasting of traditional-metric emergence on non-multiple-choice tasks — exactly the question of which measurement scales are valid for capability claims in this setting.

Problem Statement

Identify or design a continuous performance metric for string-match/exact-match tasks (and, where possible, generative and structured-output tasks more broadly; this wider scope is an optional generalization beyond the string-match core) that satisfies the two criteria that Wu and Lo (2024) establish for extending difficulty-stratified emergence forecasting beyond multiple-choice tasks: (i) using only pre-threshold evaluation data, the metric discriminates easy from hard examples well enough to form difficulty groups whose scaling trends are informative; and (ii) the metric is sufficiently correlated with the traditional metric of interest (exact match) that a forecast of the continuous metric on unseen, larger models converts into an accurate prediction of the traditional metric, assessed by fitting on strictly pre-threshold data, projecting into the post-threshold region, and comparing the projected traditional-metric prediction against the observed values. Use this metric to extend the Slice-and-Sandwich pipeline (or an equivalent difficulty-stratified procedure) to at least one non-multiple-choice task family with an emergent ability under its traditional metric, forecasting the emergence threshold and post-threshold traditional-metric performance from strictly pre-threshold data, where the emergence threshold is the log compute at which the model accuracy exhibits a sharp improvement, and the pre-threshold training set is bounded by a maximum model log compute whose principled selection is among the open components identified below. The forecasting objective comprises four components: (1) the metric satisfying criteria (i) and (ii); (2) the extension of the forecasting procedure to at least one non-multiple-choice task family, forecasting the emergence threshold and post-threshold traditional-metric performance from strictly pre-threshold data; (3) a principled basis for the per-group functional form, replacing the current heuristic polynomial-degree choice; and (4) a principled basis for the maximum model compute included in the pre-threshold training set. These components serve one objective: a validated procedure that predicts, before it happens, the onset and magnitude of emergent abilities on tasks whose traditional metrics are discontinuous.

References

  1. Tung-Yu Wu and Pei-Yu Lo. U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models. arXiv:2410.01692 (accepted to ICLR 2025), 2024. doi:10.48550/arXiv.2410.01692, https://arxiv.org/abs/2410.01692
  2. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are Emergent Abilities of Large Language Models a Mirage? arXiv:2304.15004, 2023. doi:10.48550/arXiv.2304.15004, https://arxiv.org/abs/2304.15004
  3. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models. arXiv:2206.07682 (Transactions on Machine Learning Research, 2022), 2022. doi:10.48550/arXiv.2206.07682, https://arxiv.org/abs/2206.07682
  4. Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting Emergent Abilities with Infinite Resolution Evaluation. arXiv:2310.03262, 2023. doi:10.48550/arXiv.2310.03262, https://arxiv.org/abs/2310.03262
  5. Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding Emergent Abilities of Language Models from the Loss Perspective. arXiv:2403.15796 (NeurIPS 2024), 2024. doi:10.48550/arXiv.2403.15796, https://arxiv.org/abs/2403.15796
  6. Charlie Snell, Eric Wallace, Dan Klein, and Sergey Levine. Predicting Emergent Capabilities by Finetuning. arXiv:2411.16035, 2024. doi:10.48550/arXiv.2411.16035, https://arxiv.org/abs/2411.16035
  7. Leonardo Berti, Flavio Giorgi, and Gjergji Kasneci. Emergent Abilities in Large Language Models: A Survey. arXiv:2503.05788, 2025. doi:10.48550/arXiv.2503.05788, https://arxiv.org/abs/2503.05788
  8. Rosie Zhao, Tian Qin, David Alvarez-Melis, Sham Kakade, and Naomi Saphra. Random Scaling of Emergent Capabilities. arXiv:2502.17356, 2025. doi:10.48550/arXiv.2502.17356, https://arxiv.org/abs/2502.17356