# A metric can make gradual model gains look like a sudden leap

> Three analyses test whether discontinuous scoring rules create apparent emergent abilities in language-model scaling curves.

Canonical: https://aischolar.cc/blog/metric-choice-emergence

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# A metric can make gradual model gains look like a sudden leap

Three analyses test whether discontinuous scoring rules create apparent emergent abilities in language-model scaling curves.

November 18, 2026

_A benchmark score can jump abruptly even when model outputs improve gradually._

### The scoring rule changes the picture

Schaeffer, Miranda, and Koyejo build a mathematical explanation for apparent emergence in [their analysis of language-model scaling](https://arxiv.org/abs/2304.15004v2): a thresholded or nonlinear metric can convert smooth changes in outputs into sharp transitions. They test predictions using the InstructGPT/GPT-3 family on tasks previously described as emergent, a meta-analysis of BIG-Bench claims, and demonstrations across vision networks.

Their analyses report that some apparent emergent abilities disappear under alternative metrics or improved statistical treatment. They also show that metric choices can produce seemingly emergent patterns in vision tasks. The result reframes how to read a curve: a sharp boundary in a score need not mark an equally sharp change in the underlying outputs.

The authors analyze fixed outputs and prior results available to them; they note that public outputs constrained parts of the reanalysis. Their study challenges an interpretation of measured curves, not every claim that scaling produces new capabilities. When an evaluation reports a thresholded score, pair it with a continuous measure of the same outputs where possible.

### Bibliography

-   Rylan Schaeffer; Brando Miranda; Sanmi Koyejo. (2023). Are Emergent Abilities of Large Language Models a Mirage? arXiv:2304.15004v2. https://arxiv.org/abs/2304.15004v2

Original source: [https://arxiv.org/abs/2304.15004v2](https://arxiv.org/abs/2304.15004v2)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
