Are Emergent Abilities of Large Language Models a Mirage?

By Rylan Schaeffer, Brando Miranda, Oluwasanmi Koyejo. Neural Information Processing Systems, 2023

707 citations recorded by the source corpus.

Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance. We present our alternative explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities; (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show to choose metrics to produce never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep networks. Via all three analyses, we provide evidence that alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/29c7f009df21d0112c48dec254ff80cc45fac3af

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Erik Cambria
2. J. Kolter
3. Percy Liang
4. Zhuang Liu
5. Xuanjing Huang
6. Mingjie Sun
7. Efstratios Gavves
8. Qika Lin

Citations in the corpus

• Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023, background): “Recent research has unveiled an unpredictable phenomenon known as emergent abilities, which manifest exclusively in larger language models while eluding their smaller counterparts (Schaeffer et al., 2023).”
• A Simple and Effective Pruning Approach for Large Language Models (2023, background): “6B), large magnitude features start to emerge and strongly affect all layers, which is considered as one of the emergent properties of LLMs [14, 58, 66].”
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024, neutral): “While it includes simple math questions with detailed solutions, making it suitable for techniques like Chain-of-Thought (CoT) prompting, it provides only a single metric on a fixed set of questions.”
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, neutral): “The dying declaration exception is not just “commonly” invoked in homicide prosecutions; FRE 804(b)(2) specifically limits the admissibility of dying declarations to cases involving “a prosecution for homicide or in a civil case,” not any case.”
• Yi: Open Foundation Models by 01.AI (2024, methodology): “Considering the discussions of whether emergent ability is an artifact of measurement [65], we use difference to the target (target number - model prediction) as a continuous measure, and exact match (target number == model prediction) as a discontinuous measure.”
• Multi-Agent Collaboration Mechanisms: A Survey of LLMs (2025, neutral): “These emergent behaviors allow LLMs to solve tasks they were not explicitly trained on, such as analogical reasoning and zero-shot learning, where the model can tackle new problems without additional fine-tuning [113].”
• DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, background): “And this has been discussed in Schaeffer et al. (2024), "sharp and unpredictable changes might be induced by the researcher’s choice of measurement, even though the model family’s per-token error rate changes smoothly, continuously and predictably with increasing scale."”
• Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “However, the use of an external GPT4 model as a gold standard is still debatable, and new efforts in LLM for benchmarking and identifying the corner cases have been reported to address the limitations of existing evaluation measures, e.g., [36], [173], [174].”
• A Survey on Model Compression for Large Language Models (2023, neutral): “At the same time, Recent research [Wei et al. , 2022a; Schaeffer et al. , 2023; Zhao et al. , 2023] underscores that with the emphasis on augmenting model size, LLMs like GPT-3 (175B parameters) and PaLM (540B parameters) showcase unique behaviors when compared to smaller models like BERT (330M…”
• A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, background): “Despite sharing many technical components, it is remarkable that simply scaling up these models leads to the emergence of novel behaviors, enabling qualitatively distinct capabilities [18].”
• Mechanistic Interpretability for AI Safety - A Review (2024, methodology): “Other examples include analyzing circuits for Python docstrings (Heimersheim & Jett, 2023), "an" vs "a" usage (Miller & Neo, 2023), and price tagging (Wu et al., 2023b).”
• HyQE: Ranking Contexts with Hypothetical Query Embeddings (2024, unknown): “…the small-size open source models such as Mistral-7b (Jiang et al., 2023) to the large-size proprietary mod-13015 els such as GPT-4 (Achiam et al., 2023), are pre-trained on trillions of tokens, exhibiting un-paralleled emergent and generalization abilities across tasks (Schaeffer et al., 2023).”