Emergent Abilities of Large Language Models

By Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler and 6 more. 2022

3,700 citations recorded by the source corpus.

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, background): “Finally, Wei et al. (2022) studied the effect of scaling on the abilities of large language models.”
Few-shot Learning with Retrieval Augmented Language Models (2022, background): “, 2016), but has recently seen an explosion of interest in NLP with the arrival of large pre-trained models, which exhibit emergent few-shot learning abilities (Wei et al., 2022).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “, 2022) and sometimes suddenly (Wei et al., 2022)—has led to a trend of increasing scale (Zeng et al.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Progress in natural language processing is increasingly driven by sheer compute scale alone (Sevilla et al., 2022): as more compute is expended to train large language models (LLM), they gain and exhibit powerful emergent capabilities (Brown et al., 2020; Wei et al., 2022).”
Scaling Instruction-Finetuned Language Models (2022, methodology): “This is consistent with prior observations that CoT is an emergent ability of scaling language models (Wei et al., 2022a).”
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2022, methodology): “This qualitative transition from approximately random to improved performance with scale has been referred to as an emergent ability (Wei et al., 2022a).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Providing explicit instructions to not generate certain kinds of text has shown some promise (Ouyang et al., 2022; Wei et al., 2022) and is an interesting avenue for future work.”
Inverse scaling can become U-shaped (2022, methodology): “We explore whether chain-of-thought (CoT) prompting can help in such scenarios, based on recent work showing that CoT can greatly improve performance for multi-step reasoning tasks (Wei et al., 2022b; Kojima et al., 2022; Suzgun et al., 2022, i.a. ).”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “…luQ-CmaEuhuv0 12https://huggingface.co/datasets/bigscience/evaluation-results/tree/ma in/bloom/bloomzeval/transformers/evaluation_val C.5 BIG-BENCH-LITE EVALUATION Recent works (Wei et al., 2022c; Wang et al., 2022c) reveal that LLMs are capable to do reasoning beyond conventional language tasks.”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “To facilitate the interaction between modalities, another potential solution is to ﬁne-tune smaller language models (LMs) by fusing multimodal features (Zhang et al., 2023).”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “…and study how scaling affects the model’s ability to produce appropriate prosody and expression for challenging text inputs, similar to how LLMs acquire new abilities through data and parameter scaling, a phenomenon known as "emergence" or "emergent abilities" in the LLM literature [32, 33].”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, background): “Wei et al. (2022b) have shown that chain-of-thought prompting signiﬁcantly improves the reasoning performance of language models, by adding explicit reasoning steps before the ﬁnal answer.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “In addition to competitive performance across a range of well-established NLP [Wang et al., 2019], multilingual [Clark et al., 2020a, Shi et al., 2022], and reasoning [Cobbe et al., 2021] benchmarks, we also study the impact of UL2R on a suite of challenging BigBench tasks from Wei et al. [2022a].”
Distilling Multi-Step Reasoning Capabilities of Large Language Models into Smaller Models via Semantic Decompositions (2022, background): “In many real-world scenarios, such intermediate solutions are not available, and in such cases, we rely on LLMs to generate those steps leading to additional noise in the training data.”