In-context Learning and Induction Heads

By Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova Dassarma, T. Henighan, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen and 16 more. arXiv.org, 2022

942 citations recorded by the source corpus.

"Induction heads"are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] ->[B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all"in-context learning"in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning in transformer models of any size. For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

View Paper: https://arxiv.org/abs/2209.11895

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Emergent Abilities of Large Language Models (2022, neutral): “Moreover, better exploration of what makes prompting successful (Wei et al., 2021a; Xie et al., 2022; Min et al., 2022; Olsson et al., 2022) could lead to insights on how to elicit emergent abilities at a smaller model scale.”
STaR: Bootstrapping Reasoning With Reasoning (2022, background): “Some have studied in-context learning based on the language modeling objective in terms of Bayesian inference [13] while others have attempted to describe the process more mechanistically in terms of “induction heads” [14].”
StarCoder: may the source be with you! (2023, background): “While API access allows researchers to experiment with these models, it limits their ability to research LLM safety (Perez et al., 2022), inspect the models’ inner workings (Olsson et al., 2022), and contribute to model improvements (Togelius & Yannakakis, 2023).”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, neutral): “This could be related to some attention heads having more important functions [VTM + 19, Vig19, OEN + 22].”
One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “Olsson et al. (2022) further discusses a multi-layer transformer can do more complex n-gram estimation using an induction heads mechanism.”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “” For example, LLMs are trained to model a distribution of internet text; this training leads to emergent behavior such as learning from very few examples [Olsson et al. 2022], or arithmetic [Brown et al. 2020], or even the ability itself to perform sequential reasoning [Wei et al. 2023].”
Progress measures for grokking via mechanistic interpretability (2023, methodology): “The style of post-hoc mechanistic interpretability in Section 4 is heavily inspired by the Circuits approach of Cammarata et al. (2020), Elhage et al. (2021), and Olsson et al. (2022).”
SnapKV: LLM Knows What You are Looking for Before Generation (2024, methodology): “In LLMs, information retrieval and generation rely on features with high attention weight and are supplemented by copying the rest in context using induction heads [15].”
Large Language Models (2023, background): “An example of a circuit which appears in trained transformer models is the induction head [42, 107].”
Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective (2023, background): “The works of [20, 39] studied in-context learning via the concept of “induction heads”.”
Transformers learn to implement preconditioned gradient descent for in-context learning (2023, background): “…property has become the focus of body of recent research that aims to shed light on the underlying mechanism of large language models (Garg et al., 2022; Akyürek et al., 2022; von Oswald et al., 2023; Li and Malik, 2017; Min et al., 2021; Xie et al., 2021; Elhage et al., 2021; Olsson et al., 2022).”
The debate over understanding in AI’s large language models (2022, background): “…to develop new kinds of benchmarks and probing methods that can yield insight into the mechanisms of diverse types of intelligence and understanding, including the novel forms of ”exotic, mind-like entities” [71] we have created, perhaps along the lines of some promising initial eﬀorts [49, 64].”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Interestingly, Olsson et al. [48] observe a similar jump in the in-context learning ability of a language model which they attribute to the formation of “induction heads”.”
When large language models meet personalization: perspectives of challenges and opportunities (2023, neutral): “In-context learning [148], [149], [150], [151], [152], [153], [154]”
Transformers learn in-context by gradient descent (2022, background): “Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers.”