What learning algorithm is in-context learning? Investigations with linear models

By Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou. International Conference on Learning Representations, 2022

748 citations recorded by the source corpus.

Neural sequence models, especially transformers, exhibit a remarkable capacity for in-context learning. They can construct new predictors from sequences of labeled examples $(x, f(x))$ presented in the input without further parameter updates. We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context. Using linear regression as a prototypical problem, we offer three sources of evidence for this hypothesis. First, we prove by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. Second, we show that trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression, transitioning between different predictors as transformer depth and dataset noise vary, and converging to Bayesian estimators for large widths and depths. Third, we present preliminary evidence that in-context learners share algorithmic features with these predictors: learners' late layers non-linearly encode weight vectors and moment matrices. These results suggest that in-context learning is understandable in algorithmic terms, and that (at least in the linear case) learners may rediscover standard estimation algorithms. Code and reference implementations are released at https://github.com/ekinakyurek/google-research/blob/master/incontext.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/7aa801b907b59b8ee4cfb1296d9dac22c5164c5d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Looped Transformers as Programmable Computers (2023, methodology): “Motivated by the above experimental work, in Akyürek et al. [2022], the authors investigate the hypothesis that TF-based in-context learners emulate standard learning algorithms implicitly at inference time. The authors provide evidence for this hypothesis by constructing transformers that implement”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “Few-shot CoT exemplars also trivial-ize stereotype benchmarks for two reasons: (1) providing in-context examples may be similar to finetuning (Akyürek et al., 2022) and (2) models could learn to simply repeat neutral responses.”
A Survey on In-context Learning (2022, methodology): “Taking linear regression as a starting point, Akyürek et al. (2022) found that Transformer-based in-context learners can implement standard finetuning algorithms implicitly, and von Oswald et al. (2022) showed that linear attention-only Transform-ers with hand-constructed parameters and models…”
Larger language models do in-context learning differently (2023, background): “…that models primarily rely on semantic prior knowledge), other work has shown that transformers in simple settings (without language modeling pretraining) implement learning algorithms such as ridge regression and gradient descent (Akyürek et al., 2022; von Oswald et al., 2022; Dai et al., 2022).”
Large Language Models (2023, methodology): “A few recent interesting works are [2, 51, 103].”
Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective (2023, methodology): “Subsequent works further revealed that Transformers can implement learning algorithms such as linear regression [1], gradient descent [1, 54, 16], and Bayesian inference [59].”
Transformers learn to implement preconditioned gradient descent for in-context learning (2023, methodology): “This capability inspired a line of research that examines the algorithmic power of transformers (Pérez et al., 2021; Wei et al., 2022; Giannou et al., 2023; Akyürek et al., 2022; Olsson et al., 2022).”
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2022, background): “We note that what we find does not in any way diminish the potential of learning to reason in context for LLMs; recent work has also shown evidence that learning in context is possible and could be powerful (Garg et al., 2022; Akyürek et al., 2023).”
A Survey for In-context Learning (2023, methodology): “Taking linear regression as a starting point, Akyürek et al. (2022) found that Transformer-based in-context learners can implement standard ﬁnetuning algorithms implicitly.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, neutral): “3.3.”
In-Context Learning Creates Task Vectors (2023, methodology): “For example, it was shown that when training a transformer from scratch to perform linear regression in context, the emerging learning algorithm is similar to Stochastic Gradient Descent (Akyürek et al., 2022; von Oswald et al., 2022).”
When large language models meet personalization: perspectives of challenges and opportunities (2023, neutral): “In-context learning [148], [149], [150], [151], [152], [153], [154]”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, background): “Note that the layer-norm operation can be bypassed as explained by Akyurek et al. (2023).”
A Survey of Large Language Models (2023, positive): “Specifically, the authors in [242] find that LLMs essentially encode implicit models through their parameters during pre-training.”