Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers

By Stephanie C. Y. Chan, Adam Santoro, Andrew Kyle Lampinen, Jane X. Wang, Aaditya K Singh, H. Pierre, Richemond, James L. McClelland, Felix Hill

Subfields: Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Yi Tay
2. Denny Zhou
3. Donald Metzler
4. Jason Wei
5. Xinyun Chen
6. Joseph Dean
7. William Fedus
8. Percy Liang

In Prompting & Reasoning

1. Denny Zhou
2. Xinyun Chen
3. Jason Wei
4. Ed H. Chi
5. Percy Liang
6. Tengyu Ma
7. Yi Tay
8. Albert Webson

Citations in the corpus

• Emergent Abilities of Large Language Models (2022, positive): “, long-range coherence, having many rare classes) have also been shown to correlate with emergent few-shot prompting and could potentially enable it in smaller models (Xie et al., 2022; Chan et al., 2022), and few-shot learning can require certain model architectures in some scenarios (Chan et al.”
• Grounding Language Models to Images for Multimodal Generation (2023, methodology): “As FROMAGe uses a frozen LLM as its backbone, it is also capable of in-context learning (Brown et al., 2020; Chan et al., 2022), where it generalizes rapidly from a few input examples.”
• What learning algorithm is in-context learning? Investigations with linear models (2022, background): “Past work has characterized what kinds of functions ICL can learn (Garg et al., 2022; Laskin et al., 2022; M ¨ uller et al., 2021) and the distributional properties of pretraining that can elicit in-context learning (Xie et al., 2021; Chan et al., 2022).”
• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Chan et al. [12] demonstrate that distributional properties such as long-tailedness are crucial for in-context learning on an image-based few-shot dataset.”
• Larger language models do in-context learning differently (2023, background): “Although we have shown that this behavior is emergent with respect to model scale, the investigation of why scaling unlocks such behaviors (Xie et al., 2022; Chan et al., 2022) is still an open question that we leave for future work.”