It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
By Timo Schick, Hinrich Schütze. 2020
1,122 citations recorded by the source corpus.
When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much “greener” in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.
Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Graham Neubig
- Yejin Choi
- Noah A. Smith
- Hannaneh Hajishirzi
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Hannaneh Hajishirzi
- Yejin Choi
- Timo Schick
- Mohit Bansal
- Graham Neubig
- Zhiyuan Liu
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Timo Schick
- Swaroop Mishra
- Michael Lewis
- Hinrich Schütze
- Denny Zhou
- Mohit Bansal
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, background): “…can also be accomplished by combining textual templates (“prompts”) and various forms of model finetuning, either fully updating a model’s parameters, e.g. for classification (Schick & Schütze, 2021a; Schick & Schutze, 2021; Gao et al., 2021; Tam et al., 2021) or generation (Schick & Schütze, 2021b).”
- Emergent Abilities of Large Language Models (2022, neutral): “There has also been work on improving the general few-shot prompting abilities of language models (Gao et al., 2021; Schick & Schütze, 2021, inter alia ).”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “There has also been work on eliciting prompting behavior in smaller models (Schick and Schütze, 2020; Gao et al., 2021b; Li and Liang, 2021; Lester et al., 2021; Scao and Rush, 2021), improving the flexi-bility of prompting (Shin et al., 2020), and understanding why and how prompting works (Liu et…”
- Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2021, methodology): “The results for different few-shot learning meth-ods can be quite different across datasets and seeds for the training set (Zhao et al., 2021; Schick and Schütze, 2021a).”
- Improving and Simplifying Pattern Exploiting Training (2021, methodology): “For a description of the different patterns used across the tasks (Schick and Schütze, 2020b), we refer the reader to Appendix A.”
- How many data points is a prompt worth? (2021, methodology): “We further thank Timo Schick for making the code for PET available and for discussions about performance replication.”
- Few-Shot Text Generation with Natural Language Instructions (2020, methodology): “As recommended by Schick and Schütze (2021a), we train all models for 250 steps using a batch size of 8.”
- Large Language Models Can Self-Improve (2022, neutral): “Language models are known to preserve parametric knowledge (Schick and Schütze, 2020a,b) during the pre-training stage.”
- Large Language Models are Zero-Shot Reasoners (2022, methodology): “The method of conditioning the language model is called “prompting” [Liu et al., 2021b], and designing prompts either manually [Schick and Schütze, 2021, Reynolds and McDonell, 2021] or automatically [Gao et al., 2021, Shin et al., 2020] has become a hot topic in NLP. Figure 1: Example inputs and…”
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, methodology): “We take the perspective that using the unlabeled training data is permitted (Kodirov et al., 2015; Wang et al., 2019b; Schick and Schütze, 2021).”
- Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “While these values have not been standardized, two prominent papers, Schick and Schütze (2021b) and Brown et al. (2020), use 32 and 64 prompt examples, respectively.”
- Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “(14)Although sometimes the same might be achieved with less: PET/iPET (Schick and Schütze, 2021b) outperform GPT3 on SuperGLUE (Wang et al.”
- A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, background): “…they can perform a variety of NLP tasks by expressing the task as a prompt that encourages the model to output the desired answer as the continuation (Puri and Catan-zaro, 2019; Weller et al., 2020; Brown et al., 2020; Schick and Schütze, 2021, inter alia ; see Liu et al. (2021a) for a survey).”
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “Similar to text classiication tasks, for text pair classiication tasks, cloze prompts are commonly used [117, 120].”
- Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021, background): “Independent and concurrent work also shows that when finetuning masked language models on few examples, the choice of training examples can impact results (Schick & Sch ¨ utze, 2020; Gao et al., 2020).”
- HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, methodology): “We provide the same analysis extended to BART, T5-Large, and HTLM using the same PET prompts provided in Schick and Sch¨utze (2020).”
- Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Other work has shown that good few-shot performance can also be obtained by combining gradient based optimization with textual prompts (Schick and Sch¨utze, 2021b; Gao et al., 2021; Tam et al., 2021; Wang et al., 2021).”