Finetuned Language Models Are Zero-Shot Learners
By Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le. 2021
5,221 citations recorded by the source corpus.
This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Yejin Choi
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Timo Schick
- Hannaneh Hajishirzi
- Mohit Bansal
- Furu Wei
- Jianfeng Gao
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Denny Zhou
- Swaroop Mishra
- Timo Schick
- Hinrich Schütze
- Yejin Choi
- Mohit Bansal
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “In Section 6.3, we empirically demonstrated how chain-of-thought prompting (Wei et al., 2022b) can drastically improve prediction accuracy in multi-step reasoning tasks.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Wei et al. (2021) obtained zero-shot performance on unseen tasks by fine-tuning LLMs on numerous datasets.”
- Training language models to follow instructions with human feedback (2022, methodology): “We compare GPT-3 fine-tuned on our human preference data (i.e. InstructGPT) to GPT-3 fine-tuned on two different compilations of public NLP tasks: the FLAN (Wei et al., 2021) and T0 (Sanh et al., 2021) (in particular, the T0++ variant).”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “One recent line of work aims to improve the ability of language models to perform a task by providing instructions that describe the task (Raffel et al., 2020; Wei et al., 2022; Ouyang et al., 2022; Sanh et al., 2022; Wang et al., 2022b).”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “Both T0 [59] and FLAN [70] have taken this path and have shown that such an approach can improve zero-shot learning capabilities of language models.”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “Our analysis uses the same methodology as Wei et al. (2021), which, in turn closely follows Brown et al. (2020).”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “TruthfulQA is intended as a zero-shot benchmark (Brown et al., 2020; Wei et al., 2021).”
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “…across three levels of generalization: 1) model performance on tasks from fully held-out task categories not used for tuning, as in prior work (Wei et al., 2022a; Sanh et al., 2022), and additionally, 2) performance on unseen tasks from categories seen during instructiontuning, and, 3)…”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “The finetuning hyperparameters are loosely based on T0 (Sanh et al., 2022) and FLAN (Wei et al., 2021).”
- Emergent Abilities of Large Language Models (2022, neutral): “Although Wei et al. (2022a) initially found that instruction-based finetuning only worked for 68B parameter or larger decoder-only models, Sanh et al. (2022) induced similar behavior in a 11B model with an encoder-decoder architecture, which typically has higher performance after finetuning than…”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “There is also evidence that step-function changes in capabilities may occur at a scale that is much smaller than 175B (Wei et al., 2021), indicating a need to examine a wider range of scales for different research applications.”
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “There is a significant and growing body of work on the influence of the syntactic structure of finetuning data on downstream performance (Zhong et al., 2021; Tan et al., 2021; Sanh et al., 2021; Wei et al., 2021).”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Moreover, fine-tuning would allow our models to be better able to condition on natural language instructions and other indications of human intent [76, 65, 48].”
- Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, methodology): “2Code and data are available at https://github.com/ yizhongw/self-instruct can follow natural language instructions (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022; Wang et al., 2022; Ouyang et al., 2022; Chung et al., 2022, i.a.).”
- Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, background): “Instruction tuning enables pretrained language models to generalize to unseen tasks in a zero-shot setting (Sanh et al., 2021; Wei et al., 2021).”
- Galactica: A Large Language Model for Science (2022, methodology): “Separately, research such as FLAN and T0 showed prompt tuning can boost downstream performance (Wei et al., 2021; Sanh et al., 2021; Chung et al., 2022).”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “can execute a broad range of natural language programs, the way these programs are processed may not be intuitive for humans, and the quality of instruction can only be measured when executing these instructions on a downstream task (Sanh et al., 2022; Wei et al., 2021).”
- Scaling Instruction-Finetuned Language Models (2022, methodology): “This degradation may be surprising in light of multiple prior studies finding that instruction finetuning improves performance on unseen tasks (Wei et al., 2021; Sanh et al., 2021; Wang et al., 2019a; Min et al., 2022, inter alia ).”