Multitask Prompted Training Enables Zero-Shot Task Generalization
By Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja and 32 more. 2021
2,030 citations recorded by the source corpus.
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.
Subfields: Artificial Intelligence, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Quoc V. Le
- Noah A. Smith
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Denny Zhou
- Hannaneh Hajishirzi
- Swaroop Mishra
- Chitta Baral
- Wenhu Chen
- Tushar Khot
- Timo Schick
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “While these are promising improvements, this performance is still below the performance of task-adapted models (e.g., Sanh et al. (2021) when framed as an entailment task) and human performance of 95.9% (Rudinger et al., 2018).”
- Training language models to follow instructions with human feedback (2022, methodology): “We additionally compare InstructGPT to fine-tuning 175B GPT-3 on the FLAN (Wei et al., 2021) and T0 (Sanh et al., 2021) datasets, which both consist of a variety of NLP tasks, combined with natural language instructions for each task (they differ in the NLP datasets included, and the style of…”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “Reward Modelling Phase : In the second phase the SFT model is prompted with prompts x to produce pairs of answers ( y 1 , y 2 ) ⇠ ⇡ SFT ( y | x ) . denoted These are then presented to human labelers who express preferences for one answer, as y w � y l | x where y w and y l denotes the preferred and…”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “One recent line of work aims to improve the ability of language models to perform a task by providing instructions that describe the task (Raffel et al., 2020; Wei et al., 2022; Ouyang et al., 2022; Sanh et al., 2022; Wang et al., 2022b).”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “Both T0 [59] and FLAN [70] have taken this path and have shown that such an approach can improve zero-shot learning capabilities of language models.”
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “Perhaps the papers most related to ours are the recent Sanh et al. (2021) and Min et al. (2021), which were released after our initial preprint.”
- Emergent Abilities of Large Language Models (2022, neutral): “…does not explain why the quality of intermediate steps suddenly emerges to above random, and using evaluation metrics that do not give partial credit are at best an incomplete explanation, because emergent abilities are still observed on many classification tasks (e.g., the tasks in Figure 2D–”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “We show that a model trained on this collection outperforms other public collections on all tested evaluation benchmarks, including the original Flan 2021 (Wei et al., 2021), T0++ (Sanh et al., 2021), Super-Natural Instructions (Wang et al., 2022c), and the concurrent work on OPT-IML (Iyer et al., 2”
- Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “Experiments show that fine-tuning an 11Bparameter T5 model (Raffel et al., 2020) on Unnatural Instructions can outperform both T0++ (Sanh et al., 2021) and Tk-Instruct (Wang et al., 2022) across several benchmarks, including SuperNatural Instructions (Wang et al., 2022), BIGbench Hard (Suzgun et…”
- Galactica: A Large Language Model for Science (2022, methodology): “Separately, research such as FLAN and T0 showed prompt tuning can boost downstream performance (Wei et al., 2021; Sanh et al., 2021; Chung et al., 2022).”
- Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “can execute a broad range of natural language programs, the way these programs are processed may not be intuitive for humans, and the quality of instruction can only be measured when executing these instructions on a downstream task (Sanh et al., 2022; Wei et al., 2021).”
- Scaling Instruction-Finetuned Language Models (2022, methodology): “This degradation may be surprising in light of multiple prior studies finding that instruction finetuning improves performance on unseen tasks (Wei et al., 2021; Sanh et al., 2021; Wang et al., 2019a; Min et al., 2022, inter alia ).”