Multitask Prompted Training Enables Zero-Shot Task Generalization

By Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja and 32 more. 2021

2,030 citations recorded by the source corpus.

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Yejin Choi
  6. Graham Neubig
  7. Quoc V. Le
  8. Noah A. Smith

In Prompting & Reasoning

  1. Luke S. Zettlemoyer
  2. Denny Zhou
  3. Hannaneh Hajishirzi
  4. Swaroop Mishra
  5. Chitta Baral
  6. Wenhu Chen
  7. Tushar Khot
  8. Timo Schick

Citations in the corpus