Scaling Instruction-Finetuned Language Models

By Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma and 22 more. Journal of machine learning research, 2022

4,301 citations recorded by the source corpus.

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.

Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF

View Paper: https://arxiv.org/abs/2210.11416

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Graham Neubig
  6. Quoc V. Le
  7. Yejin Choi
  8. Noah A. Smith

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Timo Schick
  5. Jianfeng Gao
  6. Hannaneh Hajishirzi
  7. Noah A. Smith
  8. Graham Neubig

In Alignment & RLHF

  1. Luke S. Zettlemoyer
  2. Jianfeng Gao
  3. Ari Holtzman
  4. Tim Dettmers
  5. Artidoro Pagnoni
  6. Yejin Choi
  7. Xinyun Chen
  8. Mari Ostendorf

Citations in the corpus