The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

By Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei and 1 more. 2023

953 citations recorded by the source corpus.

We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022). Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+) performance in all settings. In further experiments, we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks, motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks. Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available at https://github.com/google-research/FLAN/tree/main/flan/v2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Alignment & RLHF, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Graham Neubig
  6. Quoc V. Le
  7. Yejin Choi
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Jitendra Malik
  6. Ming-Hsuan Yang
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Electrical and Electronic Engineering

  1. Tim Dettmers
  2. Luke S. Zettlemoyer
  3. Ari Holtzman
  4. Artidoro Pagnoni
  5. Wenhu Chen
  6. Quoc V. Le
  7. Yi Tay
  8. Tianyu Pang

In Alignment & RLHF

  1. Luke S. Zettlemoyer
  2. Tianyi Zhang
  3. Xinyun Chen
  4. Noah A. Smith
  5. Ari Holtzman
  6. Percy Liang
  7. Carlos Guestrin
  8. Tatsunori B. Hashimoto

In Evaluation & Benchmarks

  1. Hannaneh Hajishirzi
  2. Mohit Bansal
  3. Luke S. Zettlemoyer
  4. Ashish Sabharwal
  5. Noah A. Smith
  6. Yejin Choi
  7. Graham Neubig
  8. Sebastian Ruder

Citations in the corpus