The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
By Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei and 1 more. 2023
953 citations recorded by the source corpus.
We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022). Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+) performance in all settings. In further experiments, we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks, motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks. Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available at https://github.com/google-research/FLAN/tree/main/flan/v2.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Alignment & RLHF, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Quoc V. Le
- Yejin Choi
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
- Andrea Vedaldi
In Electrical and Electronic Engineering
- Tim Dettmers
- Luke S. Zettlemoyer
- Ari Holtzman
- Artidoro Pagnoni
- Wenhu Chen
- Quoc V. Le
- Yi Tay
- Tianyu Pang
In Alignment & RLHF
- Luke S. Zettlemoyer
- Tianyi Zhang
- Xinyun Chen
- Noah A. Smith
- Ari Holtzman
- Percy Liang
- Carlos Guestrin
- Tatsunori B. Hashimoto
In Evaluation & Benchmarks
- Hannaneh Hajishirzi
- Mohit Bansal
- Luke S. Zettlemoyer
- Ashish Sabharwal
- Noah A. Smith
- Yejin Choi
- Graham Neubig
- Sebastian Ruder
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Chung et al. (2022) and Longpre et al. (2023) investigate the impact of instruction tuning as a function of number of tasks, model size, prompt settings, etc. Prompts used for instruction tuning can be created by humans or by LLMs themselves (Zhou et al., 2022), and follow-up instructions can be…”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “• Instruction-following benchmarks, such as Flan [23, 38], Self-instruct [36], NaturalInstructions [24], Super-NaturalInstructions [37], expand to slightly more open-ended questions and more diverse tasks and are used to evaluate LLMs after instruction fine-tuning.”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “The FLAN v2 collection [39] is a collection of 1836 tasks augmented with hundreds of manually curated templates and rich formatting patterns into over 15M examples.”
- PaLM 2 Technical Report (2023, background): “The Flan dataset (Chung et al., 2022; Longpre et al., 2023) contains more than 1,800 tasks, and at least 20 instruction templates per task (10 zero-shot templates, 10 few-shot templates).”
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “Large language models (LLMs) [9, 12, 44] have demonstrated unprecedented capabilities in following diverse and open-ended instructions [45, 5, 36].”
- LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “For the latter, our are motivated by FLAN [136, 88, 144].”
- Aligning Large Language Models with Human: A Survey (2023, methodology): “(c) Evaluating LLMs PromptSource (Bach et al., 2022), SuperNaturalInstruction (Wang et al., 2022b), FLAN (Longpre et al., 2023), Unnatural Instructions (Honovich et al., 2022), OIG (Nguyen et al., 2023) Hand-crafted Instructions Dolly-v2 (Conover et al., 2023), OpenAssistant (Kopf et al., 2023),…”
- DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping (2023, methodology): “Instruction tuning serves as a promising method, involving the fine-tuning of LLMs using training data and instructions from a collection of upstream tasks(Sanh et al., 2022; Mishra et al., 2022; Wei et al., 2022; Chung et al., 2022; Longpre et al., 2023; Peng et al., 2023).”
- Specializing Smaller Language Models towards Multi-Step Reasoning (2023, background): “Concurrent works also observe that models after finetuning on Flan perform better then their raw pretrained checkpoint (Longpre et al., 2023; Shen et al., 2023) The intuitive explanation is because instruction-tuning elicits the model’s full ability while raw pretrained models’ ability are not fully”
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (2023, methodology): “Scaling tasks and instructions: We utilize the Flan 2022 Collection [19] as it provides an extensive public assortment of tasks and instructions.”
- Adapted large language models can outperform medical experts in clinical text summarization (2023, positive): “Given the model’s lower error rate in each category, this suggests that incorporating LLMs could actually reduce fabricated information in clinical practice.”
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (2023, positive): “In particular, honesty will be assessed by TruthfulQA and FLAN as they both contain reference answers, based on which it is easier for the annotator to judge if the uncertainty expressed in LLMs’ responses calibrates with the accuracy.”
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning (2023, methodology): “The technique of instruction tuning serves this purpose (Wei et al. 2022; Longpre et al. 2023).”
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (2023, neutral): “Closed domain instruction tuning Early instruction-following training work (Wei et al., 2021; Longpre et al., 2023) 2022) raised the number of tasks to the thousands.”
- LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, methodology): “…data is generally of two types: (1) task-based datasets curated from traditional NLP tasks (Wang et al., 2022; Sanh et al., 2022; Wei et al., 2022b; Longpre et al., 2023), and (2) open-ended instruction following datasets, broadly covering a wide range of topics (Taori et al., 2023; Conover et…”
- Instruction Tuning for Large Language Models: A Survey (2023, neutral): “Despite significant progress, several challenges persist in the construction and use of IT datasets.”