Self-Instruct: Aligning Language Models with Self-Generated Instructions
By Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi. Annual Meeting of the Association for Computational Linguistics, 2022
3,444 citations recorded by the source corpus.
Large “instruction-tuned” language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning.
Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF
View Paper: https://arxiv.org/abs/2212.10560
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Percy Liang
- Graham Neubig
- Noah A. Smith
- Quoc V. Le
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Noah A. Smith
- Jianfeng Gao
- Timo Schick
- Michael Lewis
- Graham Neubig
In Alignment & RLHF
- Luke S. Zettlemoyer
- Jianfeng Gao
- Ari Holtzman
- Tim Dettmers
- Artidoro Pagnoni
- Tianyi Zhang
- Noah A. Smith
- Yu Meng
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Attempts to close this gap have emerged, with distillation-based models such as Vicuna (Chiang et al., 2023) and Alpaca (Taori et al., 2023) adopting a unique approach to training with synthetic instructions (Honovich et al., 2022; Wang et al., 2022).”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Honovich et al., 2022; Wang et al., 2022): Given just a handful of human-written examples of how an API can be used, we let a LM annotate a huge language modeling dataset with potential API calls.”
- The False Promise of Imitating Proprietary LLMs (2023, background): “To date, recent works have looked to imitate OpenAI’s best systems, e.g., Self-Instruct (Wang et al., 2022a) and Alpaca (Taori et al., 2023), and initial results suggest that these models have achieved near parity with proprietary models.”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “The Second Wave A second wave of instruction tuning collections expanded prior resources: combining more datasets and tasks into one resource, like SuperNatural Instructions (Wang et al., 2022c) or OPT-IML (Iyer et al., 2022), adding multilingual instruction tuning in xP3 (Muennighoff et al., 2022),”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “While our initial evaluations using MT-bench and Chatbot Arena rely on real human ratings, collecting human preferences can be costly and laborious [36, 33, 27, 2].”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “Self-Instruct, Alpaca, Unnatural Instructions The Self-Instruct, Alpaca, and Unnatural Instructions datasets [60, 56, 26] are instruction tuning datasets collected with various approaches of model distillation from GPT-3 Instruct and ChatGPT.”
- Visual Instruction Tuning (2023, background): “In the natural language processing (NLP) community, to enable LLMs such as GPT-3 [7], T5 [37], PaLM [9], and OPT [56] to follow natural language instructions and complete real-world tasks, researchers have explored methods for LLM instruction-tuning [33, 48, 47], leading to instruction-tuned counter”
- LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “The dataset is split 23,652/2,042/2,045 into training/validation/test.”
- Instruction Tuning with GPT-4 (2023, methodology): “Instruction-tuning of OPT models is considered in (Iyer et al., 2022), where a larger and more comprehensive benchmark OPT-IML Bench is employed, covering FLAN (Wei et al., 2021), Super-NaturalInstructions (Wang et al., 2022b), and UnifiedSKG (Xie et al., 2022).”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Given 52K instruction-output data [62] and a pre-trained LLaMA [61] with an N -layer transformer, we adopt a set of learnable adaption prompts for instruction-following fine-tuning.”
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, neutral): “As LLMs become widely adopted in open-ended scenarios involving interaction with humans (e.g., chatbot), many have considered human evaluation using domain experts or crowd raters such as Amazon Mechanical Turk [23, 46] to examine models’ response quality.”
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, background): “However, common instruction-following abilities could be easily acquired through a lightweight instruction tuning process, as demonstrated in recent work (Wang et al., 2022; Taori et al., 2023).”
- CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility (2023, methodology): “Following the SelfInstruct method (Wang et al., 2023b), we use the prompts from the pool to augment more prompts using ChatGPT (OpenAI, 2022).”
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, positive): “4 LLAMA: [2] is an open-source decoder-only model based on publicly available data, and achieves competitive results compared with the existing LLMs. 5 ALPACA: [11] is a model fine-tuned on LLAMA-7b using 52 K instruction data generated by the techniques in the Self-Instruct [38].”
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, methodology): “The instructions and inputs of test data are sampled from the human evaluation dataset of self-instruct [19], with responses generated by different LLMs and each label independently provided by three different human evaluators.”
- Learning to Compress Prompts with Gist Tokens (2023, methodology): “To obtain the largest possible set of tasks for instruction finetuning, we create a dataset called Alpaca+, which combines the Self-Instruct (Wang et al., 2022a) and Stanford Alpaca (Taori et al.”