Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
By Or Honovich, Thomas Scialom, Omer Levy, Timo Schick. 2022
487 citations recorded by the source corpus.
Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs. Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks. These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification.
Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Furu Wei
- Percy Liang
- Yejin Choi
- Noah A. Smith
In Natural Language Processing
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Hannaneh Hajishirzi
- Graham Neubig
- Furu Wei
- Noah A. Smith
In Alignment & RLHF
- Luke S. Zettlemoyer
- Jianfeng Gao
- Ari Holtzman
- Tim Dettmers
- Artidoro Pagnoni
- Noah A. Smith
- Jon Ander Campos
- Pieter Abbeel
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Attempts to close this gap have emerged, with distillation-based models such as Vicuna (Chiang et al., 2023) and Alpaca (Taori et al., 2023) adopting a unique approach to training with synthetic instructions (Honovich et al., 2022; Wang et al., 2022).”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Honovich et al., 2022; Wang et al., 2022): Given just a handful of human-written examples of how an API can be used, we let a LM annotate a huge language modeling dataset with potential API calls.”
- The False Promise of Imitating Proprietary LLMs (2023, background): “A surge of recent publications have attempted to both locally imitate proprietary models for specific tasks (Sun et al., 2023; Hsieh et al., 2023; Honovich et al., 2022) and broadly imitate models, e.g., Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), GPT4ALL…”
- Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “In parallel with our work, Honovich et al. (2022a) also propose to generate large-scale instruction data (so-called Unnatural Instructions) with GPT3 models.”
- Code Llama: Open Foundation Models for Code (2023, neutral): “FIM Size HumanEval MBPP Test loss pass@1 pass@10 pass@100 pass@1 pass@10 pass@100”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “Self-Instruct, Alpaca, Unnatural Instructions The Self-Instruct, Alpaca, and Unnatural Instructions datasets [60, 56, 26] are instruction tuning datasets collected with various approaches of model distillation from GPT-3 Instruct and ChatGPT.”
- LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “However, relying solely on generation via LLMs has drawbacks: the overall validity of Self-Instruct is only 54% (Wang et al., 2023b).”
- Instruction Tuning with GPT-4 (2023, background): “ChatCompletion.create( 13 model="gpt-4", 14 messages=["role": "user", "content": prompt], 15 temperature = 1.0, 16 top p=1.0, # nucleus sampling over entire vocabulary 17 max tokens=512 # the max number of generated tokens 18 ) instructions (Honovich et al., 2022).”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “To improve instruction-following capabilities, we perform language-only instruction tuning using Unnatural Instructions [24] and FLANv2 [25] datasets.”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “explored self-instruct, which aims to instruct an LLM by itself or by other LLMs [129], [259]–[261].”
- DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping (2023, background): “Instruction-Tuning Data Collection The collection of high-quality instruction-tuning data (Xu et al., 2023; Yin et al., 2023; Honovich et al., 2023) is a pressing issue in enhancing the capability of instruction-following.”
- MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, unknown): “Synthetic data generation is constrained by the languages supported by existing models and suffers from validity (Wang et al., 2023) and creativity (Honovich et al., 2023) issues.”
- Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, unknown): “3.3.”
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning (2023, methodology): “A.1 Number of Pre-Experience Data Following the findings from LIMA that 1000 high-quality samples are enough to train a reasonably good model, we set the amount of data used for our pre-experienced model as 1000 .”
- LIMA: Less Is More for Alignment (2023, methodology): “While some recent works avoid manual labor via distillation and other automatic means [Honovich et al., 2022, Wang et al., 2022a, Taori et al., 2023, Chiang et al., 2023, Sun et al., 2023], optimizing for quantity over quality, this work explores the effects of investing in diversity and quality…”
- MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (2023, methodology): “To fix this, we add the language dataset, Unnatural Instruction [17] into our model’s third-stage training for helping recover the language generation ability.”