Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
By Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap and 30 more. 2022
1,112 citations recorded by the source corpus.
How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions—training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.
Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Graham Neubig
- Yejin Choi
- Hannaneh Hajishirzi
- Noah A. Smith
In Information Systems
- Hannaneh Hajishirzi
- Wen-tau Yih
- Luke S. Zettlemoyer
- Akari Asai
- Percy Liang
- Wayne Xin Zhao
- Ji-rong Wen
- Sebastian Riedel
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Timo Schick
- Hinrich Schütze
- Hannaneh Hajishirzi
- Noah A. Smith
- Michael Lewis
- Sebastian Riedel
In Natural Language Processing
- Luke S. Zettlemoyer
- Mohit Bansal
- Graham Neubig
- Percy Liang
- Yejin Choi
- Hannaneh Hajishirzi
- Noah A. Smith
- Jianfeng Gao
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Timo Schick
- Michael Lewis
- Pengfei Liu
- Hongsheng Li
- Lei Li
- Or Honovich
Citations in the corpus
- Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “Tk-Instruct Wang et al. (2022) fine-tune T5 v1.”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “• Instruction-following benchmarks, such as Flan [23, 38], Self-instruct [36], NaturalInstructions [24], Super-NaturalInstructions [37], expand to slightly more open-ended questions and more diverse tasks and are used to evaluate LLMs after instruction fine-tuning.”
- LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “Self-Instruct (Wang et al., 2023b) and Unnatural Instructions (Honovich et al., 2023) challenge academic benchmarks by generating instruction data (input, instruction and output) via LLMs to provide a more diverse dataset.”
- GPTScore: Evaluate as You Desire (2023, methodology): “3 For instruction-based pre-trained models, we use prompts from NaturalInstruction (Wang et al., 2022) since it’s the main training source for those instruction-based pre-train models.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The subfield of language models learning instruction-following capabilities aims to generate responses based on natural language commands, which have been extensively researched in language [64, 63, 3, 46], and multi-modality [59, 42] domains.”
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “There have been an number of works studying instruction following as a cross-task generalization across a pool of NLP tasks [39, 69, 51, 4, 68].”
- DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping (2023, methodology): “Firstly, methods like S UPER -NI (Wang et al., 2022) and D OLLY (Conover et al., 2023) rely on hiring professionals to create instructions for diverse NLP tasks.”
- A Survey of Large Language Models (2023, positive): “With the increasing of the task number, the model performance initially shows a continuous growth pattern, while the gain becomes negligible when it reaches a certain level [77, 82].”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, background): “To further expand the instruction dataset, a large community effort of hand-crafted instructions is conducted [258].”
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “Hand-crafted instruction instances are available in [120], while leveraging previously crowdsourced NLP datasets is a less labor-intensive curation approach [121, 71, 78, 53].”
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, background): “Compared to the above two main categories, alignment data can also be generated by transforming existing data [54, 45, 20].”
- Orthogonal Subspace Learning for Language Model Continual Learning (2023, methodology): “In contrast, O-LoRA employs instruction tuning (Wang et al., 2022b) Large number of tasks Our method’s performance on longer task sequences, posing a greater challenge, is evaluated through experiments on a continual learning benchmark of 15 datasets (Razdaibiedina et al., 2023).”