Cross-Task Generalization via Natural Language Crowdsourcing Instructions

By Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh Hajishirzi. 2021

909 citations recorded by the source corpus.

Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.

Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “There has been a range of work in this domain (Yi et al., 2019; Mishra et al., 2021; Wei et al., 2021; Khashabi et al., 2020; Sanh et al., 2021; Aribandi et al., 2021), which differ in training and evaluation data, formatting of instructions, size of pretrained models, and other experimental…”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and humanwritten completions [21, 33, 13, 36].”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “The Second Wave A second wave of instruction tuning collections expanded prior resources: combining more datasets and tasks into one resource, like SuperNatural Instructions (Wang et al., 2022c) or OPT-IML (Iyer et al., 2022), adding multilingual instruction tuning in xP3 (Muennighoff et al., 2022),”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “One way to collect examples of instructions and their execution is to reformulate existing NLP datasets in an explicit instruction-input-output format via prompt engineering (Mishra et al., 2022; Wang et al., 2022).”
Scaling Instruction-Finetuned Language Models (2022, background): “, 2022a) or few-shot with exemplars (Ye et al., 2021; Wei et al., 2021; Mishra et al., 2021; Min et al., 2022; Wang et al., 2022c), our process finetunes on a mixture of both formats to enable both zero-shot and few-shot.”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “• Instruction-following benchmarks, such as Flan [23, 38], Self-instruct [36], NaturalInstructions [24], Super-NaturalInstructions [37], expand to slightly more open-ended questions and more diverse tasks and are used to evaluate LLMs after instruction fine-tuning.”
Can language models learn from explanations in context? (2022, background): “In-context explanations and reasoning decomposition: Mishra et al. (2021b) explore prompts that contain task descriptions and examples with explanations (as well as other features such as guidelines).”
PromptChainer: Chaining Large Language Model Prompts through Visual Programming (2022, neutral): “Beyond this toy example, non-ML experts have used prompting to achieve various ML functionalities in real-time, including code generation, question answering, creative writing, etc. [3, 13, 15].”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “At the same time, some work reports brittleness and over-sensitivity for in-context learning (Lu et al., 2021; Zhao et al., 2021; Mishra et al., 2021a).”
AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Pretrained on billions of inputs from the Internet [29], generative models like GPT-3 can now perform a wide variety of tasks [10], ranging from translation [12], to question answering [47], and even advanced story writing [60].”
Learning by Distilling Context (2022, background): “Many recent works show that language models can learn from abstract task deﬁnitions (Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2022a; Sanh et al., 2022), natural language explanations (Scheurer et al., 2022; Wang et al., 2022b), and concrete in-context examples (Min et al., 2022; Chen et…”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “The current mainstream of zero-shot learning is based on transfer learning: By converting a set of tasks with abundant annotations into instruction templates (Mishra et al., 2021; Sanh et al., 2021; Wei et al., 2022; Xu et al., 2022), entailment pairs (Yin et al.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, methodology): “The NLP community has witnessed great progress in building models for generalization to unseen tasks via in-context instructions (Mishra et al., 1SUPER-NATURALINSTRUCTIONS represents a supersized expansion of NATURALINSTRUCTIONS (Mishra et al., 2022b) which had 61 tasks.”
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, background): “Furthermore, training directly on collections of prompts can enable large models to generalize to new prompts more robustly (Sanh et al., 2021; Wei et al., 2021; Min et al., 2021; Mishra et al., 2021).”
MetaICL: Learning to Learn In Context (2021, methodology): “…(e.g., a question answering format) (Khashabi et al., 2020; Zhong et al., 2021), or rely heavily on task-speciﬁc templates (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022) which are difﬁcult to engineer due to high variance in performance from very small changes (Mishra et al., 2021).”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, methodology): “(Mishra et al., 2021) built a dataset with task instructions, and CROSSFIT (Ye et al.”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Our goal to follow written image editing instructions is inspired by recent work teaching large language models to better follow human instructions for language tasks [39, 43, 68].”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “For example, the learning from instruction paradigm [39, 42, 52, 38, 44, 24], where the task level explanation is provided in the form of instruction, improves model performance signiﬁcantly.”