Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions

By Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh Hajishirzi. 2021

54 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Most recently, Mishra et al. (2021) finetune 140M parameter BART on instructions with few-shot exemplars, and evaluate its few-shot abilities on unseen tasks—this is very similar to our few-shot instruction tuning result from §4.3.”
Augmented Language Models: a Survey (2023, methodology): “For the latter, examples of reasoning are typically used in the larger context of instruction tuning (Mishra et al., 2021; Sanh et al., 2022; Wang et al., 2022d; Ouyang et al., 2022), where, more generally, an LM’s ability to follow instructions is improved based on human-labeled examples.”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “While there are several approaches for prompting pretrained LLMs (Liu et al., 2021b), a recent and promising direction is demonstration-based prompting (Gao et al., 2021; Mishra et al., 2021).”
Memory-assisted prompt editing to improve GPT-3 after deployment (2022, background): “Learning from instruction has become important for large LMs that can perform a task based on direct instruction rather than examples (Wei et al., 2021; Mishra et al., 2021).”
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021, background): “This is also remotely relevant to the literature on learning from instructions which expect a model to adapt its behavior according declarative instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021).”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “Our approach is complementary to that of Weller et al. (2020), which asks turkers to generate tasks, and that of Mishra et al. (2021), which generates tasks by decomposing existing templates used to construct reading comprehension datasets.”
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2021, background): “At a high-level, our ﬁndings contradict Mishra et al. (2021)’s claim that models beneﬁt from elaborate instructions adapted from crowdsourcing annotation guides.”
Reframing Instructional Prompts to GPTk’s Language (2021, methodology): “On the other hand, task instructions written by non-expert users are often long and contain abstractive descriptions (Mishra et al., 2021) which”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “This motivates researchers to investigate better ways to automatically search and/or manually construct a proper prompt (Jiang et al., 2020a; Shin et al., 2020; Reynolds and McDonell, 2021; Mishra et al., 2021b).”
Learning to Generate Task-Specific Adapters from Task Description (2021, background): “, 2020) and instructions (Mishra et al., 2021) can be considered as a sub-category in zeroshot learning, with the goal of generalizing to unseen tasks during inference.”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “After discussing general observations on CFT in Section 4.3.1, we will delve into each sub-category in detail.”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, background): “Following Peng et al. (2020b); Kale and Rastogi (2020b), the “Naive” canonical representation”
RAFT: A Real-World Few-Shot Text Classification Benchmark (2021, background): “Existing few-shot classification benchmarks are typically designed to highlight areas where models fall short [29] or to study particular model abilities [5, 37, 21].”
PandaGPT: One Model To Instruction-Follow Them All (2023, methodology): “With alignment techniques such as supervised instruction tuning [13, 21, 28] and reinforcement learning from human feedback [16, 23], LLMs exhibit surprisingly effective zero-and few-shot generalization abilities to perform almost any NLP tasks.”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “Most recent work (Mishra et al., 2021; Sanh et al., 2022; Wei et al., 2022) on instruction tuning has focused on general NLP tasks such as para-phrase detection and reading comprehension, but not specifically on dialogue.”
LIMA: Less Is More for Alignment (2023, methodology): “To enable this transfer, various methods for aligning language models have thus been proposed, primarily focusing on instruction tuning [Mishra et al., 2021, Wei et al., 2022a, Sanh et al., 2022] over large multi-million-example datasets [Chung et al., 2022, Beeching et al., 2023, Köpf et al.,…”