Learning from Task Descriptions

By Orion Weller, Nicholas Lourie, Matt Gardner, Matthew E. Peters. 2020

98 citations recorded by the source corpus.

Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.

Subfields: Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

Citations in the corpus

Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A series of works have found evidence that vanilla language models can be effective at following general language instructions if tuned with annotated “instructional” data – datasets containing language instructional commands and their desired outcome based on human judgement (Weller et al., 2020; M”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “Weller et al. (2020) construct a crowdsourced dataset with short question-like task descriptions.”
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021, background): “This is also remotely relevant to the literature on learning from instructions which expect a model to adapt its behavior according declarative instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021).”
A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, background): “…they can perform a variety of NLP tasks by expressing the task as a prompt that encourages the model to output the desired answer as the continuation (Puri and Catan-zaro, 2019; Weller et al., 2020; Brown et al., 2020; Schick and Schütze, 2021, inter alia ; see Liu et al. (2021a) for a survey).”
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “, 2020); (3) textual data can provide useful task information when serving as a model input, and hence this is a natural way to learn a prior over tasks (Brown et al., 2020; Weller et al., 2020) (4) retrieval is more scalable than conditioning on a global set of explanations, and (5) using explanati”
Generating Datasets with Pretrained Language Models (2021, methodology): “Providing PLMs with task descriptions for zero-or few-shot learning has been studied extensively (e.g., Radford et al., 2019; Puri and Catanzaro, 2019; Brown et al., 2020; Schick and Schütze, 2020, 2021b,a; Weller et al., 2020; Gao et al., 2021; Tam et al., 2021).”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “…motivated by building models that are generalizable across a variety of NLP tasks, when prompted with either a few examples (Ye and Ren, 2021; Bragg et al., 2021) or language deﬁnitions (Efrat and Levy, 2020; Weller et al., 2020; Zhong et al., 2021; Mishra et al., 2022b,a; Parmar et al., 2022).”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “Our approach is complementary to that of Weller et al. (2020), which asks turkers to generate tasks, and that of Mishra et al. (2021), which generates tasks by decomposing existing templates used to construct reading comprehension datasets.”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, methodology): “In the absence of training examples, textual class labels or richer task descriptions [73] must be provided.”
Competency Problems: On Finding and Removing Artifacts in Language Data (2021, background): “This could involve using images as prompts instead of text (Novikova et al., 2017; Weller et al., 2020), or randomly sampling words to include in the generated text.”
Few-Shot Text Generation with Pattern-Exploiting Training (2020, background): “…work, it would be interesting to see whether it is also possible to make pretrained language models understand more complex task descriptions than the simple prompts we have used, especially as some concurrent work (Weller et al., 2020; Efrat and Levy, 2020) suggests this might not be the case.”
How Many Data Samples is an Additional Instruction Worth? (2022, background): “Recently introduced instruction-paradigm empowers non-expert users, practitioners and domain experts in other fields to leverage NLP resources(Weller et al., 2020) as they now can describe their tasks in natural language”
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions (2021, background): “Weller et al. (2020) proposed a crowdsourced dataset with short question-like task descriptions.”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “Several works investigating the limitations of instruction-based few-shot approaches find that current LMs are mostly unable to understand complex instructions that go beyond short prompts or simple questions (Efrat and Levy, 2020; Weller et al., 2020; Webson and Pavlick, 2021) and that they are hig”
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources (2023, background): “Instruction Tuning of LMs Finetuning language models on diverse instruction sets alongside regular samples has been shown to greatly improve zero-shot performance on unseen tasks [34, 45, 43, 27, 9, 42], and serves as a good base for further finetuning in supervised settings [26].”
Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5) (2022, background): “[68] Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E Peters.”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “…models that are generalizable across a variety of language tasks, either when prompted with a few examples (Ye and Ren, 2021; Min et al., 2021; Bragg et al., 2021) or by language deﬁnitions (Efrat and Levy, 2020; Weller et al., 2020; Zhong et al., 2021; Mishra et al., 2022b,a; Xu et al., 2022).”
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture (2021, background): “The idea of specifying tasks through natural language descriptions is also being actively explored [59].”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, methodology): “…learning using human-annotated datasets (Ye et al., 2021; Wei et al., 2022; Sanh et al., 2022; Min et al., 2021; Xu et al., 2022) via instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021; Ouyang et al., 2022) and retrieved examples (Hu et al., 2022; Lin et al., 2022).”