Reframing Instructional Prompts to GPTk’s Language

By Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, Hannaneh Hajishirzi. 2021

277 citations recorded by the source corpus.

What kinds of instructional prompts are easier to follow for Language Models (LMs)? We study this question by conducting extensive empirical analysis that shed light on important features of successful instructional prompts. Specifically, we study several classes of reframing techniques for manual reformulation of prompts into more effective ones. Some examples include decomposing a complex task instruction into multiple simpler tasks or itemizing instructions into sequential steps. Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories. Compared with original instructions, our reframed instructions lead to significant improvements across LMs with different sizes. For example, the same reframed prompts boost few-shot performance of GPT3-series and GPT2-series by 12.5% and 6.7% respectively averaged over all tasks. Furthermore, reframed instructions reduce the number of examples required to prompt LMs in the few-shot setting. We hope these empirically-driven techniques will pave the way towards more effective future prompting algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Can language models learn from explanations in context? (2022, background): “In-context explanations and reasoning decomposition: Mishra et al. (2021b) explore prompts that contain task descriptions and examples with explanations (as well as other features such as guidelines).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “At the same time, some work reports brittleness and over-sensitivity for in-context learning (Lu et al., 2021; Zhao et al., 2021; Mishra et al., 2021a).”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, methodology): “…motivated by building models that are generalizable across a variety of NLP tasks, when prompted with either a few examples (Ye and Ren, 2021; Bragg et al., 2021) or language deﬁnitions (Efrat and Levy, 2020; Weller et al., 2020; Zhong et al., 2021; Mishra et al., 2022b,a; Parmar et al., 2022).”
MetaICL: Learning to Learn In Context (2021, background): “…(e.g., a question answering format) (Khashabi et al., 2020; Zhong et al., 2021), or rely heavily on task-speciﬁc templates (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022) which are difﬁcult to engineer due to high variance in performance from very small changes (Mishra et al., 2021).”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “We also include an example of a multi-step factored cognition prompt [32], see Figure 6, where a challenging task is broken down into atomic steps.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “For example, the learning from instruction paradigm [40, 43, 53, 39, 45, 25], where the task level explanation is provided in the form of instruction, improves model performance significantly.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, background): “Here we do not input any context when presenting a question to the LM. Talmor et al. (2020) tested masked LMs on 2-hop questions.”
Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach (2022, methodology): “Previous work along this direction mainly use manually designed prompts/templates suited for specific datasets (Paranjape et al., 2021; Mishra et al., 2021; Shwartz et al., 2020; Wei et al., 2022); instead, we seek to develop a general method which can learn to construct appropriate prompts…”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “Moreover, Mishra et al. (2021b) proposed Natural Instructions which break down each task to multiple sub-tasks that help models in following instructions and subsequently generalize to unseen tasks (i.e., cross-task generalization).”
Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022, methodology): “Previous work along this direction mainly use manually designed prompts/templates suited for speciﬁc datasets (Paranjape et al., 2021; Mishra et al., 2021; Shwartz et al., 2020); instead, we seek to develop a general method which can learn to construct appropriate prompts automatically.”
Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, background): “More related to us, [33, 20] reframe prompts by decomposing a complex task instruction into multiple simple ones.”
Large language models are few-shot clinical information extractors (2022, background): “, multiple choice), more complex tasks require creativity in their implementation (Mishra et al., 2021).”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Since Brown et al. [10] demonstrated the in-context learning ability of GPT-3, there has been a significant interest in improving and understanding this capability [36, 39, 79, 38, 59, 40, 14, 43, 33].”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, methodology): “…et al., 2021; Webson and Pavlick, 2021; Lu et al., 2021), a number of existing works seek to locate better prompts by augmenting human-written prompts with heuristics such as paraphrasing (Jiang et al., 2020; Gao et al., 2021), editing (Prasad et al., 2022), and re-framing (Mishra et al., 2021a).”
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2023, background): “The larger LMs become, the easier they are to steer with natural language prompts (Kandpal et al., 2022; Carlini et al., 2023; Mishra et al., 2022a; Srivas-tava et al., 2023).”
HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models (2022, background): “, 2022b), reframing (Mishra et al., 2022a), least-tomost prompting (Zhou et al.”
Continual-T0: Progressively Instructing 50+ Tasks to Language Models Without Forgetting (2022, background): “Instruction tuning There has been a range of work in the domain of instruction-tuning (Mishra et al., 2021b; Sanh et al., 2022; Wei et al., 2022; Mishra et al., 2021a; Ouyang et al., 2022) which differs in training and evaluation data, formatting of instructions, size of pre-trained models, and…”
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2022, background): “, 2022b], or suggests prompt reframing techniques [Mishra et al., 2021] and prompt retrieval methods [Rubin et al.”