Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
By Laria Reynolds, Kyle McDonell. 2021
1,354 citations recorded by the source corpus.
Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe models’ novel capabilities. Using GPT-3 as a case study, we show that 0-shot prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking the role of prompts in controlling and evaluating powerful language models. We discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We explore techniques for exploiting the capacity of narratives and cultural anchors to encode nuanced intentions and techniques for encouraging deconstruction of a problem into components before producing a verdict. Informed by this more encompassing theory of prompt programming, we also introduce the idea of a metaprompt that seeds the model to generate its own natural language prompts for a range of tasks. Finally, we discuss how these more general methods of interacting with language models can be incorporated into existing and future benchmarks and practical applications.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Jianfeng Gao
- Graham Neubig
- Quoc V. Le
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Bernt Schiele
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Hinrich Schütze
- Hannaneh Hajishirzi
- Yi Tay
- Noah A. Smith
- Graham Neubig
- Percy Liang
In Prompting & Reasoning
- Denny Zhou
- Luke S. Zettlemoyer
- Swaroop Mishra
- Jason Wei
- Yejin Choi
- Xinyun Chen
- Hannaneh Hajishirzi
- Chitta Baral
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, positive): “We found that in most cases, using the language names to induce translation (0-shot setting) provided stronger performance than only using input-output exemplars (1-shot and few-shot settings), in line with previous work (Reynolds & McDonell, 2021).”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “Although there is variance among different chain of thought annotations, as would be expected when using exemplar-based prompting (Le Scao and Rush, 2021; Reynolds and McDonell, 2021; Zhao et al., 2021), all sets of chain of thought prompts outperform the standard baseline by a large margin.”
- Emergent Abilities of Large Language Models (2022, positive): “Augmenting few-shot exemplars with intermediate steps (Reynolds & McDonell, 2021; Nye et al., 2021; Wei et al., 2022b) has also enabled models to perform multi-step reasoning tasks not possible in the standard prompting formulation from Brown et al. (2020).”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “…et al., 2022; Ziegler et al., 2019), in-context learning (Brown et al., 2020), and several forms of prompt generation (Gao, 2021), including both differentiable tuning of soft prompts (Qin & Eisner, 2021; Lester et al., 2021) and natural language prompt engineering (Reynolds & McDonell, 2021).”
- Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, neutral): “We build models using pre-trained LMs with encoder-decoder architectures (GPT3 (Brown et al., 2020) and BART (Lewis et al., 2019)).”
- Large Language Models are Zero-Shot Reasoners (2022, methodology): “While few-shot prompts are naturally so due to task-specific in-context samples [Brown et al., 2020, Wei et al., 2022], majority of zero-shot prompts have also focused on per-task engineering (of templates) [Liu et al., 2021b, Reynolds and McDonell, 2021].”
- Can language models learn from explanations in context? (2022, background): “Other researchers have questioned whether what occurs is really “learning,” or simply recovering previously-experienced tasks (Reynolds and McDonell, 2021).”
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “This is in line with Reynolds and McDonell (2021) who claim that the demonstrations are for task location and the intrinsic ability to perform the task is obtained at pretraining time.”
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “To overcome users’ own systematic omissions [74], an upstream unit could be designed to help users create sub-tasks to begin with, similar to metaprompting [58].”
- A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, methodology): “Reynolds and McDonell (2021) find that prompting a model for a task is more akin to locating an already-learned task than truly learning a new one.”
- Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021, background): “Efficiently discovering the right prompt is difficult and has become an active area of research (Reynolds and McDonell, 2021; Shin et al., 2020; Jiang et al., 2020).”
- Reframing Instructional Prompts to GPTk’s Language (2021, background): “Discrete Prompts Constructing effective discrete prompts for language models to perform NLP tasks is an active area of research (Schick and Schütze, 2021; Le Scao and Rush, 2021; Tam et al., 2021; Logan IV et al., 2021; Reynolds and McDonell, 2021).”
- What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, background): “As the scale of language models grows, the potential of replacing the full fine-tuning paradigm with the prompt-based approach has been reported (Reynolds and McDonell, 2021; Li and Liang, 2021), as learning via prompts is ef-ficient regarding time and space complexity.”
- BARTScore: Evaluating Generated Text as Text Generation (2021, background): “Prompting is a practice of adding short phrases to the input or output to encouraged pre-trained models to perform specific tasks, which has been proven effective in several other NLP scenarios [19,48,49,51,53].”
- The Creativity of Text-to-Image Generation (2022, background): “For instance, OpenAI maintains a collection of input templates for use with GPT-3 [34] and similar resources and tools are emerging in the text-to-image art communities.”
- Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction (2022, background): “Finding the best query to accomplish the given task is known as prompt engineering [22].”