Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
By Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig. 2021
5,562 citations recorded by the source corpus.
This article surveys and organizes research works in a new paradigm in natural language processing, which we dub “prompt-based learning.” Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x′ that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x̂, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: It allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this article, we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g., the choice of pre-trained language models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts but also release other resources, e.g., a website NLPedia–Pretrain including constantly updated survey and paperlist.
Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Yejin Choi
- Quoc V. Le
- Yoshua Bengio
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Caiming Xiong
- Richard Socher
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Swaroop Mishra
- Chitta Baral
- Jason Wei
- Percy Liang
- Quoc V. Le
- Hannaneh Hajishirzi
- Hongsheng Li
Citations in the corpus
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “The recent success of large-scale language models has led to growing interest in improving their capability to perform tasks via prompting (Brown et al. (2020), and see Liu et al. (2021) for a survey).”
- Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Though these methods are very similar to ours, they mostly focus on multi-task learning instead of zero-shot learning, and—as noted by Liu et al. (2021)—they are generally not motivated by using existing knowledge in pretrained LMs.”
- ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “However, designing prompts is notoriously challenging (Liu et al., 2021c).”
- Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “…if we remain within the few-shot paradigm, more accurate prompt optimization, like in-context example selection, can further boost results [23], while, constrained decoding [35, 36] can be a way to condition the model at inference-time, by explicitly grounding an answer in the provided…”
- Memory-assisted prompt editing to improve GPT-3 after deployment (2022, background): “Recent work such as Liu et al. (2021a) investigate using dynamic prompts for better generation.”
- Large Language Models are Zero-Shot Reasoners (2022, methodology): “…is versatile and task-agnostic , unlike most prior task-specific prompt engineering in the forms of examples (few-shot) or templates (zero-shot) [Liu et al., 2021b]: it can facilitate step-by-step answers across various reasoning tasks, including arithmetic (MultiArith [Roy and Roth, 2015],…”
- Can language models learn from explanations in context? (2022, methodology): “Although it is not always clear what is being learned or inferred from these prompts (e.g. Min et al., 2022; Webson and Pavlick, 2021), prompting is a growing subfield (Liu et al., 2021).”
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “These features can help facilitate various downstream applications, through a “pre-train, prompt, and predict” paradigm [45]: rather than having to train a new model for every custom task, users can just customize the model by feeding it a natural language prompt at run time.”
- A Recipe for Arbitrary Text Style Transfer with Large Language Models (2021, background): “…they can perform a variety of NLP tasks by expressing the task as a prompt that encourages the model to output the desired answer as the continuation (Puri and Catan-zaro, 2019; Weller et al., 2020; Brown et al., 2020; Schick and Schütze, 2021, inter alia ; see Liu et al. (2021a) for a survey).”
- PAL: Program-aided Language Models (2022, background): “Few-shot prompting leverages the strength of large-language models to solve a task with a set of k examples that are provided as part of the test-time input (Brown et al., 2020; Liu et al., 2021; Chowdhery et al., 2022), where k is usually a number in the low single digits.”
- Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “Invasive meth-ods, which are built on a strong assumption that the inner structure (e.g., self-attention and feed-forward layers) of the PLM can be modified, includes Prefix-Tuning (Li and Liang, 2021), Bit-fit (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), L”
- PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, background): “Prompts and prompt engineering come in several varieties (Liu et al., 2021).”
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “As an alternative solution to fine-tuning, prompt tuning has recently been proposed in the NLP community, which originated from prompting methods [41].”
- Learning to Prompt for Vision-Language Models (2021, background): “…language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “Many algorithms have been developed for various efficiency considerations, including adapters [23] and prompt tuning [35, 37].”
- Visual Prompt Tuning (2022, methodology): “Transformer models [73] have gained huge success in NLP [17,66,7].”