What Makes Good In-Context Examples for GPT-3?
By Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen. 2021
1,798 citations recorded by the source corpus.
GPT-3 has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its in-context learning abilities. Despite its success, we found that the empirical results of GPT-3 depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-3’s in-context learning capabilities.Inspired by the recent success of leveraging a retrieval module to augment neural networks, we propose to retrieve examples that are semantically-similar to a test query sample to formulate its corresponding prompt. Intuitively, the examples selected with such a strategy may serve as more informative inputs to unleash GPT-3’s power of text generation. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random selection baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (44.3% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset).
Subfields: Artificial Intelligence, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Graham Neubig
- Yejin Choi
- Noah A. Smith
- Hannaneh Hajishirzi
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Denny Zhou
- Swaroop Mishra
- Timo Schick
- Hannaneh Hajishirzi
- Hinrich Schütze
- Jie Huang
- Chitta Baral
Citations in the corpus
- OPT: Open Pre-trained Transformer Language Models (2022, background): “…eliciting prompting behavior in smaller models (Schick and Schütze, 2020; Gao et al., 2021b; Li and Liang, 2021; Lester et al., 2021; Scao and Rush, 2021), improving the flexi-bility of prompting (Shin et al., 2020), and understanding why and how prompting works (Liu et al., 2021; Min et al., 2022).”
- Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “Inspired by recent work on utilizing language models for data generation (Schick and Schütze, 2021b; Lee et al., 2021; Liu et al., 2022a), we collect data in a fully automatic manner by prompting a pretrained language model with three examples from the Super-Natural Instructions2 dataset (Mishra et…”
- ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “To generate both benign and toxic responses from LLMs that mention minority groups, we first collect many examples.”
- Memory-assisted prompt editing to improve GPT-3 after deployment (2022, background): “Recent work such as Liu et al. (2021a) investigate using dynamic prompts for better generation.”
- Large Language Models are Zero-Shot Reasoners (2022, background): “…exhibit properties conducive to few-shot learning [Brown et al., 2020], by way of in context learning, where one can use a text or template known as a prompt to strongly guide the generation to output answers for desired tasks, thus beginning an era of “pre-train and prompt” [Liu et al., 2021a].”
- PromptChainer: Chaining Large Language Model Prompts through Visual Programming (2022, background): “To invoke the desired functionalities, users need to write prompts [1, 11, 12] that are appropriate for the task.”
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, methodology): “…the problem (Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2021a), better ways of choosing labeled examples for the demonstrations (Liu et al., 2021; Lu et al., 2021; Rubin et al., 2021), meta-training with an explicit in-context learning objective (Chen et al., 2021; Min et al.,…”
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Various studies therefore focus on prompt engineering [9, 43, 45].”
- Rationale-Augmented Ensembles in Language Models (2022, background): “Another line of work proposes to better retrieve prompts closer to the target question to further improve task performance (Liu et al., 2022b; Rubin et al., 2021).”
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, neutral): “This approach bears resemblance to several recent works (Poesia et al., 2022; Gao et al., 2020; Liu et al., 2021; Rubin et al., 2021).”
- True Few-Shot Learning with Language Models (2021, background): “However, the few-shot performance of LMs is very sensitive to the textual task description [“prompt”; 3–6], order of training examples [6–8], decoding strategy [9, 10], and other hyperparameters [3, 5, 9, 11, 12], as well as the learning algorithm itself [3, 12].”
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “We also run captioning experiments both with (i) changing the wording of the LLM prompts (Kojima et al., 2022; Liu et al., 2021), and (ii) ablations that remove VLM categories.”
- GPT Understands, Too (2021, background): “GPT-3 (Brown et al., 2020) uses in-context examples (Liu et al., 2021; Zhao et al., 2021b) as a way of prompting to transfer knowledge from pretraining to downstream tasks.”
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “To better use these available examples, below, we explore two approaches: (i) improving the example quality by careful in-context example selection (Liu et al. 2021), and (ii) using more examples via multi-query ensemble.”
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Take multi-option question answering as an example, the standard prompt [35, 56, 28] decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp : M : { I i } n , I t → a, lect, exp .”
- Using In-Context Learning to Improve Dialogue Safety (2023, background): “Liu et al. (2022b)”