Learning To Retrieve Prompts for In-Context Learning

By Ohad Rubin, Jonathan Herzig, Jonathan Berant. 2021

945 citations recorded by the source corpus.

In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters. However, performance has been shown to strongly depend on the selected training examples (termed prompts). In this work, we propose an efficient method for retrieving prompts for in-context learning using annotated data and an LM. Given an input-output pair, we estimate the probability of the output given the input and a candidate training example as the prompt, and label training examples as positive or negative based on this probability. We then train an efficient dense retriever from this data, which is used to retrieve training examples as prompts at test time. We evaluate our approach on three sequence-to-sequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

Citations in the corpus

Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “While perhaps the mainstream view places scaling models’ parameters as the primary way to increase their few-shot performance, our results add to the stream of work that emphasizes instead better use of the models’ powerful prompting abilities (Rubin et al., 2021; Liu et al., 2021a).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “…2021; Holtzman et al., 2021; Min et al., 2021a), better ways of choosing labeled examples for the demonstrations (Liu et al., 2021; Lu et al., 2021; Rubin et al., 2021), meta-training with an explicit in-context learning objective (Chen et al., 2021; Min et al., 2021b), and learning to follow…”
Rationale-Augmented Ensembles in Language Models (2022, background): “Another line of work proposes to better retrieve prompts closer to the target question to further improve task performance (Liu et al., 2022b; Rubin et al., 2021).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, neutral): “This approach bears resemblance to several recent works (Poesia et al., 2022; Gao et al., 2020; Liu et al., 2021; Rubin et al., 2021).”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “Notably, Rubin et al. (2022) retrieved the semantically similar demonstrations with the test instance.”
A Survey of Large Language Models (2023, neutral): “Besides, EPR [226] proposes a two-stage retrieval approach that first recalls similar examples with an unsupervised method (e.”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “This framework has attracted attention especially in few-shot natural language processing tasks [11, 20, 50].”
A Survey of Deep Learning for Mathematical Reasoning (2022, neutral): “To address the limitation, recent work has investigated various methods to optimize the in-context examples selection process (Rubin et al., 2022; Zhang et al., 2023; Lu et al., 2022b; Yu et al., 2023; Fu et al., 2023).”
Selective Annotation Makes Language Models Better Few-Shot Learners (2022, background): “Although in-context learning was originally proposed for few-shot learning, recent works show that retrieving prompts from a large set of annotated examples is necessary to achieve good performances (Liu et al., 2022; Rubin et al., 2022).”
Recommender Systems in the Era of Large Language Models (LLMs) (2023, neutral): “Several works have explored the utilization of ICL in various tasks, such as SG-ICL [46] and EPR [47].”
What does a platypus look like? Generating customized prompts for zero-shot image classification (2022, background): “Additional works have proposed learning prompts via gradient based methods [65, 41, 29, 28, 51], retrieval from a database [46], or reformatting/rephrasing existing prompts [24, 46].”
Complexity-Based Prompting for Multi-Step Reasoning (2022, background): “retrieval from a large training set (Rubin et al., 2022).”
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2022, background): “, 2020), researchers have been studying prompt engineering for text-to-text generation (e.g., Liu et al., 2022; Lu et al., 2022; Rubin et al., 2022).”
A Survey for In-context Learning (2023, methodology): “Supervised Method Rubin et al. (2022) proposed a two-stage retrieval method to select demonstrations.”
Reasoning with Language Model Prompting: A Survey (2022, background): “Recent works show that retrieving prompts for incontext learning is a nice means to achieve good performance (Liu et al., 2022a; Rubin et al., 2022).”
Learning Performance-Improving Code Edits (2023, background): “Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”