Reordering Examples Helps during Priming-based Few-Shot Learning

By Sawan Kumar, Partha P. Talukdar. 2021

71 citations recorded by the source corpus.

The ability to learn from limited data, or few-shot learning, is a desirable and often critical requirement for NLP systems. While many existing methods do poorly at learning from a handful of examples, large pretrained language models have recently been shown to be efficient few-shot learners. One approach to few-shot learning, which does not require finetuning of model parameters, is to augment the language model's input with priming text which is typically constructed using task specific descriptions and examples. In this work, we further explore priming-based few-shot learning, with focus on using examples as prompts. We show that presenting examples in the right order is key for generalization. We introduce PERO (Prompting with Examples in the Right Order), where we formulate few-shot learning as search over the set of permutations of the training examples. We show that PERO can learn to generalize efficiently using as few as 10 examples, in contrast to existing approaches. While the newline token is a natural choice for separating the examples in the prompt, we show that learning a new separator token can potentially provide further gains in performance. We demonstrate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval. Finally, we analyze the learned prompts to reveal novel insights, including the idea that two training examples in the right order alone can provide competitive performance for sentiment classification and natural language inference.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “, 2022), the choice of few-shot examples, and the order in which they are presented (Kumar and Talukdar, 2021; Lu et al., 2022; Zhou et al., 2022).”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, background): “, 2022b) and when permuting the in-context demonstrations (Kumar and Talukdar, 2021; Lu et al., 2022).”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “We focus on instructional prompts, as opposed to exemplar prompts which are already well-studied (Lu et al., 2021; Zhao et al., 2021; Kumar and Talukdar, 2021).”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, background): “, 2021); as well as studies on the effect of example order (Kumar and Talukdar, 2021; Lu et al., 2022).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “As for sample ordering , Kumar and Talukdar [83] searched for the best permutation of prompts and also learned a segmentation token to separate the prompts.”
Co-training Improves Prompt-based Learning for Large Language Models (2022, background): “While intriguing, these methods can be sensitive to trivial cosmetic artifacts, including variations in prompt wording and the ordering of examples (Lu et al., 2021; Zhao et al., 2021; Kumar & Talukdar, 2021).”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, methodology): “Instruction-based learning has also been used in few-shot settings; popular variants include in-context learning, where the model’s parameters are fixed and examples are provided as additional context (Brown et al., 2020; Lu et al., 2021; Kumar and Talukdar, 2021; Zhao et al., 2021; Min et al., 2021”