True Few-Shot Learning with Language Models

By Ethan Perez, Douwe Kiela, Kyunghyun Cho. 2021

532 citations recorded by the source corpus.

Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “We call this the true zero-shot setting, following the definition of “true few-shot learning” in [32].”
OPT: Open Pre-trained Transformer Language Models (2022, background): “Furthermore, it is challenging to write prompts without a development set, which leads to questions about the extent to which we are actually achieving zero- or few-shot learning in practice (Perez et al., 2021).”
Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “Because we have optimized the instructions using a small portion of the question and answer pairs as training demonstrations, our results are not “true few-shot learning” (Perez et al., 2021).”
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2021, methodology): “As a consequence, prompts are either designed using human intuition that is hard to replicate and apply in a principled manner (Perez et al., 2021), or using automated methods (Shin et al.”
Red Teaming Language Models with Language Models (2022, background): “Designing an effective prompt can be hard (Perez et al., 2021), but simple one-sentence prompts were effective at generating the kinds of test cases we desired (e.g., about a certain topic).”
Can language models learn from explanations in context? (2022, background): ““Few-shot” performance results can be misleading if many more examples are used to tune the prompt (Perez et al., 2021).”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, methodology): “Following Perez et al. (2021), we avoid prompt tuning on the full training and development sets of the datasets studied, in order to ensure that our methods represent a true few-shot setting.”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, background): “4 Zero/One-Shot Prompting Perez et al. (2021) argue that zero/few-shot learning cannot happen when prompts are created by tuning on a large amount of development data.”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “However, the quality of LM outputs is sensitive to text inputs in unintuitive ways (Perez et al., 2021; Lu et al., 2022), adding hard-to-predict variance to the quality of the resulting evaluation; see Appendix §A.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “The most strict few-shot learning setting does not assume access to any unlabeled data or large validation sets for hyperparameter tuning [48], where prompt-based methods [5, 13, 33, 35, 55–57, 59, 63, 84] are prominently deployed to inject task descriptions into PLMs and make effective use of their”
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (2021, background): “However, this is not desirable in a few-shot setting where the size of the development set is very limited, even unavailable (Perez et al., 2021) .”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “However, in-context learning is known to be highly sensitive to various aspects of the demonstrations [80, 148], and its inference compute cost and absolute performance scale poorly with the number of shots beyond this low-data regime.”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “They also require huge models to work well but choosing good prompts becomes harder with larger model sizes [9].”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “This makes the recipe a realistic option for few-shot learning settings where validation sets are tiny by definition [31, 32].”
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, background): “The process of prompt engineering is critical for successful deployment as choices in prompting can affect downstream predictions signiﬁcantly, particularly in the zero-shot setting (Perez et al., 2021; Zhao et al., 2021; Webson and Pavlick, 2021).”
MetaICL: Learning to Learn In Context (2021, methodology): “Moreover, it can have high variance and poor worst-case accuracy (Perez et al., 2021; Lu et al., 2021).”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, background): “Recently, there has been a surge of interest in few-shot learning, where language models learn a specific task from a few examples [46, 3].”
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021, background): “, 2021), though most assume access to a validation set (Perez et al., 2021).”