How many data points is a prompt worth?

By Teven Le Scao, Alexander M. Rush. 2021

323 citations recorded by the source corpus.

When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Due to its simplicity, in this work we either employ simple prompts or simply feed in inputs without preprocessing, and perform full-model ﬁnetuning, a method similar to Le Scao & Rush (2021).”
OPT: Open Pre-trained Transformer Language Models (2022, background): “…and Schütze, 2020; 11https://huggingface.co/bigscience/ tr11-176B-ml-logs/tensorboard Gao et al., 2021b; Li and Liang, 2021; Lester et al., 2021; Scao and Rush, 2021), improving the flexibility of prompting (Shin et al., 2020), and understanding why and how prompting works (Liu et al., 2021;…”
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2021, methodology): “Prompt-Based Finetuning Rather than using frozen LMs, prompt-based ﬁnetuning methods ﬁnetune all of the LM’s parameters (Schick and Schütze, 2021a; Scao and Rush, 2021; Gao et al., 2021).”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “Our work is closely related to the recent literature in prompting LMs (Schick and Schütze, 2020; Le Scao and Rush, 2021; Reynolds and McDonell, 2021; Tam et al., 2021).”
Memory-assisted prompt editing to improve GPT-3 after deployment (2022, methodology): “Our use of recalled memories is a form of “prompt engineering”, where GPT -3’s behavior is modified by adding to the query (prompt) (Le Scao and Rush, 2021).”
Can language models learn from explanations in context? (2022, background): “Le Scao and Rush (2021) estimated that task prompts can be worth many examples, at least when training a classiﬁer.”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “Data augmentation [26, 109] Improving downstream tasks’ performance by introducing additional samples Data augmentation introduce additional training samples in an explicit way while prompts can be regarded as highlycondensed training samples [65].”
True Few-Shot Learning with Language Models (2021, background): “In other words, one reason that prompts are so effective [“worth many examples”; 15] is that they are often tuned using many examples.”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “Recent work has analyzed training with discrete prompts, demonstrating a boost in performance with prompting when training on various numbers of examples [71], finding that models perform similarly when trained on good and bad prompts [11], and exploring which prompts work well for few-shot and full”
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, neutral): “…contexts, prompting has been shown to have advantages over traditional classiﬁcation, for example facilitating adaptation of language models to ad-hoc tasks and improving sample efﬁciency in low-data settings (Brown et al., 2020; Schick and Schütze, 2021b; Le Scao and Rush, 2021; Gao et al., 2021).”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “Large language models - whose only training objective is to predict the next word given the context - have acquired a surprising ability to perform ZSC (Radford et al., 2019; Brown et al., 2020; Le Scao and Rush, 2021).”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “Discrete Prompts Constructing effective discrete prompts for language models to perform NLP tasks is an active area of research (Schick and Schütze, 2020; Le Scao and Rush, 2021; Tam et al., 2021; Logan IV et al., 2021).”
BARTScore: Evaluating Generated Text as Text Generation (2021, background): “This suggests a promising future direction for metric design: searching for proper prompts to better leverage knowledge stored in pre-trained language models instead of training on human judgment data [31].”