Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

By Robert L Logan IV, Ivana Balavzevi'c, Eric Wallace, Fabio Petroni, Sameer Singh, Sebastian Riedel. 2021

245 citations recorded by the source corpus.

Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, background): “, 2020) or by only updating parts of the model (Logan et al., 2021), or learning “soft-prompts” (Lester et al.”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “Example template We follow Ye et al. (2021); Min et al. (2021b); Logan IV et al. (2021) in us-ing the minimal format to transform the input to a sequence (e.g. a concatenation of multiple inputs) and using the label words from each dataset as it is.”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, methodology): “We relax the assumption from prior work of an equal number of training examples per label (Gao et al., 2021; Logan IV et al., 2021), for more realistic and challenging evaluation.”
OpenPrompt: An Open-source Framework for Prompt-learning (2021, methodology): “Logan IV et al. (2021) design null template with simple concatenation of the inputs and an appended <mask> token.”
Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation (2021, background): “Such robustness to prompt perturbation comes from not only the all-parameter training (§4.2) (Logan IV et al., 2021) but also the huge output space of GDG tasks (compared to NLU tasks).”
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, background): “Speciﬁcally, we consider this as null-prompting (Logan et al., 2021), where using the mask token as the prompt can achieve competitive results with manually engineered prompts.”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, background): “In particular, ﬁnetuning is still very important to improve the downstream performance (IV et al., 2021).”
Do Prompts Solve NLP Tasks Using Natural Language? (2022, methodology): “Null prompts (Logan IV et al., 2021) are the most succinct version, directly adding a masked token to the end of the input.”
A Review on Language Models as Knowledge Bases (2022, background): “On the other hand, Logan et al. (2021) takes a more manual approach to crafting prompts with comparable accuracy to manual prompts by creating null prompts, those of which are task-agnostic and are a simple concatenation of the input and a [MASK] token.”