Prompt-free and Efficient Few-shot Learning with Language Models
By Rabeeh Karimi Mahabadi, Luke S. Zettlemoyer, James Henderson, Marzieh Saeidi, Lambert Mathias, Ves Stoyanov, Majid Yazdani. 2022
76 citations recorded by the source corpus.
Current methods for few-shot fine-tuning of pretrained masked language models (PLMs) require carefully engineered prompts and verbalizers for each new task to convert examples into a cloze-format that the PLM can score. In this work, we propose Perfect, a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting, which is highly effective given as few as 32 data points. Perfect makes two key design choices: First, we show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning and reduce memory and storage costs by roughly factors of 5 and 100, respectively. Second, instead of using handcrafted verbalizers, we learn new multi-token label embeddings during fine-tuning, which are not tied to the model vocabulary and which allow us to avoid complex auto-regressive decoding. These embeddings are not only learnable from limited data but also enable nearly 100x faster training and inference. Experiments on a wide range of few shot NLP tasks demonstrate that Perfect, while being simple and efficient, also outperforms existing state-of-the-art few-shot learning methods. Our code is publicly available at https://github.com/rabeehk/perfect.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Cordelia Schmid
- Ivan Laptev
- Josef Sivic
- Andrew Zisserman
- Antoine Miech
- Caiming Xiong
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Cordelia Schmid
- Mohit Bansal
- Luc Van Gool
- Andrew Zisserman
- Trevor Darrell
- Ivan Laptev
- Dacheng Tao
- Jitendra Malik
In Large Language Models
- Mohit Bansal
- Derek Tam
- Ivan Laptev
- Josef Sivic
- Antoine Miech
- Antoine Yang
- Moshe Wasserblat
- Colin Raffel
In Meta & Continual Learning
- Mohit Bansal
- Cordelia Schmid
- Tyler L. Hayes
- Christopher Kanan
- Pietro Zanuttigh
- Umberto Michieli
- Ali Ayub
- Rahaf Aljundi
Citations in the corpus
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “[76] find that PEFT compares favorably in few-shot fine-tuning against discrete prompts (e.”
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[72] Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyanov, and Majid Yazdani.”
- Efficient Few-Shot Learning Without Prompts (2022, methodology): “In recent work P ERFECT P ERFECT (Karimi Mahabadi et al., 2022b) is another cloze-based fine-tuning method, but unlike PET or A DAPET , it does not require handcrafted task prompts and verbalizers.”