True Few-Shot Learning with Prompts—A Real-World Perspective

By Timo Schick, Hinrich Schütze. 2021

87 citations recorded by the source corpus.

Abstract Prompt-based approaches excel at few-shot learning. However, Perez et al. (2021) recently cast doubt on their performance as they had difficulty getting good results in a “true” few-shot setting in which prompts and hyperparameters cannot be tuned on a dev set. In view of this, we conduct an extensive study of Pet, a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, Pet performs strongly in true few-shot settings without a dev set. Crucial for this strong performance is a number of design choices, including Pet’s ability to intelligently handle multiple prompts. We put our findings to a real-world test by running Pet on RAFT, a benchmark of tasks taken from realistic NLP applications for which no labeled dev or test sets are available. Pet achieves a new state of the art on RAFT and performs close to non-expert humans for 7 out of 11 tasks. These results demonstrate that prompt-based learners can successfully be applied in true few-shot settings and underpin our belief that learning from instructions will play an important role on the path towards human-like few-shot learning capabilities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

In Meta & Continual Learning

Citations in the corpus

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “The next-best model (from Schick and Schütze [50]) achieves 6% lower accuracy and GPT-3 175B attains only 62.”
Co-training Improves Prompt-based Learning for Large Language Models (2022, background): “While we are encouraged by the observation that model conﬁgurations seem to work well across diverse datasets, investigating co-training in the context of true few-shot/zero-shot learning (Schick & Sch¨utze, 2021) is an important avenue for future work.”
Efficient Few-Shot Learning Without Prompts (2022, methodology): “paraphrase-MiniLM-L3-v2 (Schick and Schütze, 2021), the authors show that PET-based classification methods excel on the RAFT benchmark, placing second only to much larger models such as T-FEW.”
BioTABQA: Instruction Learning for Biomedical Table Question Answering (2022, result): “The reason might be that instructions for the different tasks have a set of similar words (see Appendix A), and previous studies [44, 45] have shown that the model can perform well if the instructions have similar words.”