Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
By Timo Schick, Hinrich Schütze. 2020
1,862 citations recorded by the source corpus.
Some NLP tasks can be solved in a fully unsupervised fashion by providing a pretrained language model with “task descriptions” in natural language (e.g., Radford et al., 2019). While this approach underperforms its supervised counterpart, we show in this work that the two ideas can be combined: We introduce Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the resulting training set. For several tasks and languages, PET outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Hannaneh Hajishirzi
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Mohit Bansal
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Timo Schick
- Swaroop Mishra
- Hinrich Schütze
- Yejin Choi
- Mohit Bansal
- Chitta Baral
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, background): “…can also be accomplished by combining textual templates (“prompts”) and various forms of model finetuning, either fully updating a model’s parameters, e.g. for classification (Schick & Schütze, 2021a; Schick & Schutze, 2021; Gao et al., 2021; Tam et al., 2021) or generation (Schick & Schütze, 2021b).”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “…(Yarowsky, 1995), relation extraction (Brin, 1999; Agichtein and Gravano, 2000), parsing (McClosky et al., 2006), sequence generation (He et al., 2020), few-shot text classification (Schick and Schütze, 2021a) and retrieval (Izacard and Grave, 2021) to reasoning (Zelikman et al., 2022).”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “, 2020), by first conditioning on K example input-output demonstrations and then performing a cloze-style prompt completion task (Schick and Schütze, 2021), has emerged as a promising alternative to fine-tuning.”
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “Schick & Schütze (2021) (also see Gao et al., 2021; Tam et al., 2021) use task descriptions in cloze-style phrases to help language models assign soft labels for few-shot and semi-supervised learning, though this line of work finetunes new checkpoints for each downstream task.”
- Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2021, methodology): “The results for different few-shot learning meth-ods can be quite different across datasets and seeds for the training set (Zhao et al., 2021; Schick and Schütze, 2021a).”
- Improving and Simplifying Pattern Exploiting Training (2021, methodology): “We run experiments on SuperGLUE, and follow the same data split as Schick and Schütze (2021b), which consists of 32 labeled examples for each task.”
- How many data points is a prompt worth? (2021, methodology): “We further thank Timo Schick for making the code for PET available and for discussions about performance replication.”
- Few-Shot Text Generation with Natural Language Instructions (2020, methodology): “As recommended by Schick and Schütze (2021a), we train all models for 250 steps using a batch size of 8.”
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “, 2020] builds early exit “off ramps” in BERT to accelerate inference for test instances based on an entropy threshold; the work of Schwartz et al. [2020] implements the same idea with only minor difference in details.”
- It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “As finding ways to reformulate tasks as cloze questions that are understood well by LMs is diffi-cult (Jiang et al., 2019), Schick and Sch¨utze (2020a) propose P ET , a method that uses knowledge distillation (Hinton et al., 2015) to easily combine several reformulations.”
- Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “For example, AutoPrompt (Shin et al., 2020) searches for a sequence of discrete trigger words and concatenates it with each input to elicit sentiment or factual knowledge from BERT and RoBERTa.”
- Large Language Models Can Self-Improve (2022, neutral): “Language models are known to preserve parametric knowledge (Schick and Schütze, 2020a,b) during the pre-training stage.”
- Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “While these values have not been standardized, two prominent papers, Schick and Schütze (2021b) and Brown et al. (2020), use 32 and 64 prompt examples, respectively.”
- Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “Few-Shot Learning We study natural language prompts (Brown et al., 2020; Schick and Schütze, 2021a) to establish the first approach to few-shot self-rationalization.”
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “Prompting methods are particularly useful in the latter case [9, 32, 117], as there are generally not enough training examples to fully specify the desired behavior, and thus using a prompt to push the model in the right direction is particularly effective.”