Improving and Simplifying Pattern Exploiting Training

By Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, Colin Raffel. 2021

157 citations recorded by the source corpus.

Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few-shot learning without any unlabeled data and introduce ADAPET, which modifies PET’s objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, background): “…can also be accomplished by combining textual templates (“prompts”) and various forms of model ﬁnetuning, either fully updating a model’s parameters, e.g. for classiﬁcation (Schick & Schütze, 2021a; Schick & Schutze, 2021; Gao et al., 2021; Tam et al., 2021) or generation (Schick & Schütze, 2021b).”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Schick & Schütze (2021) (also see Gao et al., 2021; Tam et al., 2021) use task descriptions in cloze-style phrases to help language models assign soft labels for few-shot and semi-supervised learning, though this line of work finetunes new checkpoints for each downstream task.”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, positive): “We build models using pre-trained LMs with encoder-decoder architectures (GPT3 (Brown et al., 2020) and BART (Lewis et al., 2019)).”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, background): “Follow-up work (Tam et al., 2021) improves upon these results without depending on unlabeled data.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Other work has shown that good few-shot performance can also be obtained by combining gradient based optimization with textual prompts (Schick and Sch¨utze, 2021b; Gao et al., 2021; Tam et al., 2021; Wang et al., 2021).”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, methodology): “…assume access to any unlabeled data or large validation sets for hyperparameter tuning [48], where prompt-based methods [5, 13, 33, 35, 55–57, 59, 63, 84] are prominently deployed to inject task descriptions into PLMs and make effective use of their language modeling capability for improved…”
True Few-Shot Learning with Language Models (2021, methodology): “However, the few-shot performance of LMs is very sensitive to the textual task description [“prompt”; 3–6], order of training examples [6–8], decoding strategy [9, 10], and other hyperparameters [3, 5, 9, 11, 12], as well as the learning algorithm itself [3, 12].”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “To improve performance on classification and multiple-choice tasks, we add unlikelihood [16, 17] and length normalization-based [4] loss terms.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “PET can be further improved by generating prompts automatically and incorporating demon- strations into the input (Gao et al., 2020); and by densifying the supervision signal with label con- ditioning (Tam et al., 2021).”
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, background): “The ongoing research on PL has explored (i) methods of prompt designing, including discrete prompts (Schick & Schütze, 2020; Shin et al., 2020; Tam et al., 2021) and continuous or soft prompts (Li & Liang, 2021; Liu et al.”
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2021, methodology): “Conﬁrming prior work (Schick and Schütze, 2021b; Tam et al., 2021), we ﬁnd that ALBERT consistently yields the best performance, so we use it as our baseline model.”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “More recent work, mainly focusing on text classification, has reported further gains with cloze-style formats [55, 56, 65], prompt engineering [24], or calibration [78].”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “Discrete Prompts Constructing effective discrete prompts for language models to perform NLP tasks is an active area of research (Schick and Schütze, 2020; Le Scao and Rush, 2021; Tam et al., 2021; Logan IV et al., 2021).”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, methodology): “We then instruct pretrained language models using in-context learning [39, 13, 50, 47] to interpret visual tokens and frame captions into the target text content.”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, methodology): “One recent approach uses a label-conditioning ob-jective (Tam et al., 2021) as a clever way to introduce a channel-like model with existing masked LMs. Extending and further integrating these different approaches would be important for using channel models in a wider range of scenarios.”
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2022, background): “, 2020) and few-shot learning for downstream NLP tasks (Tam et al., 2021).”
FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding (2021, methodology): “Moreover, the beneﬁts of some few-shot methods (e.g., ADAPET) decrease on larger pretrained models.”