PPT: Pre-trained Prompt Tuning for Few-shot Learning

By Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang. 2021

499 citations recorded by the source corpus.

Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting large-scale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tuning performs comparably with conventional full-model tuning when downstream data are sufficient, whereas it is much worse under few-shot learning settings, which may hinder the application of prompt tuning. We attribute this low performance to the manner of initializing soft prompts. Therefore, in this work, we propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. We name this Pre-trained Prompt Tuning framework “PPT”. To ensure the generalization of PPT, we formulate similar classification tasks into a unified task form and pre-train soft prompts for this unified task. Extensive experiments show that tuning pre-trained prompts for downstream tasks can reach or even outperform full-model fine-tuning under both full-data and few-shot settings. Our approach is effective and efficient for using large-scale PLMs in practice.

Subfields: Artificial Intelligence, Prompting & Reasoning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

In Meta & Continual Learning

Citations in the corpus

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Finally, we demonstrate the benefits of pre-training the (IA)3 parameters before fine-tuning [18, 19].”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “To overcome the shortcomings of hand-crafting, some works optimize continuous soft prompts such as prompt-tuning (Lester et al., 2021), P-Tuning (Liu et al., 2021c), Ppt (Gu et al., 2021)”
Continual Training of Language Models for Few-Shot Learning (2022, methodology): “Following (Gu et al., 2021), we use 32 training samples for Restaurant and AGNews, 48 training samples for ACL and 56 training samples for AI due to different numbers of classes in each dataset.”
Continual Prompt Tuning for Dialog State Tracking (2022, background): “Gu et al. (2021) and Vu et al. (2021) further 161 explore the transferability of soft prompts across 162 tasks.”
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “On one hand, the random initialization of continuous prompts brings in difficulties for optimization, which can be alleviated with more training data or task-related unsupervised pretraining (Gu et al., 2021); on the other hand, continuous prompts serve as perfect carriers of taskspecific knowledge ”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “In concurrent work, Gu et al. (2021) also explore the effectiveness of prompt transfer.”
Continual Pre-training of Language Models (2023, methodology): “In NLP, CL has been used for slot ﬁlling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classiﬁcation (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).”
Data Contamination: From Memorization to Exploitation (2022, background): “Recent years have seen improvements in prompt-based methods for zero-and few-shot learning (Shin et al., 2020; Schick and Schütze, 2021; Gu et al., 2021).”
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022, methodology): “used for further boost performance by providing a better initialization (Gu et al., 2021).”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “Inspired by the recent works on pattern-exploiting training (Schick and Schütze, 2021a,b) and prompt tuning (Gu et al., 2021), we define each pattern evaluator as E = (f, g), which consists of two pattern functions to build the input and output sequence of text infilling tasks, respectively.”
OpenPrompt: An Open-source Framework for Prompt-learning (2021, result): “Gu et al. (2021) report a mix of manual template tokens and soft (trainable) to-kens sometimes yields better results than separate manual template and soft template.”
Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5) (2022, background): “Meanwhile, another line of work [18, 28, 33, 38, 45, 81] exploited continuous vector embeddings as prompts.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “PPT [119] points out that better initialization can be obtained by adding soft prompts to the pre-train stage.”
Leveraging QA Datasets to Improve Generative Data Augmentation (2022, background): “Other work proposes to pre-train prompts by adding soft prompts into the pre-training stage (Gu et al., 2022; Vu et al., 2022b,a).”
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2021, result): “Similar phenomena have been observed in a concurrent study (Gu et al., 2021) on few-shot prompt-tuning.”