P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
By Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, Jie Tang. 2021
1,125 citations recorded by the source corpus.
Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning \cite{li2021prefix,qin2021learning} optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.Our code and data are released at https://github.com/THUDM/P-tuning-v2.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Yoshua Bengio
- Caiming Xiong
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Bernt Schiele
- Yu Qiao
- Jitendra Malik
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Jonathan Herzig
- Michael Lewis
- Chengwei Qin
- Derek Tam
- Dawn Song
- Yu Qiao
- Eric P. Xing
Citations in the corpus
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “[54] introduce several tricks to improve prompt tuning, An et al.”
- Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “Invasive meth-ods, which are built on a strong assumption that the inner structure (e.g., self-attention and feed-forward layers) of the PLM can be modified, includes Prefix-Tuning (Li and Liang, 2021), Bit-fit (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), L”
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “Specifically, following P-Tuning v2 [42], learnable prompt tokens with random initialization are added at each layer of the Transformer encoder, and class labels with linear heads are added for classification tasks.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “For extractive question answering, we follow P-tuning v2 (PT2) [38] to fine-tune the RoBERTa large [40] model on SQuAD [54] v1.”
- Visual Prompt Tuning (2022, methodology): “Transformer models [73] have gained huge success in NLP [17,66,7].”
- Progressive Prompts: Continual Learning for Language Models (2023, background): “At the same time, Liu et al. (2021) report that prompt embedding reparameterization can hinder performance on certain tasks and is highly sensitive to its initialization.”
- Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection (2022, background): “Prompt Learning Conditioning on large-scale pre-trained language models, prompt learning (Brown et al., 2020; Lester et al., 2021; Chen et al., 2021; Liu et al., 2021; Wang et al., 2021a)”
- DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “Most existing prompt-related work simply place prompts only at the first MSA [60, 24], or at every MSA layer [25, 29].”
- Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “Prompt tuning (Lester et al., 2021; Li and Liang, 2021; Liu et al., 2021b) 3 Experimental Setup”
- VeRA: Vector-based Random Matrix Adaptation (2023, result): “(2023); Liu et al. (2021), we also observe a higher adaptation for the later layers compared to earlier ones.”
- DeepStruct: Pretraining of Language Models for Structure Prediction (2022, background): “Figure 6 shows that the performance improvements follow the scaling law (Raffel et al., 2019; Lester et al., 2021; Wei et al., 2021; Sanh et al., 2021; Liu et al., 2021).”
- On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “These methods are found to be more stable than fine-tuning the full model (He et al., 2021b; Lee et al., 2019; Houlsby et al., 2019; Zaken et al., 2021; Sung et al., 2021; Liu et al., 2021).”
- Personalize Segment Anything Model with One Shot (2023, methodology): “Prompt tuning (Lester et al., 2021; Zhou et al., 2022; Jia et al., 2022; Liu et al., 2021) suggests using learnable soft prompts alongside frozen models to perform specific downstream tasks, achieving more competitive performance with scale and robust domain transfer compared to full model tuning.”
- Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Restrictions apply. approaches include Low-rank Adaptation (LoRA) [194] and its variants such as QLoRA [195] and GLoRA [196], Prefix tuning [197], [198], Adapters [164], [166], Prompt tuning [199], [200].”
- Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “, P-tuning v2 [192]) can improve the zero-shot retrieval performance of DPR and ColBERT.”
- Black-Box Tuning for Language-Model-as-a-Service (2022, background): “(2) P-Tuning v2 (Liu et al., 2021a) is an improved variant of prompt tuning.”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “For instance, p-tuning v2 [121] removes reparameterization and expands its usage to broader model scales and NLP tasks.”