P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
By Xiao Liu, Kaixuan Ji, Yicheng Fu, W. Tam, Zhengxiao Du, Zhilin Yang, Jie Tang. 2022
1,117 citations recorded by the source corpus.
Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning (CITATION) optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Prompting & Reasoning, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Electrical and Electronic Engineering
- Matei A. Zaharia
- Ming Ding
- Steven C. H. Hoi
- Ajmal Saeed Mian
- Tianjun Zhang
- Xia Hu
- Ion Stoica
- Traian Rebedea
In Prompting & Reasoning
In Efficiency & Compression
- Ion Stoica
- Joseph E. Gonzalez
- Matei A. Zaharia
- Akhilesh Deepak Gotmare
- Ming Ding
- Nick Barnes
- Steven C. H. Hoi
- Ajmal Saeed Mian
Citations in the corpus
- GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Table 21: The results of GLM-130B on the SuperGLUE dataset obtained using the P-tuning v2 (Liu et al., 2022).”
- A Comprehensive Overview of Large Language Models (2023, positive): “The study in [176] demonstrates that LLMs, trained on vast text data, can serve as effective human models for certain HRI tasks, achieving predictive performance comparable to specialized machine-learning models.”
- ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “It has achieved satisfied performance in both NLP tasks [24, 27, 35] and CV tasks [47, 50, 57].”
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “The common techniques for PET are LoRA [49], Prefix Tuning [68], P-Tuning [72, 73].”
- RAFT: Adapting Language Model to Domain Specific RAG (2024, background): “…LLMs. Addressing these challenges, research in the realm of parameter-efficient fine-tuning (Houlsby et al., 2019), such as Prompt Tuning (Lester et al., 2021), Prefix-Tuning (Li & Liang, 2021), P-Tuning (Liu et al., 2022a) and Low-Rank based fine-tuning (Hu et al., 2021), has gained traction.”
- SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (2023, methodology): “, 2021), and (iii) prompting using P-Tuning v2 (Liu et al., 2022).”
- Graph Retrieval-Augmented Generation: A Survey (2024, positive): “GNNs can directly encode graph data, capturing complex relationships and node features inherent in the graph structure.”
- Prompttts: Controllable Text-To-Speech With Text Descriptions (2022, methodology): “Considering that solely adding the style representation to the input embedding may not be powerful enough to guide the model [16], we prepend the style representation to the input of every Transformer block.”
- Adapting Language Models to Compress Contexts (2023, methodology): “Soft prompts Soft prompt tuning is an effective method to adapt pre-trained Transformers without updating existing parameters (Lester et al., 2021; Zhong et al., 2021; Liu et al., 2022).”
- NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023, methodology): “While most alignment methods provide general embedded rails, in a similar way developers can add app-specific embedded rails to an LLM via prompt tuning (Lester et al., 2021; Liu et al., 2022).”
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, background): “Another common feature is the use of prompting, either with continuous or discrete prompts, to efficiently align models to downstream tasks [Liu et al., 2021, Lester et al., 2021, Liu et al., 2022, Ponti et al., 2023].”