GPT Understands, Too

By Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, Jie Tang. 2021

1,460 citations recorded by the source corpus.

Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that manual discrete prompts often lead to unstable performance -- e.g., changing a single word in the prompt might result in substantial performance drop. We propose a novel method P-Tuning that employs trainable continuous prompt embeddings in concatenation with discrete prompts. Empirically, P-Tuning not only stabilizes training by minimizing the gap between various discrete prompts, but also improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is generally effective for both frozen and tuned language models, under both the fully-supervised and few-shot settings.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “Though many successful prompt tuning methods perform optimization over a continuous space using gradient-based methods (Liu et al., 2021; Qin & Eisner, 2021; Lester et al., 2021), this becomes less practical with scale, as computing gradients becomes increasingly expensive and access to models…”
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2021, methodology): “…GPT-3 (Brown et al., 2020) None Manual PET (Schick and Schütze, 2021a) All Manual LM-BFF (Gao et al., 2021) All Learned (Discrete) P-Tuning (Liu et al., 2021) All + Prompt Token Embeds Learned (Continuous) Null Prompts + Bitﬁt (Ours) Bias Terms None on in-context learning (a.k.a. priming),…”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “[77] propose łP-tuning", where continuous prompts are learned by inserting trainable variables into the embedded input.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, methodology): “…setting does not assume access to any unlabeled data or large validation sets for hyperparameter tuning [48], where prompt-based methods [5, 13, 33, 35, 55–57, 59, 63, 84] are prominently deployed to inject task descriptions into PLMs and make effective use of their language modeling…”
LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “We also compare with the other tuning methods such as [21, 25].”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “Invasive meth-ods, which are built on a strong assumption that the inner structure (e.g., self-attention and feed-forward layers) of the PLM can be modiﬁed, includes Preﬁx-Tuning (Li and Liang, 2021), Bit-ﬁt (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), L”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “When the training data is sufﬁcient, prompt tuning performs slightly worse than ﬁne-tuning [43].”
Learning to Prompt for Vision-Language Models (2021, background): “…language models, formally deﬁned by Petroni et al. (2019) as “ﬁll-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Prompt tuning appends a collection of trainable prompt tokens to pre-trained large models, which are inserted either to the input embeddings only [30, 39], or to all of the intermediate layers [35, 38].”
Continual Prompt Tuning for Dialog State Tracking (2022, background): “some works (Qin and Eisner, 2021; Liu et al., 2021; Han et al., 2021) combine hard text prompts and soft prompts whose embeddings are learned through back-propagation.”
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, background): “…including discrete prompts (Schick & Sch¨utze, 2020; Shin et al., 2020; Tam et al., 2021) and continuous or soft prompts (Li & Liang, 2021; Liu et al., 2021c; Lester et al., 2021), ( ii ) applications of PL (Han et al., 2021; Ben-David et al., 2021; Ding et al., 2021), and analysis of…”
CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “In Section 3.3, we provide several variants of CLIP-Adapter.”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “(2) Most prompt engineering approaches apply lightweight changes to the original prompt (Liu et al., 2021a), whereas our reframed prompts here are often very different from the raw instructions.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “We refer readers to Liu et al. (2021a) for a recent survey on prompt-based learning research.”
PTR: Prompt Tuning with Rules for Text Classification (2021, background): “Moreover, human-picked prompts have achieved promising results on few-class classification tasks such as sentiment classification and natural language inference (Schick and Schütze, 2021; Liu et al., 2021).”
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2021, methodology): “We follow Zhang et al. (2021) and Gao et al. (2021) to use the original validation set as the test set D test , which means Hybrid Prompt Tuning In hybrid prompt tuning, both soft and hard prompts are used (Liu et al., 2021; Han et al., 2021b).”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, methodology): “…calibration (Zhao et al., 2021; Holtz-man et al., 2021), or learning better prompts, either in a discrete space (Shin et al., 2020; Jiang et al., 2020; Gao et al., 2021) or in a continuous space (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021; Zhong et al., 2021; Qin and Eisner, 2021).”