Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners

By Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, Huajun Chen. 2021

209 citations recorded by the source corpus.

Large-scale pre-trained language models have contributed significantly to natural language processing by demonstrating remarkable abilities as few-shot learners. However, their effectiveness depends mainly on scaling the model parameters and prompt design, hindering their implementation in most real-world applications. This study proposes a novel pluggable, extensible, and efficient approach named DifferentiAble pRompT (DART), which can convert small language models into better few-shot learners without any prompt engineering. The main principle behind this approach involves reformulating potential natural language processing tasks into the task of a pre-trained language model and differentially optimizing the prompt template as well as the target label with backpropagation. Furthermore, the proposed approach can be: (i) Plugged to any pre-trained language models; (ii) Extended to widespread classification tasks. A comprehensive evaluation of standard NLP tasks demonstrates that the proposed approach achieves a better few-shot performance. Code is available in https://github.com/zjunlp/DART.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “…et al. (2019); Krittanawong et al. (2020); Wang et al. (2019b) Branwen (2020); Dale (2021); Lacker (2020); Lin et al. (2021); Rae et al. (2021); Zhang et al. (2021b) 2.3.3 Causing material harm by disseminating misinformation e.g. in medicine or law Quach (2020) 2.3.4 Leading users to perform…”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, neutral): “, 2021), where prompt-based methods (Brown et al., 2020; Gao et al., 2021a; Liu et al., 2021b; Logan IV et al., 2021; Scao & Rush, 2021; Schick & Schütze, 2021a;b;c; Tam et al., 2021; Zhang et al., 2022) are prominently deployed to inject task descriptions into PLMs and make effective use of their l”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “There have also been efforts to develop methods that find discrete prompts [73, 74] and training prompts using methods similar to prompt tuning [75].”
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “, 2021), few-shot learning (Zheng et al., 2021; Xu et al., 2021a; Zhao et al., 2021; Gu et al., 2021; Zhang et al., 2021a), relation extraction (Chen et al.”
Taxonomy of Risks posed by Language Models (2022, methodology): “This can simplify the development of task-specific LMs, for example by reducing the adaptation process to prompt design [204].”
Few-shot Learning with Multilingual Generative Language Models (2021, methodology): “This is inspired by the masked language modeling prompts constructed by recent work (Schick and Schütze, 2021; Zhang et al., 2021).”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “, 2022), and optimizing soft prompts (Li and Liang, 2021; Qin and Eisner, 2021; Lester et al., 2021; Gu et al., 2021; Zhang et al., 2022).”
Few-shot Learning with Multilingual Language Models (2021, methodology): “This is inspired by the masked language modeling prompts constructed by recent work (Schick and Schütze, 2021; Zhang et al., 2021).”
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024, unknown): “Continuous prompt-based methods, which only tune parameters of some input tokens (Li & Liang, 2021; Liu et al., 2021b;a; Zhang et al., 2021) attract lots of attention.”
The Creativity of Text-to-Image Generation (2022, background): “standing of the underlying technologies, simply by writing prompts in natural language (a practice called prompt engineering [20], prompt programming [48], prompt design [61], or prompting for short).”
A Review on Language Models as Knowledge Bases (2022, background): “, 2021a) and label token optimization (Zhang et al., 2021).”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, methodology): “, 2020) or parameterizing prompts as continuous learnable embeddings (Lester et al., 2021; Zhang et al., 2022; Zhong et al., 2021).”