Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

By Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juan-Zi Li, Maosong Sun. 2021

439 citations recorded by the source corpus.

Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompttuning (KPT), to improve and stabilize prompttuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Xipeng Qiu
4. Minlie Huang
5. Jianfeng Gao
6. Mohit Bansal
7. Graham Neubig
8. Yang Liu

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, methodology): “By designing, generating and searching discrete (Petroni et al., 2019; Gao et al., 2021; Hu et al., 2021) or continuous (Liu et al., 2021b; Han et al., 2021; Lester et al., 2021) prompts and using MLM for specific downstream tasks, these models could (1) bridge the gap between pre-training and…”
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “Prompting’s advantage and effectiveness in a wide range of NLP applications has been verified in recent literature, including text classification (Hu et al., 2021; Min et al., 2021; Sun et al., 2021; Li et al., 2021; Zhang et al., 2021b), entity typing (Ding et al.”
• On Robust Prefix-Tuning for Text Classification (2022, methodology): “…et al., 2019; Shi et al., 2020; Xu et al., 2020; Ye et al., 2020), adversary detection (Pruthi et al., 2019; Zhou et al., 2019), and adversarial training (Miyato et al., 2017; 2019; Zhu et al., 2020; Jiang et al., 2020a; Liu et al., 2020; Wang et al., 2021; Dong et al., 2021; Zhou et al., 2021).”
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, background): “The success of adapters in NLP has also led to widespread interest in applying adapter techniques to visual tasks.”
• AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022, methodology): “It demonstrates that AdaPrompt has a strong generalization ability when query data and test set are different.”
• CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, methodology): “2021a; Hu et al. 2021) transforms the input to a discrete textual string as a “prompt” to feed to a PLM. For classification tasks, a verbalizer is often used to map the PLM’s output to task labels. The verbalizer can also be learned for classification tasks (Gao, Fisch, and Chen 2021; Hu et al. 2021”
• Black-Box Tuning for Language-Model-as-a-Service (2022, background): “More advanced techniques such as prompt engineering (Gao et al., 2021), label words engineering (Schick et al., 2020; Shin et al., 2020; Hu et al., 2021b), prompt pre-training (Gu et al., 2021), and prompt ensembling (Lester et al., 2021) are orthogonal to this work and therefore can further…”
• What does a platypus look like? Generating customized prompts for zero-shot image classification (2022, methodology): “[20] use a LLM to add labels to text to improve text classification accuracy.”
• Prompt-Learning for Fine-Grained Entity Typing (2021, methodology): “Despite the success of fine-tuning PLMs, the huge objective form gap between pre-training and fine-tuning still hinders the full use of per-trained knowledge for downstream tasks (Liu et al., 2021b; Han et al., 2021b; Hu et al., 2021).”
• OpenPrompt: An Open-source Framework for Prompt-learning (2021, methodology): “Additional to manually-defined verbalizers, we implement automatic verbalizers like AutomaticVerbalizer and KnowledgeableVerbalizer (Hu et al., 2021).”
• Pre-trained Language Models Can be Fully Zero-Shot Learners (2022, methodology): “To concentrate on the verbalizer and reduce the influence of templates, we adopt multiple fixed manual templates following (Hu et al., 2022).”
• Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning (2021, background): “…tasks and adding some prompt tokens into the input, including human-designed explainable prompts (Brown et al., 2020; Schick and Schütze, 2021a,b; Han et al., 2021b; Ding et al., 2021; Hu et al., 2021) and automatically searched prompts (Jiang et al., 2020; Shin et al., 2020; Gao et al., 2021).”
• Explaining Patterns in Data with Language Models via Interpretable Autoprompting (2022, background): “Recently, autoprompting (i.e. automatically searching for a prompt or prompt-embedding via optimization) has emerged, with methods such as prefix-tuning [8], P-tuning [23], prompt-tuning with rules [24], knowledgeable prompt tuning [25] and many more [6].”
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction (2021, background): “(Hu et al. 2021) propose an approach to incorporate external knowledge into the verbalizer with calibration.”