PTR: Prompt Tuning with Rules for Text Classification

By Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, Maosong Sun. 2021

596 citations recorded by the source corpus.

Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. By using additional prompts to fine-tune PLMs, we can further stimulate the rich knowledge distributed in PLMs to better serve downstream tasks. Prompt tuning has achieved promising results on some few-class classification tasks such as sentiment classification and natural language inference. However, manually designing lots of language prompts is cumbersome and fallible. For those auto-generated prompts, it is also expensive and time-consuming to verify their effectiveness in non-few-shot scenarios. Hence, it is still challenging for prompt tuning to address many-class classification tasks. To this end, we propose prompt tuning with rules (PTR) for many-class text classification and apply logic rules to construct prompts with several sub-prompts. In this way, PTR is able to encode prior knowledge of each class into prompt tuning. We conduct experiments on relation classification, a typical and complicated many-class classification task, and the results show that PTR can significantly and consistently outperform existing state-of-the-art baselines. This indicates that PTR is a promising approach to take advantage of both human prior knowledge and PLMs for those complicated classification tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, background): “, 2020; Le Scao and Rush, 2021), while the latter inserts learnable parameters in the template and directly optimize them during training (Han et al., 2021; Liu et al., 2021; Lester et al., 2021).”
Continual Prompt Tuning for Dialog State Tracking (2022, background): “some works (Qin and Eisner, 2021; Liu et al., 2021; Han et al., 2021) combine hard text prompts and soft prompts whose embeddings are learned through back-propagation.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “By designing, generating and searching discrete (Petroni et al., 2019; Gao et al., 2021; Hu et al., 2021) or continuous (Liu et al., 2021b; Han et al., 2021; Lester et al., 2021) prompts and using MLM for specific downstream tasks, these models could (1) bridge the gap between pre-training and…”
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “Prompt tuning and P-tuning have been proved quite effective in many NLU tasks, including knowledge probing, sentimental analysis (Li et al., 2021), relation extraction (Han et al., 2021), entity typing (Ding et al., 2021) and so on.”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “For example, they are studied in the context of classification (Petroni et al., 2019; Schick and Schütze, 2021) or applied as prompt composition and prompt decomposition for relation extraction (Han et al., 2021) and named entity recognition (Cui et al., 2021) tasks, respectively.”
PPT: Pre-trained Prompt Tuning for Few-shot Learning (2021, methodology): “Hybrid Prompt Tuning In hybrid prompt tuning, both soft and hard prompts are used (Liu et al., 2021; Han et al., 2021b).”
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2021, methodology): “, 2021) , Relation Extraction (Han et al., 2021; Chen et al., 2021), and Knowledge Probing (Petroni et al.”
Pre-trained Language Models Can be Fully Zero-Shot Learners (2022, methodology): “Prompting has been applied to a large variety of tasks such as Text Classification (Schick and Schütze, 2021a), Natural Language Understanding (Xu et al., 2022), Knowledge Probing (Petroni et al., 2019), and Relation Extraction (Han et al., 2021).”
Prompt Distribution Learning (2022, background): “Recently, some works [12,22,24,48], also known as prompt tuning , attempt to learn soft (continuous) prompts directly instead of searching for discrete prompts.”
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, background): “Han et al. [31] researched strategies for designing prompts for classiﬁcation tasks.”
Explaining Patterns in Data with Language Models via Interpretable Autoprompting (2022, background): “Recently, autoprompting (i.e. automatically searching for a prompt or prompt-embedding via optimization) has emerged, with methods such as preﬁx-tuning [8], P-tuning [23], prompt-tuning with rules [24], knowledgeable prompt tuning [25] and many more [6].”
Prompt-Learning for Fine-Grained Entity Typing (2021, background): “Despite the success of ﬁne-tuning PLMs, the huge objective form gap between pre-training and ﬁne-tuning still hinders the full use of per-trained knowledge for downstream tasks (Liu et al., 2021b; Han et al., 2021b; Hu et al., 2021).”
A Review on Language Models as Knowledge Bases (2022, background): “A number of papers show that soft prompting, when optimized for specific tasks such as relation extraction and natural language understanding, can achieve better performance with minimal tuning (Lester et al., 2021; Han et al., 2021).”
Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models (2022, methodology): “Currently, prompting methods are dominantly NLP-speci c [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], despite the fact that the framework serves a general purpose: adapt a pre-trained model by modifying the data space.”