AdaPrompt: Adaptive Model Training for Prompt-based NLP

By Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang. 2022

53 citations recorded by the source corpus.

Prompt-based learning, with its capability to tackle zero-shot and few-shot NLP tasks, has gained much attention in community. The main idea is to bridge the gap between NLP downstream tasks and language modeling (LM), by mapping these tasks into natural language prompts, which are then filled by pre-trained language models (PLMs). However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining. First, prompt information is not necessarily sufficiently present during LM pretraining. Second, task-specific data are not necessarily well represented during pretraining. We address these two issues by proposing AdaPrompt, adaptively retrieving external data for continual pretraining of PLMs by making use of both task and prompt characteristics. In addition, we make use of knowledge in Natural Language Inference models for deriving adaptive verbalizers. Experimental results on five NLP benchmarks show that AdaPrompt can improve over standard PLMs in few-shot settings. In addition, in zero-shot settings, our method outperforms standard prompt-based methods by up to 26.35\% relative error reduction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Heng Ji
2. Jiawei Han
3. Peng Liu
4. Chenguang Zhu
5. Yang Liu
6. Yuning Mao
7. Ming Zhong
8. Yizhu Jiao

In Computer Vision and Pattern Recognition

1. Heng Ji
2. Da Yin
3. Jiawei Han
4. Peng Liu
5. Ming Zhong
6. Yuning Mao
7. Yang Liu
8. Chenguang Zhu

In Prompting & Reasoning

1. Jiawei Han
2. Yuning Mao
3. Yang Liu
4. Chenguang Zhu

Citations in the corpus

• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, neutral): “[56] improve prompt embeddings through continued pre-training.”
• Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, background): “, 2019) or prompting (Liu et al., 2021b; Chen et al., 2022), as it can accommodate more evaluation dimensions by modifying the input question.”
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, unknown): “Various training strategies have been proposed to address the problem of catastrophic forgetting (CF) in continual learning [17], [18], [19], [20], [21].”