PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization
By Xiaochen Liu, Yu Bai, Jiawei Li, Yinan Hu, Yang Gao. 2022
25 citations recorded by the source corpus.
Few-shot abstractive summarization has become a challenging task in natural language generation. To support it, we developed a novel soft prompts architecture coupled with a prompt pre-training plus prompt fine-tuning paradigm, which is effective and tunes only extremely light parameters. To meet the structure of the generation models, the soft prompts comprise continuous input embeddings across an encoder and a decoder. Importantly, a new inner-prompt placed in the text is introduced to capture document-level information. The aim is to devote attention to understanding the document that better prompts the model to generate document-related content. In the training process, the prompt pre-training with self-supervised pseudo-data firstly teaches the model basic summarizing capability. Then, with few-shot examples, only the designed lightweight soft prompts are fine-tuned. Experimental results on the CNN/DailyMail and XSum datasets show that our method, with only 0.1% of the parameters, outperforms full-model tuning where all model parameters are tuned. It also surpasses Prompt Tuning by a large margin and delivers competitive results against Prefix-Tuning with 3% of the parameters.
Subfields: Artificial Intelligence, Prompting & Reasoning, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Derek Tam
- Colin Raffel
- Haokun Liu
- Luke S. Zettlemoyer
- Mohammed Muqeeth
- Tenghao Huang
- Jay Mohta
In Prompting & Reasoning
In Summarization
Citations in the corpus
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Prompt tuning has been applied to various applications for NLP including continual learning [59], model robustness [60, 61], summarization [62], machine translation [63], co-training [64], probing language models [65, 65], inverse prompting [66] and transfer learning [67].”