Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation

By Alexander R. Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq R. Joty, Dragomir R. Radev, Yashar Mehdad. 2020

104 citations recorded by the source corpus.

Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. However, these models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche domains. In this work, we introduce a novel and generalizable method, called WikiTransfer, for fine-tuning pretrained models for summarization in an unsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such as the length and level of abstraction of the desired summaries. WikiTransfer models achieve state-of-the-art, zero-shot abstractive summarization performance on the CNN-DailyMail dataset and demonstrate the effectiveness of our approach on three additional diverse datasets. These models are more robust to noisy data and also achieve better or comparable few-shot performance using 10 and 100 training examples when compared to few-shot transfer from other summarization datasets. To further boost performance, we employ data augmentation via round-trip translation as well as introduce a regularization term for improved few-shot transfer. To understand the role of dataset aspects in transfer performance and the quality of the resulting output summaries, we further study the effect of the components of our unsupervised fine-tuning data and analyze few-shot performance using both automatic and human evaluation.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

CM3: A Causal Masked Multimodal Model of the Internet (2022, background): “Recent advancements in large-scale generative sequence modeling have significantly improved zeroshot performance on multiple modalities, including text Brown et al. (2020); Fabbri et al. (2020); Aghajanyan et al. (2021) and images Ramesh et al. (2021).”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, background): “WikiTransfer (Fabbri et al., 2021) fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such as the length and level of abstraction.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “For summarization, we also compare our method to WikiTransfer (Fabbri et al., 2021) and Pega-sus (Zhang et al., 2020) – both of which use data augmentation techniques tailored towards summarization.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Fine-tuning can be improved with advanced techniques, such as multi-stage methods [138], intermediate fine-tuning [139, 140, 141], or self-supervised training [142].”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “[42] performed summarization on intermediate pseudo-summaries created from Wikipedia to improve the zeroshot and few-shot performance of abstractive summarization, and Mao et al.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “The authors in [75], [76], [77] have also shown that fine-tuning can be embedded in state-of-the-art meta-learning or semi-supervised learning frameworks for optimizing model parameters.”
BRIO: Bringing Order to Abstractive Summarization (2022, unknown): “11http://cs.umd.edu/~snover/tercom/ 2021; Fabbri et al., 2021) has shown that few-shot learning can be an effective fine-tuning method of pre-trained models for text generation tasks.”