Sample Efficient Text Summarization Using a Single Pre-Trained Transformer

By Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, Lukasz Kaiser. 2019

90 citations recorded by the source corpus.

Language model (LM) pre-training has resulted in impressive performance and sample efficiency on a variety of language understanding tasks. However, it remains unclear how to best use pre-trained LMs for generation tasks such as abstractive summarization, particularly to enhance sample efficiency. In these sequence-to-sequence settings, prior work has experimented with loading pre-trained weights into the encoder and/or decoder networks, but used non-pre-trained encoder-decoder attention weights. We instead use a pre-trained decoder-only network, where the same Transformer LM both encodes the source and generates the summary. This ensures that all parameters in the network, including those governing attention over source states, have been pre-trained before the fine-tuning step. Experiments on the CNN/Daily Mail dataset show that our pre-trained Transformer LM substantially improves over pre-trained Transformer encoder-decoder networks in limited-data settings. For instance, it achieves 13.1 ROUGE-2 using only 1% of the training data (~3000 examples), while pre-trained encoder-decoder models score 2.3 ROUGE-2.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Fine-Tuning Language Models from Human Preferences (2019, background): “Kry´sci´nski et al. (2019) found that 30% of samples from the supervised summarization models they tested contained inconsistencies, and Khandelwal et al. (2019) found that their pretrained encoder-decoder model “hallucinates facts...which are topical but never appear in the source”.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “Khandelwal et al. (2019) pre-trained a Transformer language model on Wikipedia, and fine-tuned using 3000 examples, achieving 13.1 ROUGE-2.”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “More advanced pretrained text generators (Radford et al., 2018, 2019; Dong et al., 2019; Song et al., 2019; Khandelwal et al., 2019; Rothe et al., 2020) are even better at capturing world knowledge as they are informed by a vast amount of background text.”
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019, background): “This requires natural language generation techniques with high level of semantic understanding (Chopra et al., 2016; Rush et al., 2015; Khandelwal et al., 2019; Zhang et al., 2019; See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018).”
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2019, methodology): “To overcome the issue that the encoder-decoder attention is not pre-trained, Khandelwal et al. (2019) pre-trained a single transformer language model that encodes the source and generates the target.”
Planning with Learned Entity Prompts for Abstractive Summarization (2021, methodology): “Most models focus on task-agnostic pretraining using the left-to-right language modeling objective (Radford et al., 2018; Khandelwal et al., 2019; Dong et al., 2019) or reconstructing the corrupted input text using a sequence-to-sequence framework (Song et al.”
Generalization in Generation: A closer look at Exposure Bias (2019, unknown): “, 2017), summarization (Rush et al., 2015; Khandelwal et al., 2019), dialogue (Serban et al.”
Uncertainty Estimation for Language Reward Models (2022, background): “A common approach is to use supervised learning to mimic some reference text written by a human [36, 31, 22].”
Symbolic Behaviour in Artificial Intelligence (2021, background): “[34] Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser.”
QURIOUS: Question Generation Pretraining for Text Generation (2020, methodology): “Unsupervised or semi-supervised pretrained encoder-decoder models are quickly becoming the standard for text generation (Khandelwal et al., 2019; Dong et al., 2019; Song et al., 2019; Rothe et al., 2019; Lewis et al., 2019), following the success of pretraining methods on popular Natural Language…”
Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow (2019, background): “proposed a decoder-only transformer model for summarization across multiple Wikipedia articles and several later works have explored Transformer-based architectures for abstractive summarization over Newswire (Khandelwal et al., 2019; Zhang et al., 2019b; Hoang et al., 2019).”
Controlling the Amount of Verbatim Copying in Abstractive Summarization (2019, background): “While it is tempting to use pretrained deep representations to “warm-start” the encoder/decoder, Khandelwal et al. (2019) find that results can be less satisfying as the attention weights are still not pretrained.”