Efficient Adaptation of Pretrained Transformers for Abstractive Summarization

By Andrew Hoang, Antoine Bosselut, Asli Celikyilmaz, Yejin Choi. 2019

44 citations recorded by the source corpus.

Large-scale learning of transformer language models has yielded improvements on a variety of natural language understanding tasks. Whether they can be effectively adapted for summarization, however, has been less explored, as the learned representations are less seamlessly integrated into existing neural text production architectures. In this work, we propose two solutions for efficiently adapting pretrained transformer language models as text summarizers: source embeddings and domain-adaptive training. We test these solutions on three abstractive summarization datasets, achieving new state of the art performance on two of them. Finally, we show that these improvements are achieved by producing more focused summaries with fewer superfluous and that performance improvements are more pronounced on more abstractive datasets.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Mohit Bansal
3. Noah A. Smith
4. Asli Celikyilmaz
5. Luke S. Zettlemoyer
6. Jianfeng Gao
7. Mirella Lapata
8. Antoine Bosselut

In Model Architectures

1. Saadia Gabriel
2. Ari Holtzman
3. Ivan Titov
4. Yejin Choi
5. Asli Celikyilmaz
6. Mirella Lapata
7. Arthur Bravzinskas
8. Antoine Bosselut

In Efficiency & Compression

1. Ari Holtzman
2. Ivan Titov
3. Yejin Choi
4. Shruti Tople

In Summarization

1. Yejin Choi
2. Mirella Lapata
3. Ryan T. McDonald

Citations in the corpus

• Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow (2019, background): “proposed a decoder-only transformer model for summarization across multiple Wikipedia articles and several later works have explored Transformer-based architectures for abstractive summarization over Newswire (Khandelwal et al., 2019; Zhang et al., 2019b; Hoang et al., 2019).”
• Analyzing Leakage of Personally Identifiable Information in Language Models (2023, background): “Language Models (LMs) are fundamental to many natural language processing tasks [20, 45, 68].”
• Few-Shot Learning for Opinion Summarization (2020, methodology): “Unlike recent approaches to language model adaptation for abstractive singledocument summarization (Hoang et al., 2019; Raffel et al., 2019) that utilize hundreds of thousands of summaries, our two annotated datasets consist of only 60 and 100 annotated data-points.”
• Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “The overarching goal is to enhance the reliability and utility of LLMs through comprehensive exploration and resolution of these vulnerabilities and offer pathways for future research toward secure and privacy-preserving LLM systems.”