Generating Abstractive Summaries with Finetuned Language Models
By Sebastian Gehrmann, Zachary M. Ziegler, Alexander M. Rush. 2019
23 citations recorded by the source corpus.
Neural abstractive document summarization is commonly approached by models that exhibit a mostly extractive behavior. This behavior is facilitated by a copy-attention which allows models to copy words from a source document. While models in the mostly extractive news summarization domain benefit from this inductive bias, they commonly fail to paraphrase or compress information from the source document. Recent advances in transfer-learning from large pretrained language models give rise to alternative approaches that do not rely on copy-attention and instead learn to generate concise and abstractive summaries. In this paper, as part of the TL;DR challenge, we compare the abstractiveness of summaries from different summarization approaches and show that transfer-learning can be efficiently utilized without any changes to the model architecture. We demonstrate that the approach leads to a higher level of abstraction for a similar performance on the TL;DR challenge tasks, enabling true natural language compression.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Hinrich Schütze
- Zhiguo Wang
- K. McKeown
- Bing Xiang
- C. D. Santos
- Mirella Lapata
- Graham Neubig
In Computer Vision and Pattern Recognition
- Bing Xiang
- Benno Stein
- Hinrich Schütze
- Feng Nan
- K. McKeown
- Ramesh Nallapati
- C. D. Santos
- Dejiao Zhang
Citations in the corpus
- Entity-level Factual Consistency of Abstractive Text Summarization (2021, background): “Many recent advances in deep neural networks have led to significant improvement in the quality of abstractive summarization (Radford et al., 2019; Gehrmann et al., 2019; Lewis et al., 2019).”
- Improving Factual Consistency of Abstractive Summarization via Question Answering (2021, background): “Recent advances in neural text generation have led to significant improvement in the quality of abstractive summarization (Radford et al., 2019; Gehrmann et al., 2019; Lewis et al., 2019).”
- Towards Summarization for Social Media - Results of the TL;DR Challenge (2019, background): “Gehrmann et al. (2019) leveraged fine-tuned language models to generate abstractive summaries.”