Text Summarization with Pretrained Encoders
By Yang Liu, Mirella Lapata. 2019
1,667 citations recorded by the source corpus.
Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state-of-the-art results across the board in both extractive and abstractive settings.
Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Caiming Xiong
- Noah A. Smith
In General Social Sciences
- Arman Cohan
- Pengfei Liu
- Alexios Gidiotis
- Grigorios Tsoumakas
- Percy Liang
- K. McKeown
- Faisal Ladhak
- Nazli Goharian
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Caiming Xiong
- Percy Liang
- Furu Wei
Citations in the corpus
- Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “The pre-trained models are usually used as the initialization for fine-tuning downstream tasks such as controllable language modeling (Shirish Keskar et al., 2019), summarization (Song et al., 2019; Liu and Lapata, 2019) and dialogue generation (Zhang et al., 2019).”
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, background): “Although sentence permutation only shows significant additive gains (Liu & Lapata, 2019) 42.13 19.60 39.18 38.81 16.50 31.27 BART 44.16 21.28 40.90 45.14 22.27 37.25 Table 3: Results on two standard summarization datasets.”
- WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “We use large, pre-trained language models as a starting point for our experiments, given their success on a variety of downstream Natural Language Processing tasks (Devlin et al., 2019), including state of the art results for text summarization (Lewis et al., 2019b; Liu and Lapata, 2019).”
- MLSUM: The Multilingual Summarization Corpus (2020, background): “In particular, Liu and Lapata (2019) added a randomly initialized decoder on top of BERT.”
- ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, neutral): “th, on QG. Table 2: Results on Question Generation (QG) and Abstractive Summarization (Summ.) tasks. QG (SQuAD) Summ. (CNN/DM) #params BLEU-1 BLEU-4 ROUGE-1 ROUGE-L BLEU-4 SemQG [56] 18.37 BertSumAbs [22] 340M 41.72 38.76 UniLM [9] 340M 22.78 43.33 40.41 PEGASUS [55] 568M 44.17 41.11 T5-large (MLE) ”
- PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, neutral): “BERTSUMABS (Liu and Lapata, 2019) 41.”
- Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “For extractive and abstractive summarization, researchers fine-tune masked language models (e.g., BERT; Devlin et al., 2019) and encode-decoder models (e.g., BART; Lewis et al., 2020), respectively (Zhong et al., 2020; Liu and Lapata, 2019; Raffel et al., 2020).”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “…use a hierarchical approach to generating final summaries, either by having a hierarchical encoder (Cohan et al., 2018; Zhang et al., 2019c; Liu and Lapata, 2019a), or by first running an extractive summarization model followed by an abstractive model (Subramanian et al., 2019; Liu et al.,…”
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “Following previous language model pre-training (Devlin et al., 2018; Liu et al., 2019) and task-agnostic pre-trained language model distillation (Sanh et al., 2019; Jiao et al., 2019; Sun et al., 2019b), we evaluate our distilled models on the extractive question answering and GLUE benchmark.”
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training (2020, positive): “BERTSUMABS (Liu & Lapata, 2019) fine-tunes a BERT encoder that is pre-trained with an autoencoding objective, concatenating with a randomly initialized decoder.”
- Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “State-of-the-art solutions utilize self-attentive Trans-former blocks (Liu, 2019; Liu and Lapata, 2019; Zhang et al., 2019), attention and copying mechanisms (See et al., 2017; Cohan et al., 2018), and multi-objective training strategies (Guo et al., 2018; Pasunuru and Bansal, 2018), including…”
- Explainable Automated Fact-Checking for Public Health Claims (2020, methodology): “In order to deploy the model proposed by (Liu and Lapata, 2019) we also implemented an explanation generation model.”
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Later, QA is incorporated in human evaluation where one person writes questions and another person answers them based on the summary (Clarke and Lapata, 2010; Liu and Lapata, 2019).”
- Pre-trained models for natural language processing: A survey (2020, neutral): “Furthermore, Liu and Lapata [116] proposed BERTSUM.”
- PTR: Prompt Tuning with Rules for Text Classification (2021, background): “, 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al.”