Text Summarization with Pretrained Encoders

By Yang Liu, Mirella Lapata. 2019

1,667 citations recorded by the source corpus.

Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state-of-the-art results across the board in both extractive and abstractive settings.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Graham Neubig
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Yejin Choi
  7. Caiming Xiong
  8. Noah A. Smith

In General Social Sciences

  1. Arman Cohan
  2. Pengfei Liu
  3. Alexios Gidiotis
  4. Grigorios Tsoumakas
  5. Percy Liang
  6. K. McKeown
  7. Faisal Ladhak
  8. Nazli Goharian

In Natural Language Processing

  1. Mohit Bansal
  2. Graham Neubig
  3. Luke S. Zettlemoyer
  4. Jianfeng Gao
  5. Yejin Choi
  6. Caiming Xiong
  7. Percy Liang
  8. Furu Wei

Citations in the corpus