Pretraining-Based Natural Language Generation for Text Summarization
By Haoyu Zhang, Yeyun Gong, Yu Yan, Nan Duan, Jianjun Xu, Ji Wang, Ming Gong, M. Zhou. 2019
228 citations recorded by the source corpus.
In this paper, we propose a novel pretraining-based encoder-decoder framework, which can generate the output sequence based on the input sequence in a two-stage manner. For the encoder of our model, we encode the input sequence into context representations using BERT. For the decoder, there are two stages in our model, in the first stage, we use a Transformer-based decoder to generate a draft output sequence. In the second stage, we mask each word of the draft sequence and feed it to BERT, then by combining the input sequence and the draft representation generated by BERT, we use a Transformer-based decoder to predict the refined word for each masked position. To the best of our knowledge, our approach is the first method which applies the BERT into text generation tasks. As the first step in this direction, we evaluate our proposed method on the text summarization task. Experimental results show that our model achieves new state-of-the-art on both CNN/Daily Mail and New York Times datasets.
Subfields: Artificial Intelligence, Natural Language Processing, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Caiming Xiong
- Richard Socher
- Luke S. Zettlemoyer
- Percy Liang
- Pengfei Liu
In Natural Language Processing
- Graham Neubig
- Pengfei Liu
- Mohit Bansal
- Xipeng Qiu
- Caiming Xiong
- Richard Socher
- Jianfeng Gao
- Kyunghyun Cho
In Summarization
Citations in the corpus
- SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019, background): “This requires natural language generation techniques with high level of semantic understanding (Chopra et al., 2016; Rush et al., 2015; Khandelwal et al., 2019; Zhang et al., 2019; See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018).”
- Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “State-of-the-art solutions utilize self-attentive Trans-former blocks (Liu, 2019; Liu and Lapata, 2019; Zhang et al., 2019), attention and copying mechanisms (See et al., 2017; Cohan et al., 2018), and multi-objective training strategies (Guo et al., 2018; Pasunuru and Bansal, 2018), including…”
- BARTScore: Evaluating Generated Text as Text Generation (2021, neutral): “Abstractive Systems A1: Ptr-Gen [62]; A2: Bottom-up [17]; A3: Fast-Abs-RL [5]; A4: Two-stage-RL [74]; A5: BERT-Ext-Abs [41]; A6: BERT-Abs [41]; A7: Trans-Abs [41]; A8: UniLM-1 [10]; A9: UniLM-2 [2]; A10: T5-base [56]; A11: T5-large [56]; A12: T5-11B [56]; A13: BART [33]; A14: SemSim [42].”
- Better Document-Level Machine Translation with Bayes’ Rule (2019, result): “2019). Language generation is another area where pretrained language models have been applied to, with existing work focusing on fine-tuning approaches for repurposing an unconditional language model (Zhang et al., 2019; Edunov et al., 2019; Song et al., 2019; Dong et al., 2019; Ziegler et al., 2019;”
- Distilling Knowledge Learned in BERT for Text Generation (2019, methodology): “Zhang et al. (2019a) Our approach is novel in the sense that we do not directly use the parameters of BERT in the Seq2Seq model.”
- Distilling the Knowledge of BERT for Text Generation (2019, methodology): “Zhang et al. (2019b) further extended the idea to Transformer with joint training, so that the forward and the backward models iteratively improve each other.”
- Learning by Semantic Similarity Makes Abstractive Summarization Better (2020, methodology): “Concurrent to our work, ProphetNet (Yan et al., 2020) and PEGASUS (Zhang et al., 2019b) also use pre-trained LMs to solve text summarization task.”
- Generating Abstractive Summaries with Finetuned Language Models (2019, methodology): “…language models based on the neural Trans-former architecture (Vaswani et al., 2017) have shown promising results in language understanding tasks (Houlsby et al., 2019; Devlin et al., 2018; Chronopoulou et al., 2019), but so far have had limited success in generation tasks (Zhang et al., 2019).”
- Structured information extraction from scientific text with large language models (2024, background): “Usecases for seq2seq are broad 35 and include machine translation 36 , answering generalfactualknowledgequestions 33,37 ,performingsimplearithmetic 33 , translating between languages 36,38 , summarizing text 28,39 , and chatbot applications 26,40 .”
- RefSum: Refactoring Neural Summarization (2021, methodology): “, 2018), twoStageRL (Zhang et al., 2019a), preSummAbs (Liu and Lapata, 2019), preSummAbsext (Liu and Lapata, 2019), HeterGraph (Wang et al.”
- Multi-stage Pretraining for Abstractive Summarization (2019, background): “Works such as (Edunov et al., 2019; Zhang et al., 2019) show that pre-trained networks can be used to improve performance on summarization benchmarks.”
- Re-evaluating Evaluation in Text Summarization (2020, background): “, 2019), twoStageRL (Zhang et al., 2019a), preSummAbs (Liu and Lapata, 2019b), preSummAbsext (Liu and Lapata, 2019b) BART (Lewis et al.”
- Multimodal Conditionality for Natural Language Generation (2021, background): “Unconditional language models can be adopted for text generation tasks such as language translation (Edunov et al., 2019), question answering (Su et al., 2019a), and summarization (Zhang et al., 2019).”
- Abstract Text Summarization with a Convolutional Seq2seq Model (2019, methodology): “Reference [27] was based on BERT to take advantage of the pre-trained language model in the seq2seq framework, and designed a two-stage decoder process to consider both sides’ context information of every word in a summary.”
- Extractive Summarization as Text Matching (2020, background): “Recent studies (Alyguliyev, 2009; Galanis and An-droutsopoulos, 2010; Zhang et al., 2019a) have attempted to build two-stage document summarization systems.”
- Encoder-Agnostic Adaptation for Conditional Language Generation (2019, methodology): “, 2019), BERT+Two-Stage from (Zhang et al., 2019), UniLM Large+ExtractiveLoss from (Dong et al.”
- Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “MLMs are generally used in downstream tasks by fine-tuning (Liu, 2019; Zhang et al., 2019), however, Zhu et al.”