Neural Document Summarization by Jointly Learning to Score and Select Sentences

By Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, M. Zhou, Tuo Zhao. 2018

346 citations recorded by the source corpus.

Sentence scoring and sentence selection are two main steps in extractive document summarization systems. However, previous works treat them as two separated subtasks. In this paper, we present a novel end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences. It first reads the document sentences with a hierarchical encoder to obtain the representation of sentences. Then it builds the output summary by extracting sentences one by one. Different from previous methods, our approach integrates the selection strategy into the scoring model, which directly predicts the relative importance given previously selected sentences. Experiments on the CNN/Daily Mail dataset show that the proposed framework significantly outperforms the state-of-the-art extractive summarization models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Furu Wei
4. Graham Neubig
5. Xipeng Qiu
6. Caiming Xiong
7. Pengfei Liu
8. Percy Liang

In Natural Language Processing

1. Graham Neubig
2. Xipeng Qiu
3. Jianfeng Gao
4. Pengfei Liu
5. Furu Wei
6. Mohit Bansal
7. Caiming Xiong
8. Yang Liu

Citations in the corpus

• Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “We identify the approximate extractive oracle for training using the method described in (Zhou et al., 2018).”
• Fine-tune BERT for Extractive Summarization (2019, background): “• N EUSUM (Zhou et al., 2018) is the state-of-the-art extractive system that jontly score and select sentences.”
• Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, neutral): “5 (Zhou et al., 2018) NA 41.”
• Bottom-Up Abstractive Summarization (2018, background): “Our naive sentence-extractor performs slightly worse than the highest reported extractive score by Zhou et al. (2018) that is specifically trained to score combinations of sentences.”
• Text Summarization with Pretrained Encoders (2019, methodology): “N EU S UM (Zhou et al., 2018) scores and selects sentences jointly and represents the state of the art in extractive summarization.”
• Neural Text Summarization: A Critical Evaluation (2019, methodology): “It is commonly used for both abstractive (See et al., 2017; Paulus et al., 2017; Kry´sci´nski et al., 2018) and extractive (Dong et al., 2018; Wu and Hu, 2018; Zhou et al., 2018) neural summarization.”
• BARTScore: Evaluating Generated Text as Text Generation (2021, neutral): “Extractive Systems E1: BanditSum [11]; E2: Refresh [48]; E3: NeuSum [80]; E4: LSTMPN-RL [79]; E5: BERT-TF-SL [79]; E6: BERT-TF-PN [79]; E7: BERT-LSTM-PN-RL [79]; E8: BERT-LSTM-PN [79]; E9: HeterGraph [68]; E10: MatchSum [78].”
• Guiding Extractive Summarization with Question-Answering Rewards (2019, background): “Neural extractive summarization has focused primarily on extracting sentences (Nallapati et al., 2017; Cao et al., 2017; Isonuma et al., 2017; Tarn-pradab et al., 2017; Zhou et al., 2018; Kedzie et al., 2018).”
• HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization (2019, methodology): “The architecture is widely adopted in recent neural ex-tractive models and is extended with reinforcement learning (Narayan et al., 2018; Dong et al., 2018), latent variable models (Zhang et al., 2018), joint scoring (Zhou et al., 2018) and iterative document representation (Chen et al., 2018).”
• GSum: A General Framework for Guided Neural Abstractive Summarization (2020, neutral): “Modern techniques for text summarization generally can be categorized as either extractive meth-ods (Nallapati et al., 2017; Narayan et al., 2018; Zhou et al., 2018), which identify the most suitable words or sentences from the input document and concatenate them to form a summary, or ab-stractive…”
• Extractive Summarization as Text Matching (2020, methodology): “B ANDIT S UM (Dong et al., 2018) 41.50 18.70 37.60 N EU S UM (Zhou et al., 2018) 41.59 19.01 37.98 J ECS (Xu and Durrett, 2019) 41.70 18.50 37.90 H I B ERT (Zhang et al., 2019b) 42.37 19.95 38.83 P N B ERT (Zhong et al., 2019a) 42 0 .”
• Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation (2019, background): “An advantage of this approach over other extractive summarizers (e.g. Zhou et al. (2018)) is that those often model dependencies between selected sentences, which is not applicable to this problem since we only aim to extract a single sentence.”
• On Generating Extended Summaries of Long Documents (2020, background): “In the past few years, there has been a significant progress on both extractive (e.g., Nallapati, Zhai, and Zhou 2017; Zhou et al. 2018; Liu and Lapata 2019; Xu et al. 2020; Jia et al. 2020) and abstractive (e.”
• Re-evaluating Evaluation in Text Summarization (2020, positive): “, 2018), NeuSum (Zhou et al., 2018), HIBERT (Zhang et al.”
• SummEval: Re-evaluating Summarization Evaluation (2020, background): “M1 - NEUSUM (Zhou et al., 2018) jointly scores and selects sentences by first building a hierarchical representation of a document and considering the partially outputted summary at each time step.”
• A Survey of Deep Learning for Scientific Discovery (2020, background): “…might be of interest are structured prediction , where the predicted output has some kind of structure, from tree structures (in e.g. parsing) [28, 246] to short, executable computer program structure [271] and summarization , where passages of text are summarized by a neural network [130, 273].”