Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation
By Han Guo, Ramakanth Pasunuru, Mohit Bansal. 2018
143 citations recorded by the source corpus.
An accurate abstractive summary of a document should contain all its salient information and should be logically entailed by the input document. We improve these important aspects of abstractive summarization via multi-task learning with the auxiliary tasks of question generation and entailment generation, where the former teaches the summarization model how to look for salient questioning-worthy details, and the latter teaches the model how to rewrite a summary which is a directed-logical subset of the input document. We also propose novel multi-task architectures with high-level (semantic) layer-specific sharing across multiple encoder and decoder layers of the three tasks, as well as soft-sharing mechanisms (and show performance ablations and analysis examples of each contribution). Overall, we achieve statistically significant improvements over the state-of-the-art on both the CNN/DailyMail and Gigaword datasets, as well as on the DUC-2002 transfer setup. We also present several quantitative and qualitative analysis studies of our model’s learned saliency and entailment skills.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yejin Choi
- Caiming Xiong
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Noah A. Smith
- Hannaneh Hajishirzi
In Computer Vision and Pattern Recognition
- Yejin Choi
- Jianfeng Gao
- Luc Van Gool
- Mohit Bansal
- Trevor Darrell
- Dacheng Tao
- Caiming Xiong
- Xiaogang Wang
In Question Answering & Reading Comprehension
- Caiming Xiong
- Bryan McCann
- Yejin Choi
- Jianfeng Gao
- Greg Durrett
- Chandra Bhagavatula
- Andreas Vlachos
- Richard Socher
In Summarization
- Bryan McCann
- Fei Liu
- Yejin Choi
- Iryna Gurevych
- Greg Durrett
- Andreas Vlachos
- Richard Socher
- Hardy Hardy
Citations in the corpus
- Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “…Lapata, 2019; Zhang et al., 2019), attention and copying mechanisms (See et al., 2017; Cohan et al., 2018), and multi-objective training strategies (Guo et al., 2018; Pasunuru and Bansal, 2018), including reinforcement learning techniques (Kry´sci´nski et al., 2018; Dong et al., 2018; Wu and Hu,…”
- Neural Text Summarization: A Critical Evaluation (2019, methodology): “…advanced attention and copying mechanisms (See et al., 2017; Tan et al., 2017; Cohan et al., 2018), multi-task and multi-reward training techniques (Guo et al., 2018; Pasunuru and Bansal, 2018; Kry´sci´nski et al., 2018), reinforcement learning strategies (Paulus et al., 2017; Narayan et al.,…”
- Continual and Multi-Task Architecture Search (2019, neutral): “…is primarily used to improve the generalization performance of a task by leveraging knowledge from related tasks (Caru-ana, 1998; Collobert and Weston, 2008; Girshick, 2015; Luong et al., 2015; Ruder et al., 2017; Au-genstein et al., 2018; Guo et al., 2018; Oh et al., 2017; Ruder and Plank, 2017).”
- Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “…et al., 2018; Pan et al., 2019), simplifying questions to make them more easily answerable (Buck et al., 2018; Talmor and Berant, 2018; Perez et al., 2020), to using questions as means for other purposes such as sentence representation and summarization (Guo et al., 2018; Potash and Suleman, 2019).”
- Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (2019, methodology): “Moreover, Guo et al. (2018) and Pasunuru and Bansal (2018) proposed to use NLI-based loss functions or multi-task learning with NLI for document summarization.”
- Guiding Extractive Summarization with Question-Answering Rewards (2019, background): “Abstractive systems often adopt the encoder-decoder architecture with an attention mechanism (Rush et al., 2015; Nallapati et al., 2016; Paulus et al., 2017; Guo et al., 2018; Gehrmann et al., 2018; Lebanoff et al., 2018; Ce-likyilmaz et al., 2018).”
- GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Recent progress in neural approaches to building semantically constrained text generation systems has shown tremendous improvements in this direction (Liu and Lapata, 2019; Guo et al., 2018; Durmus et al., 2020; Wang et al., 2020).”
- SummEval: Re-evaluating Summarization Evaluation (2020, methodology): “M14 - Multi-task (Ent + QG ) (Guo et al., 2018) propose question generation and entailment generation as auxiliary tasks in a multi-task framework along with a corresponding multi-task architecture.”
- Multi-task Learning based Pre-trained Language Model for Code Completion (2020, unknown): “Multi-task learning has been successfully used in many fields including natural language processing [10, 14, 31], speech recognition [9] and computer vision [32, 34].”
- Concept Pointer Network for Abstractive Summarization (2019, methodology): “Abstractive summarization (ABS) has gained overwhelming success owing to a tremendous development of sequence-to-sequence (seq2seq) model and its variants (Rush et al., 2015; Chopra et al., 2016; Paulus et al., 2017; Guo et al., 2018; Gao et al., 2019).”
- HighRES: Highlight-based Reference-less Evaluation of Summarization (2019, methodology): “…judgment to capture all linguistic qualities of the summary (Hsu et al., 2018; Kryściński et al., 2018; Narayan et al., 2018b; Song et al., 2018; Guo et al., 2018); we group them under “Fluency” in Table 1 with an exception of “Clarity” which was evaluated in the DUC evaluation campaigns…”
- Controlling the Amount of Verbatim Copying in Abstractive Summarization (2019, unknown): “They include lvt5k1sent (Nallapati et al. 2016), Multi-Task w/ Entailment (Pa- sunuru and Bansal 2018), SEASS (Zhou et al., 2017), DRGD (Li et al. 2017), EntailGen+QuesGen (Guo, Pasunuru, and Bansal 2018), PG Networks (See, Liu, and Manning 2017), Struct+2Way+Relation (Song, Zhao, and Liu 2018), R3S”
- The Summary Loop: Learning to Write Abstractive Summaries Without Examples (2020, background): “Recent work around measuring factuality in generated text, using Natural Language Inference (Guo et al., 2018) or rule-based fact extraction (Zhang et al., 2019b) becomes increasingly important with summaries that are more abstractive.”
- Evaluating Factuality in Generation with Dependency-level Entailment (2020, background): “Other work additionally incorporates a reward for generating summaries entailed by the the input (Li et al., 2018; Pasunuru and Bansal, 2018).”