Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models

By Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, Junji Tomita. 2020

32 citations recorded by the source corpus.

Pre-trained sequence-to-sequence (seq-to-seq) models have significantly improved the accuracy of several language generation tasks, including abstractive summarization. Although the fluency of abstractive summarization has been greatly improved by fine-tuning these models, it is not clear whether they can also identify the important parts of the source text to be included in the summary. In this study, we investigated the effectiveness of combining saliency models that identify the important parts of the source text with the pre-trained seq-to-seq models through extensive experiments. We also proposed a new combination model consisting of a saliency model that extracts a token sequence from a source text and a seq-to-seq model that takes the sequence as an additional input text. Experimental results showed that most of the combination models outperformed a simple fine-tuned seq-to-seq model on both the CNN/DM and XSum datasets even if the seq-to-seq model is pre-trained on large-scale corpora. Moreover, for the CNN/DM dataset, the proposed combination model exceeded the previous best-performed model by 1.33 points on ROUGE-L.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “It could also be keywords [112], relation triples [154] or even highlighted phrases or sentences [34, 78] to plan the content of generated texts.”
CTRLsum: Towards Generic Controllable Text Summarization (2020, methodology): “Other work mostly focus on a specific control task in limited settings (Saito et al., 2020a; Tan et al., 2020; Suhara et al., 2020), yet CTRLsum aims for a more generic framework.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “CIT [207] employed RoBERTa [151] to extract important words and sentences from the input document.”
GSum: A General Framework for Guided Neural Abstractive Summarization (2020, background): “A concurrent work by Saito et al. (2020) propose to extract keywords or highlighted sentences using saliency models and feed them to summarization models.”