Bottom-Up Abstractive Summarization

By Sebastian Gehrmann, Yuntian Deng, Alexander M. Rush. 2018

722 citations recorded by the source corpus.

Neural summarization produces outputs that are fluent and readable, but which can be poor at content selection, for instance often copying full sentences from the source document. This work explores the use of data-efficient content selectors to over-determine phrases in a source document that should be part of the summary. We use this selector as a bottom-up attention step to constrain the model to likely phrases. We show that this approach improves the ability to compress text, while still generating fluent summaries. This two-step process is both simpler and higher performing than other end-to-end content selection models, leading to significant improvements on ROUGE for both the CNN-DM and NYT corpus. Furthermore, the content selector can be trained with as little as 1,000 sentences making it easy to transfer a trained summarizer to a new domain.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Fine-Tuning Language Models from Human Preferences (2019, methodology): “This copying behavior emerged naturally from the data collection and training process; we did not use any explicit architectural mechanism for copying as in See et al. (2017); Gehrmann et al. (2018).”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “Following (See et al., 2017; Gehrmann et al., 2018), we use the F1 version of ROUGE (Lin, 2004) as the evaluation metric.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Summarization datasets thus have strong positional data biases that models pick up on and which lead to inflated results if the test set has the same biases (Gehrmann et al., 2018).”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, background): “The majority of research in abstractive summarization has focused on monolingual summarization in English (Gehrmann et al., 2018; Song et al., 2020; Narayan et al., 2018).”
Discriminative Adversarial Search for Abstractive Summarization (2020, positive): “This setup has practical applications in scenarios where limited data is available: indeed, learning to generate is harder than to discriminate and requires a large amount of examples (Gehrmann et al., 2018).”
PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, background): “…like neural machine translation (Bahdanau et al., 2015; Vaswani et al., 2017), abstractive summarization (Rush et al., 2015; See et al., 2017a; Gehrmann et al., 2018), generative question answering (QA) (Tan et al., 2017; Bi et al., 2019) and conversational response generation (Vinyals and…”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “Recent deep learning approaches to abstractive summarization naturally learn to integrate knowledge from the training data while generating an abstractive summary for a document (See et al., 2017; Gehrmann et al., 2018).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “BUTLER::B RAIN is a novel text-based game agent that generates high-level text actions in a token-by-token fashion akin to Natural Language Generation (NLG) approaches for dialogue (Sharma et al., 2017) and summarization (Gehrmann et al., 2018).”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, neutral): “72 Bottom-Up (Gehrmann et al., 2018) 41.”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, neutral): “38 Bottom-Up (Gehrmann et al., 2018) 41.”
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019, background): “This requires natural language generation techniques with high level of semantic understanding (Chopra et al., 2016; Rush et al., 2015; Khandelwal et al., 2019; Zhang et al., 2019; See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018).”
Text Summarization with Pretrained Encoders (2019, methodology): “…EN (See et al., 2017) 36.44 15.66 33.42 PTG EN +C OV (See et al., 2017) 39.53 17.28 36.38 DRM (Paulus et al., 2018) 39.87 15.82 36.90 B OTTOM U P (Gehrmann et al., 2018) 41.22 18.68 38.34 DCA (Celikyilmaz et al., 2018) 41 we focus on building a minimum-requirements model and these mechanisms may…”
Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (2019, methodology): “The Transformer has also been successfully applied to SDS (Gehrmann et al., 2018).”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, methodology): “The four tasks along with the datasets and models are as follows: • Summarization : Giganews story to head-line dataset and the pre-trained model from Gehrmann et al. (2018).”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, methodology): “…the summary, abstrac-tive (Rush et al., 2015; Paulus et al., 2017) where the important parts are paraphrased to form novel sentences, and hybrid (Gehrmann et al., 2018; Hsu et al., 2018; Chen and Bansal, 2018), combining the two methods by employing specialized extractive and abstractive…”
Pre-trained language model representations for language generation (2019, methodology): “We also compare to Gehrmann et al. (2018) who use a task-speciﬁc architecture compared to our generic sequence to sequence baseline.”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “…Pasunuru and Bansal, 2018; Kry´sci´nski et al., 2018), reinforcement learning strategies (Paulus et al., 2017; Narayan et al., 2018b; Dong et al., 2018; Wu and Hu, 2018), and hybrid extractive-abstractive models (Liu et al., 2018; Hsu et al., 2018; Gehrmann et al., 2018; Chen and Bansal, 2018).”