Controllable Abstractive Summarization

By Angela Fan, David Grangier, Michael Auli. 2017

322 citations recorded by the source corpus.

Current models for document summarization disregard user preferences such as the desired length, style, the entities that the user might be interested in, or how much of the document the user has already read. We present a neural summarization model with a simple but effective mechanism to enable users to specify these high level attributes in order to control the shape of the final summaries to better suit their needs. With user input, our system can produce high quality summaries that follow user preferences. Without user input, we set the control variables automatically – on the full text CNN-Dailymail dataset, we outperform state of the art abstractive systems (both in terms of F1-ROUGE1 40.38 vs. 39.53 F1-ROUGE and human evaluation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “…we set beam size as 5, remove duplicated trigrams in beam search, and tuned the model with min-len, max-len, length penalty on the validation set (Fan et al., 2017 Summarization To provide a comparison with the state-of-the-art in summarization, we present results on two summarization datasets,…”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “, 2017; Baevski and Auli, 2019), abstractive document summarization (Fan et al., 2018a; Liu et al., 2018; Narayan et al., 2018), story generation (Fan et al.”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, background): “…has consistently attracted attention (Cheng and Lapata, 2016; Durrett et al., 2016; Nallapati et al., 2016, 2017; See et al., 2017; Tan and Wan, 2017; Narayan et al., 2017; Fan et al., 2017; Paulus et al., 2018; Pasunuru and Bansal, 2018; Celikyilmaz et al., 2018; Narayan et al., 2018a,b).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “…a good estimate of the probability that a generative model would produce the gold label, such results may be actually quite rare under beam search (Fan, Grangier, and Auli, 2018; Holtzman et al., 2019; Welleck et al., 2020), and not representative of an actual generation of a model under beam…”
ELI5: Long Form Question Answering (2019, methodology): “nique commonly used in multi-sentence summarization (Paulus et al., 2017; Fan et al., 2018).”
Wizard of Wikipedia: Knowledge-Powered Conversational agents (2018, methodology): “All generative models employ BPE encoding (Sennrich et al., 2016), which we found effective at enabling generators to copy rare words from Wikipedia sentences (Fan et al., 2018).”
Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, background): “(4)Other approaches might also help with this problem such as using an explicit copy mechanism or to use BPE tokenization (Fan et al., 2017), but we leave those for future work.”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “…model directly for controllable generation (Chan et al., 2021; Madotto et al., 2020; Keskar et al., 2019; Ziegler et al., 2019; Rajani et al., 2019; Fan et al., 2018; Ficler and Goldberg, 2017; Yu et al., 2017; Hu et al., 2017) or using a discriminator to guide decoding (Ghazvininejad et al.,…”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, methodology): “els learn to associate the control token with the desired text properties (Fan et al., 2017), then modifying the control tokens during inference to produce the desired result.”
Neural Text Generation with Unlikelihood Training (2019, background): “Some other lines of work of note are the use of generative control (Kikuchi et al., 2016; Fan et al., 2017; Peng et al., 2018; See et al., 2019) and the retrieve-and-reﬁne setting (Weston et al., 2018; Guu et al., 2018).”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “Also, some clear discrepancies between controlled generation and prompt-based text generation are: (1) In controlled generation work, the control is generally performed over the style or content of the generations [23, 27] while the underlying task remains the same.”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “We generate long answers using beam search with beam size 5 and apply 3-gram blocking (Fan et al., 2017).”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “6 When generating summaries, we follow standard practice in tuning the maximum output length, disallowing repeating the same trigram, and we apply a stepwise length penalty (Paulus et al., 2017; Fan et al., 2017; Wu et al., 2016).”
What makes a good conversation? How controllable attributes affect human judgments (2019, methodology): “Given these constraints, we study two con-trol methods: conditional training (Fan et al., 2018; Kikuchi et al., 2016; Peng et al., 2018) and weighted decoding (Ghazvininejad et al., 2017).”
Multi-Dimensional Gender Bias Classification (2020, background): “This has been applied to a variety of different cases, including generating text of different lengths (Fan et al., 2017), generating questions in chit-chat (See et al.”
Pre-trained language model representations for language generation (2019, methodology): “When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram (Paulus et al., 2017; Fan et al., 2017).”
Deep Communicating Agents for Abstractive Summarization (2018, background): “…for summarization as well (Rush et al., 2015; Zeng et al., 2016; Nallapati et al., 2017), and pointer networks (Vinyals et al., 2015a), in particular, help address redundancy and saliency in generated summaries (Cheng and Lapata, 2016; See et al., 2017; Paulus et al., 2018; Fan et al., 2017).”
Pre-training via Paraphrasing (2020, methodology): “We use a beam of size 6 and block repeated n-grams of length 8 [Fan et al., 2017].”
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting (2018, background): “14 We also outperform the results of Fan et al. (2017) on both original and anonymized dataset versions.”
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019, background): “In terms of improving repetition and speciﬁcity, a recent alternative approach is that of control (Fan et al., 2018; Ficler and Goldberg, 2017; Ghazvininejad et al., 2017; See et al., 2019).”