Content Selection in Deep Learning Models of Summarization

By Chris Kedzie, K. McKeown, Hal Daumé Iii. 2018

156 citations recorded by the source corpus.

We carry out experiments with deep learning models of summarization across the domains of news, personal stories, meetings, and medical articles in order to understand how content selection is performed. We find that many sophisticated features of state of the art extractive summarizers do not improve performance over simpler models. These results suggest that it is easier to create a summarizer for a new domain than previous work suggests and bring into question the benefit of deep learning models for summarization for those domains that do have massive datasets (i.e., news). At the same time, they suggest important questions for new research in summarization; namely, new forms of sentence representations or external knowledge sources are needed that are better suited to the sumarization task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, Kedzie et al. (2018) demonstrate that controlling for positional bias drastically decreases model performances to the point where deep learning based models barely outperform much simpler approaches.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “PubMed, Arxiv (Kedzie et al., 2018; Cohan et al., 2018) 21.”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, neutral): “0 (Kedzie et al., 2018) NA 39.”
Neural Text Summarization: A Critical Evaluation (2019, background): “Kedzie et al. (2018) showed that in the case of news articles, the layout bias dominates the learning signal for neural models.”
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks (2018, methodology): “Moreover, if abstractive methods are trained on these datasets, they may not show much abstraction (See et al., 2017), because they are implicitly forced to learn structural patterns (Kedzie et al., 2018).”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, neutral): “The primary evaluation metric then and now is ROUGE (Lin, 2004), though much work has demonstrated the limited ability of ROUGE and its relatives to evaluate summaries (Dorr et al., 2004; Liu and Liu, 2009; Kedzie et al., 2018, i.a.).”
Guiding Extractive Summarization with Question-Answering Rewards (2019, background): “Neural extractive summarization has focused primarily on extracting sentences (Nallapati et al., 2017; Cao et al., 2017; Isonuma et al., 2017; Tarn-pradab et al., 2017; Zhou et al., 2018; Kedzie et al., 2018).”
SummEval: Re-evaluating Summarization Evaluation (2020, background): “In (Kedzie et al., 2018) the authors examined the influence of various model components on the quality of content selection.”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “, 2017) and 14 abstractive systems (Zhong et al., 2020; Wang et al., 2020; Zhong et al., 2019; Liu and Lapata, 2019; Zhou et al., 2018; Narayan et al., 2018; Dong et al., 2019; Kedzie et al., 2018; Zhou et al., 2018).”
GSum: A General Framework for Guided Neural Abstractive Summarization (2020, methodology): “3 We follow Kedzie et al. (2018) to preprocess and split the dataset.”
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization (2021, methodology): “Recent work in summarization analysis has looked at the problems of the ﬁeld in isolation, focusing on: models (Kedzie et al., 2018; Kry´s-ci´nski et al., 2019, 2020), data (Zhong et al., 2019; Jung et al., 2019), and evaluation (Fabbri et al., 2020; Steen and Markert, 2021).”
BookSum: A Collection of Datasets for Long-form Narrative Summarization (2021, background): “The lead-bias present in news articles being one example of such effects (Kedzie et al., 2018; Kryściński et al., 2019).”
Exploring Content Selection in Summarization of Novel Chapters (2020, methodology): “Some previous work in news summarization has focused on iteratively picking the best article sentence with respect to the summary, in order to get the gold extracts (Nallapati et al., 2017; Kedzie et al., 2018), using ROUGE between the set of selected sentences and the target summary.”
Towards Summarization for Social Media - Results of the TL;DR Challenge (2019, background): “Furthermore, the common structure of news articles 1 introduces bias, since the lead paragraph usually already captures the most relevant information (Kedzie et al., 2018).”
Discourse-Aware Neural Extractive Text Summarization (2019, background): “Text summarization, especially news summarization, usually suffers from the well-known ‘position bias’ issue (Kedzie et al., 2018), where most of the key information is described at the very beginning Algorithm 1 Construction of the Coreference Graph G C . of the document.”
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021, methodology): “Here we do not use Transformer because previous work (Kedzie et al., 2018) shows that this component does not matter too much for the final performance.”
Robustness Gym: Unifying the NLP Evaluation Landscape (2021, methodology): “We propose and implement 5 subpopulations that capture summary abstractiveness, content distillation, information dispersion [Grusky et al., 2018], positional bias, and information reordering [Kedzie et al., 2018].”
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (2019, background): “A similar work to us is Kedzie et al. (2018), which studies how deep learning models perform context selection in terms of several typical summarization architectures, and domains.”
Extractive Summarization of Long Documents by Combining Global and Local Context (2019, methodology): “Again following (Kedzie et al., 2018), we train each model for 50 epochs, and the best model is selected with early stopping on the validation set according to Rouge-2 F-score.”
BRIO: Bringing Order to Abstractive Summarization (2022, methodology): “We follow Kedzie et al. (2018) for data preprocessing and splitting, and use the associated archival abstracts as the summaries.”