How well do you know your summarization datasets?

By Priyam Tejaswin, Dhruv Naik, Peng Liu. 2021

31 citations recorded by the source corpus.

State-of-the-art summarization systems are trained and evaluated on massive datasets scraped from the web. Despite their prevalence, we know very little about the underlying characteristics (data noise, summarization complexity, etc.) of these datasets, and how these affect system performance and the reliability of automatic metrics like ROUGE. In this study, we manually analyze 600 samples from three popular summarization datasets. Our study is driven by a six-class typology which captures different noise types (missing facts, entities) and degrees of summarization difficulty (extractive, abstractive). We follow with a thorough analysis of 27 state-of-the-art summarization models and 5 popular metrics, and report our key insights: (1) Datasets have distinct data quality and complexity distributions, which can be traced back to their collection process. (2) The performance of models and reliability of metrics is dependent on sample complexity. (3) Faithful summaries often receive low scores because of the poor diversity of references. We release the code, annotated data and model outputs.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Are We Done with MMLU? (2024, neutral): “In NLP, similar issues have been found in summarisation [20, 21] and natural language inference [22, 23, 24, 25] benchmarks.”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, background): “…have issues that seriously limit their usability: For instance, summaries from the popular scraped benchmark summarization dataset CNN/DailyMail (Nallapati et al., 2016) contain HTML artifacts, links to other news articles, and other types of noise (Kryscinski et al., 2019; Tejaswin et al., 2021).”