Intrinsic Evaluation of Summarization Datasets

By Rishi Bommasani, Claire Cardie. 2020

74 citations recorded by the source corpus.

High quality data forms the bedrock for building meaningful statistical models in NLP. Consequently, data quality must be evaluated either during dataset construction or *post hoc*. Almost all popular summarization datasets are drawn from natural sources and do not come with inherent quality assurance guarantees. In spite of this, data quality has gone largely unquestioned for many of these recent datasets. We perform the first large-scale evaluation of summarization datasets by introducing 5 intrinsic metrics and applying them to 10 popular datasets. We find that data usage in recent summarization research is sometimes inconsistent with the underlying properties of the data. Further, we discover that our metrics can serve the additional purpose of being inexpensive heuristics for detecting generically low quality examples.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “the abstractiveness or compression-ratio in summarization datasets (Bommasani and Cardie, 2020).”
Measuring Data (2022, background): “…the properties of a collection of data or a dataset (Lai et al., 2020); “intrinsic metrics”, to quantify the properties of a dataset (Grusky et al., 2018; Bommasani and Cardie, 2020a); “statistical tendencies”, described as “present” in datasets (Meister and Cotterell, 2021a); “statistical…”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, methodology): “For the news domain dataset, we decided against using the popular CNN/Dailymail dataset since its reference summaries tend to be very extractive (Kedzie et al., 2018; Bommasani and Cardie, 2020), making it a poor choice for studying faithfulness in abstractive summarization.”
How well do you know your summarization datasets? (2021, unknown): “Large-scale automatic intrinsic dataset evaluation has been explored with some promising results (Bommasani and Cardie, 2020).”
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages (2021, background): “With the above backdrop, several works (Narayan et al., 2018; Grusky et al., 2018; Bommasani and Cardie, 2020) have proposed many automatic metrics to quantify important features of abstractive summaries (e.g., novel words, abstractivity, compression, and redundancy).”