Automatically Assessing Machine Summary Content Without a Gold Standard

By Annie Louis, Ani Nenkova. 2013

190 citations recorded by the source corpus.

The most widely adopted approaches for evaluation of summary content follow some protocol for comparing a summary with gold-standard human summaries, which are traditionally called model summaries. This evaluation paradigm falls short when human summaries are not available and becomes less accurate when only a single model is available. We propose three novel evaluation techniques. Two of them are model-free and do not rely on a gold standard for the assessment. The third technique improves standard automatic evaluations by expanding the set of available model summaries with chosen system summaries.We show that quantifying the similarity between the source text and its summary with appropriately chosen measures produces summary scores which replicate human assessments accurately. We also explore ways of increasing evaluation quality when only one human model summary is available as a gold standard. We introduce pseudomodels, which are system summaries deemed to contain good content according to automatic evaluation. Combining the pseudomodels with the single human model to form the gold-standard leads to higher correlations with human judgments compared to using only the one available model. Finally, we explore the feasibility of another measure—similarity between a system summary and the pool of all other system summaries for the same input. This method of comparison with the consensus of systems produces impressively accurate rankings of system summaries, achieving correlation with human rankings above 0.9.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Discriminative Adversarial Search for Abstractive Summarization (2020, background): “For summarization, Louis & Nenkova (2013) showed how this issue gets even more relevant when few gold references are given.”
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024, methodology): “Reference-free evaluation metrics Reference-free evaluation metrics have a long history, including classic methods (Louis & Nenkova, 2013) and more recent neural supervised learning methods (Kryscinski et al., 2020; Sinha et al., 2020; Goyal & Durrett, 2020).”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “…need for costly human references in the evaluation of text generation systems, and instead base evaluation scores on source texts and system predictions only, which would allow for ‘next-level’, unsupervised (in a double sense) and unlimited evaluation (Louis and Nenkova, 2013; B¨ohm et al., 2019).”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, background): “For these reasons, n -gram based metrics, such as ROUGE (Lin, 2004), are known to poorly reﬂect human preference (Louis and Nenkova, 2013; Novikova et al., 2017; Paulus et al., 2017; Bhandari et al., 2020).”
Question Answering as an Automatic Evaluation Metric for News Article Summarization (2019, methodology): “Louis and Nenkova (2013) further shows that ROUGE correlation to manual scores tends to drop when reducing the number of reference summaries.”
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022, methodology): “Automatic evaluation metrics are the most common method that researchers use to quickly and cheaply approximate how humans would rate the quality of a summarization system (Lin, 2004; Louis and Nenkova, 2013; Zhao et al., 2019; Zhang et al., 2020; Deutsch et al., 2021a, among others).”
Fill in the BLANC: Human-free quality estimation of document summaries (2020, methodology): “One possible route to a better automatic method for summary quality estimation is to train a model on document summaries annotated with human quality scores [13, 14, 15].”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, methodology): “nnotations to match and evaluate the SCUs in generated summaries and their ground truth summaries. Some recent works are focusing on the construction of Pyramid [38, 49, 99, 117, 138]. Responsiveness [76] measures the content selection and linguistic quality of summaries by directly rating scores. A”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “For example, many have argued that human evaluation is the ‘gold standard’ for machine translation [50, 69, 87], but waiting for humans to evaluate translations takes much longer and is much more expensive than computing B LEU [111], an automatic metric.”
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022, background): “Researchers also attempted to develop meth-ods for evaluating reference-free model summaries (Louis and Nenkova, 2013; Xenouleas et al., 2019).”
Text summarization using unsupervised deep learning (2017, background): “The selection of a metric for comparison of the generated summary with the human summary is still an open question (Louis and Nenkova, 2013).”
SMART: Sentences as Basic Units for Text Evaluation (2022, background): “…us-ing the six matching functions described in Sec-2 While they claimed to report system-level correlation, the BARTScore paper (Yuan et al., 2021) actually calculated summary -level correlation (Louis and Nenkova, 2013), where they ﬁrst get correlation for each system and then take the average.”
Learning to Score System Summaries for Better Content Selection Evaluation. (2017, methodology): “only on the candidate summary and the source documents (Lin et al., 2006; Louis and Nenkova, 2013).”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “The source-based metrics directly estimate the quality of the text through source and are useful when reference is noisy or unavailable (Louis and Nenkova, 2013; Kepler et al., 2019).”
A Decade of Automatic Content Evaluation of News Summaries: Reassessing the State of the Art (2013, background): “Recent work has shown that this is unlikely to be the case because the collection of summaries from several systems indicates better what content is important than the single best summary (Louis and Nenkova, 2013).”
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems (2017, background): “Louis and Nenkova (2013) use the distribution similarity between input and generated summaries to evaluate the quality of summary content.”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, background): “…to the machine translation cases highlighted in the introduction, reference-free evaluations have been proposed for other generation tasks, including summarization (Louis and Nenkova, 2013; Peyrard and Gurevych, 2018; Sun and Nenkova, 2019) and dialogue (Tao et al., 2018; Mehri and Eskenazi, 2020).”
Exploring Consensus in Machine Translation for Quality Estimation (2014, background): “Louis and Nenkova (2013) apply pseudo-references for summary evaluation.”