Learning to Score System Summaries for Better Content Selection Evaluation.

By Maxime Peyrard, Teresa Botschen, Iryna Gurevych. 2017

101 citations recorded by the source corpus.

The evaluation of summaries is a challenging but crucial task of the summarization field. In this work, we propose to learn an automatic scoring metric based on the human judgements available as part of classical summarization datasets like TAC-2008 and TAC-2009. Any existing automatic scoring metrics can be included as features, the model learns the combination exhibiting the best correlation with human judgments. The reliability of the new metric is tested in a further manual evaluation where we ask humans to evaluate summaries covering the whole scoring spectrum of the metric. We release the trained metric as an open-source tool.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, background): “Louis and Nenkova (2013), Peyrard et al. (2017) and Peyrard and Gurevych (2018) build feature-rich regression models to learn a summary evaluation metric directly from the human judgement scores (Pyramid and Responsiveness) provided in the TAC’08 and ’09 datasets 1 .”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “…on n-gram co-occurrence statistics, which are originally proposed for diagnostic evaluation of systems but not capable of evaluating text quality (Reiter, 2018), as they are not designed to measure if, and to what extent, the system and reference texts with distinct surface forms have conveyed…”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, background): “The later-emerged learning-based approaches aggregate multiple features to regress on human-rated quality scores for different tasks (Lowe et al., 2017; Peyrard et al., 2017; Sellam et al., 2020).”
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2021, background): “S3 (Peyrard et al., 2017) trains a regression model that takes the evaluation scores of several existing metrics and many hand-crafted features as input, and learns the best combination of them to approximate human summary judgments.”
SummEval: Re-evaluating Summarization Evaluation (2020, background): “CHRF, and METEOR tended to favor extractive models, assigning highest scores to their outputs.”
Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries (2021, background): “…2008), BEwT-E (Tratz and Hovy, 2008), MeMoG (Giannakopou-los and Karkaletsis, 2011), METEOR (Denkowski and Lavie, 2014), MoverScore (Zhao et al., 2019), NPowER (Giannakopoulos and Karkaletsis, 2013), PyrEval (Gao et al., 2019), QAEval (Deutsch et al., 2021), ROUGE-2, and S 3 (Peyrard et al., 2017).”
Re-evaluating Evaluation in Text Summarization (2020, background): “Additionally, many previous works (Novikova et al., 2017; Peyrard et al., 2017; Chaganty et al., 2018) show that metrics have much lower correlations at comparing summaries than systems.”
Objective Function Learning to Match Human Judgements for Optimization-Based Summarization (2018, background): “The last metric is S3 (Peyrard et al., 2017), a combination of several existing metrics trained explicitly to maximize its correlation with human judgments.”
Studying Summarization Evaluation Metrics in the Appropriate Scoring Range (2019, methodology): “S3 : a metric trained explicitly to maximize its correlation with manual Pyramid annotations (Peyrard et al., 2017).”
Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation (2021, methodology): “Fabbri et al. (2021) scored each summary using these evaluation metrics: ROUGE (Lin 2004), ROUGE-WE (Ng and Abrecht 2015), S 3 (Peyrard, Botschen, and Gurevych 2017), BertScore (Zhang et al. 2020), MoverScore (Zhao et al. 2019), Sentence Mover’s Similarity (SMS) (Clark, Celikyilmaz, and Smith…”
Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics (2020, background): “…strategies: (i) assessing how well each metric correlates with human judgments (Lin, 2004; Ng and Abrecht, 2015; Louis and Nenkova, 2013; Peyrard et al., 2017; Bhandari et al., 2020), which requires procuring manual annotations that are expensive and time-consuming, and (ii) measuring…”
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020, background): “Peyrard et al. (2017); Peyrard and Gurevych (2018) propose to learn an evaluation function from Pyramid and Responsiveness scores, by using classic supervised learning methods with hand-crafted features.”