Studying Summarization Evaluation Metrics in the Appropriate Scoring Range

By Maxime Peyrard. 2019

60 citations recorded by the source corpus.

In summarization, automatic evaluation metrics are usually compared based on their ability to correlate with human judgments. Unfortunately, the few existing human judgment datasets have been created as by-products of the manual evaluations performed during the DUC/TAC shared tasks. However, modern systems are typically better than the best systems submitted at the time of these shared tasks. We show that, surprisingly, evaluation metrics which behave similarly on these datasets (average-scoring range) strongly disagree in the higher-scoring range in which current systems now operate. It is problematic because metrics disagree yet we can’t decide which one to trust. This is a call for collecting human judgments for high-scoring summaries as this would resolve the debate over which metrics to trust. This would also be greatly beneficial to further improve summarization systems and metrics alike.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “As a result of this, Peyrard (2019) find that summariza- tion metrics with previously reported high correlations with humans disagree with one another when tasked to compare high quality summaries, revealing their fragility.”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “Peyrard (2019b) further show that lexical metrics cannot be trusted because Machine Translation (zh − en) Machine Translation (zh − en) Machine Translation (zh-en) Figure 3 : Correlation in similar language (de-en) and distant language (zh-en) pair, where bordered area shows correlations between…”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, background): “…design of reliable metrics for natural language generation (NLG) systems is very challenging, and still an open research problem: Novikova et al. (2017); Peyrard (2019) showed that current metrics do not correlate well with human judgments, and argued for the development of new evaluation metrics.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “Other studies analyze a broader set of metrics (Peyrard, 2019; Bhandari et al., 2020; Deutsch and Roth, 2020; Fabbri et al., 2022a; Gabriel et al., 2021; Kasai et al., 2022b), including those specific to factual consistency evaluation (Kryscinski”
Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics (2020, methodology): “Following (Peyrard, 2019) we combine the two and refer to the joined dataset as TAC.”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, unknown): “…two arbitrary measures µ ∈ M1+(X ) and ν ∈ M1+(Y) is defined through the resolution of the Monge-Kantorovitch mass transportation problem (Villani, 2003; Peyré and Cuturi, 2019): W(µ, ν) = min π ∈ U(µ,ν) (∫ X×Y ||x− y||pdπ(x, y) )1/p , (1) where U(µ, ν) = {π ∈ M1+(X × Y) :∫ π(x, y)dy = µ(x); ∫…”
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022, result): “Peyrard (2019) argues that current evaluation metrics do not work as well when they are used to evaluate high-performing systems compared to those which were evaluated in DUC/TAC.”
How well do you know your summarization datasets? (2021, unknown): “For metric reliability and meta-analysis, we build on correlation analysis presented in earlier works (Peyrard, 2019; Bhandari et al., 2020; Fabbri et al., 2020).”
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022, methodology): “We asked humans to blindly evaluate the quality of the summaries generated by two models and make a judgement on which summary was better as suggested by (Peyrard, 2019) as well.”
Intrinsic Evaluation of Summarization Datasets (2020, methodology): “From an evaluation perspective, several works have discussed the shortcomings of ROUGE/automated evaluation (Liu and Liu, 2008; Chaganty et al., 2018; Hashimoto et al., 2019; Peyrard, 2019b) as well proposed alternative metrics for summarization or natural language generation more broadly (Clark et ”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “, 2020) are unreliable when small improvements are reported (Peyrard, 2019; Fabbri et al., 2021); however large differences, on the order of say 5 ROUGE points or greater, are considered to”
InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation (2021, methodology): “5 ) which has been reported in Peyrard (2019).”
SAFEval: Summarization Asks for Fact-based Evaluation (2021, background): “…design of reliable metrics for natural language generation (NLG) systems is very challenging, and still an open research problem: Novikova et al. (2017); Peyrard (2019) showed that current metrics do not correlate well with human judgments, and argued for the development of new evaluation metrics.”