HighRES: Highlight-based Reference-less Evaluation of Summarization

By Hardy Hardy, Andreas Vlachos. 2019

82 citations recorded by the source corpus.

There has been substantial progress in summarization research enabled by the availability of novel, often large-scale, datasets and recent advances on neural network-based approaches. However, manual evaluation of the system generated summaries is inconsistent due to the difficulty the task poses to human non-expert readers. To address this issue, we propose a novel approach for manual evaluation, Highlight-based Reference-less Evaluation of Summarization (HighRES), in which summaries are assessed by multiple annotators against the source document via manually highlighted salient content in the latter. Thus summary assessment on the source document by human judges is facilitated, while the highlights can be used for evaluating multiple systems. To validate our approach we employ crowd-workers to augment with highlights a recently proposed dataset and compare two state-of-the-art systems. We demonstrate that HighRES improves inter-annotator agreement in comparison to using the source document directly, while they help emphasize differences among systems that would be ignored under other evaluation approaches.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

On Faithfulness and Factuality in Abstractive Summarization (2020, background): “…summary of a news article; these shorter summaries are relatively easier to annotate and analyze than longer summaries such as story highlights from the CNN/Dailymail dataset (Hermann et al., 2015) or abstracts from the NY Times (Sandhaus, 2008) or the WikiSum (Liu et al., 2018) dataset.”
Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022, methodology): “…around 192 variants of ROUGE have been proposed (Graham, 2015) including ROUGE with word embedding (Ng and Abrecht, 2015) and synonym (Ganesan, 2018), graph-based lexical measurement (ShaﬁeiBavani et al., 2018), Vanilla ROUGE (Yang et al., 2018) and highlight-based ROUGE (Hardy et al., 2019).”
Fill in the BLANC: Human-free quality estimation of document summaries (2020, background): “to pay higher attention (Hardy et al., 2019), as well as splitting quality scores into multiple dimensions such as fluency, informativeness, and factual correctness (Kryściński et al.”
Understanding Points of Correspondence between Sentences for Abstractive Summarization (2020, methodology): “Similar to (Hardy et al., 2019), we report Fleiss’ Kappa judged on each word (highlighted or not), yielding substantial inter-annotator agreement (κ=0.58) for annotating points of correspondence.”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “The majority of summarization research now reports results from a human study in addition to automatic metrics, but there is a general lack of consensus on what dimensions to evaluate, task design, and other factors (Hardy et al., 2019).”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “However, papers often differ in their preferred evaluation protocols (Louis and Nenkova, 2013; Hardy et al., 2019).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “As for the further understanding of human evaluation, prior work has compared approaches to human evaluation (Hardy et al., 2019), studied annotation protocols for quality dimensions such as linguistic quality (Steen and Markert, 2021) and factual consistency (Tang et al., 2022b), and noted the…”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, methodology): “These evaluations are much larger than what was previously done in the ﬁeld, e.g., 100 instances for MT in Ma et al. (2018) or around 100 instances for summarization in Kim et al. (2019) and Hardy et al. (2019).”