SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization

By Yang Gao, Wei Zhao, Steffen Eger. 2020

142 citations recorded by the source corpus.

We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). We propose SUPERT, which rates the quality of a summary by measuring its semantic similarity with a pseudo reference summary, i.e. selected salient sentences from the source documents, using contextualized embeddings and soft token alignment techniques. Compared to the state-of-the-art unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18- 39%. Furthermore, we use SUPERT as rewards to guide a neural-based reinforcement learning summarizer, yielding favorable performance compared to the state-of-the-art unsupervised summarizers. All source code is available at https://github.com/yg211/acl20-ref-free-eval.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Recursively Summarizing Books with Human Feedback (2021, background): “…2019; Cohan et al., 2018; Xiao and Carenini, 2019; Zhao et al., 2020; Sotudeh et al., 2020), and patents (Sharma et al., 2019), as well as multi-document summarization (Liu et al., 2018; Ma et al., 2020; Gharebagh et al., 2020; Chandrasekaran et al., 2020; Liu and Lapata, 2019a; Gao et al., 2020).”
The statistical advantage of automatic NLG metrics at the system level (2021, methodology): “BertScore (Zhang et al., 2020), also present in the WMT analysis, and SUPERT (Gao et al., 2020), which is a reference-less metric for summarization.”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, background): “SUPERT [49] is an unsupervised MDS evaluation metric that measures the semantic similarity between the pseudo-reference summary and the machine-generated summary.”
SummEval: Re-evaluating Summarization Evaluation (2020, methodology): “SUPERT (Gao et al., 2020) is a reference-less metric, originally designed for multi-document summarization, which measures the semantic similarity of model outputs with pseudo-reference summaries created by extracting salient sentences from the source documents, using soft token alignment techniques”
RefSum: Refactoring Neural Summarization (2021, background): “(2019); Gao et al. (2020), which have shown superior performance on measuring semantic similarity between documents and summaries.”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, background): “, 2020) AS SUPERT (Gao et al., 2020) BLANC (Vasilyev et al.”
Fill in the BLANC: Human-free quality estimation of document summaries (2020, background): “, 2019; Cohan and Goharian, 2016; Elghannam and El-Shishtawy, 2015; Ng and Abrecht, 2015; Ganesan, 2018) or embeddings (Zhao et al., 2019; Zhang et al., 2020; Gao et al., 2020).”
News Summarization and Evaluation in the Era of GPT-3 (2022, methodology): “Here, we study 2 categories of metrics: (1) quality metrics, specifically SUPERT (Gao et al., 2020), which evaluates generated summaries against automatically identified salient sentences in the input, and BLANC (Vasilyev et al.”
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021, methodology): “There are also reference-free metrics outside the field of machine translation; for example, SUPERT (Gao et al., 2020) for summarization.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “SUPERT (Gao et al., 2020) measures the semantic similarity of summaries with pseudo-reference summaries created by extracting salient sentences from the source documents.”