Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

By Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano. 2019

162 citations recorded by the source corpus.

Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similarity as well as from sub-optimal accounting for fluency and readability of the generated abstracts. We thus explore and propose alternative evaluation measures: the reported human-evaluation analysis shows that the proposed metrics, based on Question Answering, favorably compare to ROUGE – with the additional property of not requiring reference summaries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as reward.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Yet language models show potential for mitigating these problems directly (Trinh & Le, 2018; Radford et al., 2019) or indirectly (Ra-jani et al., 2019; Xenouleas et al., 2019; Scialom et al., 2019).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “metrics may blank out entities (Eyal et al., 2019; Xie et al., 2021; Scialom et al., 2019), or fully”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “There has been much development toward these QA-based approaches (Eyal et al., 2019; Scialom et al., 2019; Durmus et al., 2020; Wang et al., 2020, among others) and they can provide an alternative angle to model evaluation that does not highly correlate with other evaluation approaches (Fabbri et…”
MLSUM: The Multilingual Summarization Corpus (2020, background): “Recent works (Eyal et al., 2019; Scialom et al., 2019)”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Complementary metrics have been proposed to evaluate NLG tasks, based on Question Answering (Scialom et al., 2019) or learned from human evaluation data (B ¨ ohm et al., 2019).”
Recursively Summarizing Books with Human Feedback (2021, methodology): “Summarization tasks later in the book are conditioned on previous summaries at the same height; we show this for the blue task at height 1 using dotted arrows, but this happens at all levels of the tree.”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, methodology): “Most relatedly, Eyal et al. (2019) and Scialom et al. (2019) use QA models to evaluate summarization.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, methodology): “Recently, question answering (QA) based automatic metrics have been proposed for evaluating content selection in summarization (Eyal et al., 2019; Scialom et al., 2019; Chen et al., 2018).”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “The orange area corresponds to the recall-oriented SummaQA (Scialom et al., 2019).”
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021, methodology): “…the low scores assigned by the token-level F1 measure to semantically-identical answers that are lexically different, they use a measure of the QA conﬁdence of answerability (Scialom et al., 2019), which is the complement of the probability that the QA model gives to the “no answer” prediction.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, methodology): “When considering better-performing factuality metrics identiﬁed by the controlled error analysis, we ﬁnd that the controlled data analysis is generally able to identify better-performing metrics (SummaQA, ROUGE-(2/3) and BERTScore) for XSUM with the exception of FEQA (FEQA metric performs well on XS”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, background): “SummaQA (Scialom et al., 2019) is also a QA-based metric.”
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2020, background): “CogComp/qaeval-experiments Recently, there has been growing interest in representing semantic information in the form of question-answer (QA) pairs for evaluating different aspects of summaries (Eyal et al., 2019; Scialom et al., 2019; Wang et al., 2020; Durmus et al., 2020).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “SummaQA (Scialom et al., 2019) applies a BERT question-answering (QA) model to answer cloze input questions using generated summaries.”
Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View (2021, background): “…human (Chen and Bansal 2018) or automatic metrics using natural language inference (Kry´sci´nski et al. 2019; Falke et al. 2019), information extraction (Zhang et al. 2019; Goodrich et al. 2019) or question answering (Scialom et al. 2019; Eyal, Baumel, and Elhadad 2019; Wang, Cho, and Lewis 2020).”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, unknown): “, 2020) and question-answering models (Eyal et al., 2019; Scialom et al., 2019; Durmus et al., 2020; Wang et al., 2020).”
SAFEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “The orange area corresponds to the recall-oriented SummaQA (Scialom et al., 2019).”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “QA models are employed to measure content consistency by reading the articles to answer questions generated from the summaries (Wang et al., 2020; Durmus et al., 2020), or considering the summaries for addressing questions derived from the source (Scialom et al., 2019).”
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021, methodology): “Recent work uses pretrained question-answering models to evaluate summarization models (Chen et al., 2018; Scialom et al., 2019; Eyal et al., 2019; Vasilyev et al., 2020).”
BookSum: A Collection of Datasets for Long-form Narrative Summarization (2021, methodology): “Other recently introduced metrics, such as BERTScore and SummaQA, leverage pre-trained neural models, which in turn makes them subject to the same input length limitations as the evaluated summarization models.”