Question Answering as an Automatic Evaluation Metric for News Article Summarization

By Matan Eyal, Tal Baumel, Michael Elhadad. 2019

125 citations recorded by the source corpus.

Recent work in the field of automatic summarization and headline generation focuses on maximizing ROUGE scores for various news datasets. We present an alternative, extrinsic, evaluation metric for this task, Answering Performance for Evaluation of Summaries. APES utilizes recent progress in the field of reading-comprehension to quantify the ability of a summary to answer a set of manually created questions regarding central entities in the source article. We first analyze the strength of this metric by comparing it to known manual evaluation metrics. We then present an end-to-end neural abstractive model that maximizes APES, while increasing ROUGE scores to competitive results.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “metrics may blank out entities (Eyal et al., 2019; Xie et al., 2021; Scialom et al., 2019), or fully”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “There has been much development toward these QA-based approaches (Eyal et al., 2019; Scialom et al., 2019; Durmus et al., 2020; Wang et al., 2020, among others) and they can provide an alternative angle to model evaluation that does not highly correlate with other evaluation approaches (Fabbri et…”
MLSUM: The Multilingual Summarization Corpus (2020, neutral): “Recent works (Eyal et al., 2019; Scialom et al., 2019)”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “Other approaches do not attempt to estimate quality directly, but use information extraction or question answering as a proxy (Wiseman et al., 2017; Goodrich et al., 2019; Eyal et al., 2019).”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, methodology): “Most relatedly, Eyal et al. (2019) and Scialom et al. (2019) use QA models to evaluate summarization.”
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (2019, methodology): “This differs from the approach adopted by (Eyal et al., 2019) who trained their QA model on CNN-DM (the same data used for the summarization task).”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, methodology): “Recently, question answering (QA) based automatic metrics have been proposed for evaluating content selection in summarization (Eyal et al., 2019; Scialom et al., 2019; Chen et al., 2018).”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “Chen et al. (2017) and Eyal et al. (2019) proposed to automate this approach by automatically generating questions from the reference summary.”
Evaluation of Text Generation: A Survey (2020, methodology): “To evaluate the factual consistency of the text generation models, Eyal et al. (2019b) present a question-answering-based parametric evaluation model named Answering Performance for Evaluation of Summaries (APES).”
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021, methodology): “For example, Eyal et al. (2019) used QA models for abstractive summarization both as an evaluation metric and as an optimization criterion that improved the down-stream ROUGE scores by manually constructing questions around entities in the source document.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “A number of works have highlighted the effectiveness of QA and cloze task objectives for evaluating or improving factuality on speciﬁc domains (Eyal et al., 2019; Huang et al., 2020).”
Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View (2021, background): “…human (Chen and Bansal 2018) or automatic metrics using natural language inference (Kry´sci´nski et al. 2019; Falke et al. 2019), information extraction (Zhang et al. 2019; Goodrich et al. 2019) or question answering (Scialom et al. 2019; Eyal, Baumel, and Elhadad 2019; Wang, Cho, and Lewis 2020).”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, unknown): “, 2020) and question-answering models (Eyal et al., 2019; Scialom et al., 2019; Durmus et al., 2020; Wang et al., 2020).”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, background): “tention for summary quality dimensions beyond factual consistency (Eyal et al., 2019; Scialom et al., 2019; Deutsch et al., 2020).”
Cut the CARP: Fishing for zero-shot story evaluation (2021, methodology): “Unfortunately, methods for automatically evaluating stories, such as ROUGE (Lin, 2004), are extremely limited in their ability to accurately assess models and are easily Goodharted (Belz & Reiter, 2006; Cohan & Goharian, 2016; Schluter, 2017; Schluter & Alonso, 2016; Eyal et al., 2019).”
Evaluating Machines by their Real-World Language Use (2020, methodology): “Recent work uses pretrained question-answering models to evaluate summarization models (Chen et al., 2018; Scialom et al., 2019; Eyal et al., 2019; Vasilyev et al., 2020).”
Finding a Balanced Degree of Automation for Summary Evaluation (2021, background): “APES (Eyal et al., 2019) and QAEval (Deutsch et al., 2021) are QA-based metrics that assume similar answers will be obtained from similar system summaries and reference(s).”