Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

By Alex Wang, Kyunghyun Cho, Michael Lewis. 2020

598 citations recorded by the source corpus.

Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose QAGS (pronounced “kags”), an automatic evaluation protocol that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text. Code for QAGS will be available at https://github.com/W4ngatang/qags.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…fixed size n-grams, but other basic element metrics like PARENT (Dhingra et al., 2019) only focus on content words. generate questions (Chen et al., 2018; Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021; Rebuffel et al., 2021; Honovich et al., 2021; Deutsch et al., 2021a, inter alia).”
Recursively Summarizing Books with Human Feedback (2021, background): “Another way to evaluate summaries is to test whether they can be used to answer questions about the original text (Scialom et al., 2019; Wang et al., 2020).”
Measuring Attribution in Natural Language Generation Models (2021, methodology): “Additional automatic QA-based methods for detecting hallucinations have been proposed by Wang et al. (2020), Nan et al. (2021), inter alia.”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “, 2019b) and question answering (Arumae and Liu, 2019; Wang et al., 2020).”
Survey of Hallucination in Natural Language Generation (2022, methodology): “The results show both QAGS and FEQAhave substantially higher correlationswith human judgments of faithfulness than the baseline metrics.”
GPTScore: Evaluate as You Desire (2023, methodology): “…example, DEAM (Ghazarian et al., 2022) and Quan-tiDCE (Ye et al., 2021) were proposed for the evaluation of the coherence of the dialogue system; several evaluators (Cao et al., 2020; Durmus et al., 2020; Wang et al., 2020a) are designed for the evaluation of the consistency of text summarization.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Other possible ﬁxes include learning task-speciﬁc metrics to measure semantic overlap (Sellam et al., 2020) or metrics to check factual correctness (Zhang et al., 2020) and faithfulness to input (Wang et al., 2020; Durmus et al., 2020; Zhou et al., 2020).”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “to fairly assess accuracy, as metrics like BLEU, ROUGE, or BERTScore may not accurately convey the quality of generations (Wang et al., 2020; Durmus et al., 2020).”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, methodology): “QAGS (Wang et al., 2020) is a precision oriented metric, with questions generated from the summary.”
Evaluation of Text Generation: A Survey (2020, background): “Long text consists of groups of sentences structured together by linguistic elements known as discourse (Jurafsky & Martin, 2009).”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, methodology): “…three bilingual speakers to anno-tate the Chinese-to-English evaluation set D eval , which is composed of 150 sentences from the test set of Zh-En multi-domain dataset (Wang et al., 2020b), and 100 sentences from the COVID-19 translation benchmark dataset (Anastasopou-los et al., 2020) – TICO.”
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021, methodology): “To alleviate errors made by the automatic QG and QA models, we follow the validation step in Wang et al. (2020); We run the QA model to answer q i j with the response r as the input paragraph, and require the answer to be identical to the answer span a ri which was used to generate q i j .”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021, background): “QAGS (Wang et al., 2020): QAGS is another QGA metric.”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “(2) QAGS20 [67] collected 235 test outputs on CNNDM dataset from [16] and 239 test outputs on XSUM dataset [48] from BART ﬁne-tuned on XSUM .”
Factual Error Correction for Abstractive Summarization Models (2020, methodology): “, 2019), applying natural language inference or question answering models for consistency checking (Falke et al., 2019; Li et al., 2018; Wang et al., 2020) and training the model on artificial datasets (Kryściński et al.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, methodology): “We follow prior work that considered ROUGE in factual consistency evaluations (Wang et al., 2020), though it has also been previously noted that ROUGE can underweight good summarization examples (Novikova et al., 2017).”
Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries (2021, background): “Recent work by Wang et al. (2020) argues that many of the same evaluation metrics covered in this work do not successfully measure the faithfulness of a summary based on low correlations to ground-truth judgments.”