FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization

By Esin Durmus, He He, Mona T. Diab. 2020

475 citations recorded by the source corpus.

Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “As Durmus et al. (2022) show for summarization and dialog tasks, reference-free metrics tend to correlate with word overlap with the source, the perplexity of the model output, and its length.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “There has been much development toward these QA-based approaches (Eyal et al., 2019; Scialom et al., 2019; Durmus et al., 2020; Wang et al., 2020, among others) and they can provide an alternative angle to model evaluation that does not highly correlate with other evaluation approaches (Fabbri et…”
Measuring Attribution in Natural Language Generation Models (2021, background): “The choice of terminology is intended to be more conceptually transparent when used by annotators: unlike other terms like “meaningful”/“nonsensical” (Durmus et al., 2020), or “sensibleness” (Adiwardana et al., 2020), “interpretability” more readily alludes to the significance of the propositions…”
Survey of Hallucination in Natural Language Generation (2022, methodology): “In addition, Gabriel et al. [46] further analyze the FEQA and find that the effectiveness of QA-based metrics depends on the question.”
GPTScore: Evaluate as You Desire (2023, background): “…example, DEAM (Ghazarian et al., 2022) and Quan-tiDCE (Ye et al., 2021) were proposed for the evaluation of the coherence of the dialogue system; several evaluators (Cao et al., 2020; Durmus et al., 2020; Wang et al., 2020a) are designed for the evaluation of the consistency of text summarization.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Other possible ﬁxes include learning task-speciﬁc metrics to measure semantic overlap (Sellam et al., 2020) or metrics to check factual correctness (Zhang et al., 2020) and faithfulness to input (Wang et al., 2020; Durmus et al., 2020; Zhou et al., 2020).”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “to fairly assess accuracy, as metrics like BLEU, ROUGE, or BERTScore may not accurately convey the quality of generations (Wang et al., 2020; Durmus et al., 2020).”
Evaluation of Text Generation: A Survey (2020, background): “It is imperative that the summarization models are factually consistent and that any conflicts between a source document and its generated summary (commonly referred to as faithfulness (Durmus et al., 2020; Wang et al., 2020b)) can be easily measured, especially for domain-specific summarization tas”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, background): “, 2019) by leveraging existing semantic inference systems: (1) question answering (QA) system-based methods (Wang et al., 2020a; Durmus et al., 2020) first employ a question generation system to generate natural questions based on the machine generated summary, then questions are answered with a QA ”
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021, methodology): “This follows the intuition in Wang et al. (2020); Durmus et al. (2020) for evaluating abstractive summarization.”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021, methodology): “FEQA additionally is also negatively correlated with semantic frame errors and has low positive correlation with content verifiability errors.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, methodology): “Recent progress in neural approaches to building semantically constrained text generation systems has shown tremendous improvements in this direction (Liu and Lapata, 2019; Guo et al., 2018; Durmus et al., 2020; Wang et al., 2020).”
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization (2021, background): “The proportion will rise further as the data abstractiveness increases(Cao et al., 2018; Durmus et al., 2020; Kryscinski et al., 2020), causing factual checking an essential process to verify the credibility and usability of models.”
Read before Generate! Faithful Long Form Question Answering with Machine Reading (2022, methodology): “previous work (Wang et al., 2020; Durmus et al., 2020) which leverage a Question Generation(QG) and a QA model to generate question answer pairs, to evaluate the faithfulness of a summary5, we propose to evaluate answer faithfulness via evaluation”
News Summarization and Evaluation in the Era of GPT-3 (2022, methodology): “QA-based metrics (Fabbri et al., 2022; Durmus et al., 2020) and trained factuality models (Kryscinski et al.”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, methodology): “This has led to an increased research in faithfulness evaluation of summarization systems (Falke et al., 2019; Kryscinski et al., 2020; Durmus et al., 2020) as”
All You May Need for VQA are Image Captions (2022, background): “…et al., 2020) and as a component in solutions for other textual tasks, such as question answering (Al-berti et al., 2019; Puri et al., 2020), information retrieval (Mass et al., 2020; Gaur et al., 2021) and generation evaluation (Durmus et al., 2020; Wang et al., 2020; Honovich et al., 2021).”
Benchmarking Large Language Models for News Summarization (2023, methodology): “In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”