Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering

By Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, Omri Abend. 2021

167 citations recorded by the source corpus.

Neural knowledge-grounded generative models for dialogue often produce content that is factually inconsistent with the knowledge they rely on, making them unreliable and limiting their applicability. Inspired by recent work on evaluating factual consistency in abstractive summarization, we propose an automatic evaluation metric for factual consistency in knowledge-grounded dialogue using automatic question generation and question answering. Our metric, denoted Q^2, compares answer spans using natural language inference (NLI), instead of token-based matching as done in previous work. To foster proper evaluation, we curate a novel dataset of dialogue system outputs for the Wizard-of-Wikipedia dataset, manually annotated for factual consistency. We perform a thorough meta-evaluation of Q^2 against other metrics using this dataset and two others, where it consistently shows higher correlation with human judgements.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…fixed size n-grams, but other basic element metrics like PARENT (Dhingra et al., 2019) only focus on content words. generate questions (Chen et al., 2018; Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021; Rebuffel et al., 2021; Honovich et al., 2021; Deutsch et al., 2021a, inter alia).”
Measuring Attribution in Natural Language Generation Models (2021, background): “…extends to a much larger set of system utterances, in particular giving a treatment of interpretability2, and contex- 2We acknowledge that the term “interpretability” has come to signify “model interpretability” in the NLP and ML community (as established in Harrington et al. (1985), tual effects.”
Survey of Hallucination in Natural Language Generation (2022, methodology): “A similar phenomenon has been shown in other NLG tasks, where it has been discovered that traditional metrics have a poor correlation with human judgment in terms of the hallucination problem [28, 33, 71, 84].”
DialFact: A Benchmark for Fact-Checking in Dialogue (2021, background): “In dialogue domain, related work either focus on evaluating factual consistency (Honovich et al., 2021; Qin et al., 2021) or consistent response generation (Rashkin et al.”
SMART: Sentences as Basic Units for Text Evaluation (2022, methodology): “…leverages a zero-shot paraphrasing model and uses probabilities from force-decoding the candidate sum-mary given the source as input; (14) Q 2 (Honovich et al., 2021): employs question generation and question answering models and checks whether answers from the summary are entailed by answers…”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, methodology): “This form of question-guided agreement was previously explored by Honovich et al. (2021).”
Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation (2021, background): “While factual correctness has been studied in knowledge-grounded response generation (Li et al., 2019; Shuster et al., 2021; Honovich et al., 2021; Rashkin et al., 2021), there has been very limited work in creating a human annotated dataset and detecting factual consistency for open-domain dialog…”
Large Language Models: A Survey (2024, unknown): “• QA-Based Metrics : Assess the overlap between generated content and the source through a question-answering framework (see [152]).”
All You May Need for VQA are Image Captions (2022, background): “Generative models may hallucinate, that is, generate content that is inconsistent with its input source (Alberti et al., 2019; Honovich et al., 2021).”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Notable implementations include (Durmus et al., 2020; Wang et al., 2020a; Scialom et al., 2021; Honovich et al., 2021).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, methodology): “…and methods that target a speciﬁc task or domain, such as summarization (Falke et al., 2019; Kryscinski et al., 2020; Wang et al., 2020; Scialom et al., 2021; Deutsch et al., 2021; Xie et al., 2021) or dialogue (Dziri et al., 2021; Honovich et al., 2021; Nie et al., 2021; Qin et al., 2021).”
Attributed Text Generation via Post-hoc Research and Revision (2022, methodology): “This form of question-guided agreement was previously explored by Honovich et al. (2021).”