DialFact: A Benchmark for Fact-Checking in Dialogue

By Prakhar Gupta, C. Wu, Wenhao Liu, Caiming Xiong. 2021

83 citations recorded by the source corpus.

Fact-checking is an essential tool to mitigate the spread of misinformation and disinformation. We introduce the task of fact-checking in dialogue, which is a relatively unexplored area. We construct DialFact, a testing benchmark dataset of 22,245 annotated conversational claims, paired with pieces of evidence from Wikipedia. There are three sub-tasks in DialFact: 1) Verifiable claim detection task distinguishes whether a response carries verifiable factual information; 2) Evidence retrieval task retrieves the most relevant Wikipedia snippets as evidence; 3) Claim verification task predicts a dialogue response to be supported, refuted, or not enough information. We found that existing fact-checking models trained on non-dialogue data like FEVER fail to perform well on our task, and thus, we propose a simple yet data-efficient solution to effectively improve fact-checking performance in dialogue. We point out unique challenges in DialFact such as handling the colloquialisms, coreferences, and retrieval ambiguities in the error analysis to shed light on future research in this direction.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Measuring Attribution in Natural Language Generation Models (2021, methodology): “These have been measured via dialogue-specific natural language inference methods, often via human studies and data creation (Welleck et al. 2019; Mehri and Eskenazi 2020; Gupta et al. 2021; Honovich et al. 2021; Dziri et al. 2021; Santhanam et al. 2021).”
Survey of Hallucination in Natural Language Generation (2022, background): “[66] propose datasets that can benefit fact-checking systems specialized for dialogue systems.”
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023, background): “Another line of work focusing on evaluating the hallucination of models in different NLP tasks (Dziri et al., 2022b; Gupta et al., 2022; Dziri et al., 2022a; Rashkin et al., 2021; Li et al., 2023).”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, unknown): “This demonstrates ALIGNSCORE generalizes well to unseen data (e.g., DialFact dataset in the dialogue domain).”
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2021, unknown): “We also train classifiers on AugWow (Gupta et al., 2022), a synthetic dataset designed to evaluate factuality in dialogue systems.”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “Response entailment (Welleck et al., 2019; Nie et al., 2021; Gupta et al., 2022c) predicts whether a response entails a given premise.”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “Dialfact classification: based on the Dialfact dataset (Gupta et al., 2021) where the model must decide if an evidence supports, refutes, or does not have enough information to validate the response.”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “Q 2 (Honovich et al., 2021) 1,088 - 57.7% - DialFact (Gupta et al., 2021) 8,689 + 38.5% Fact Veriﬁcation - FEVER (Thorne et al., 2018) 18,209 - 35.1% - VitaminC (Schuster et al., 2021) 63,054 + 49.9% Paraphrasing - PAWS (Zhang et al., 2019) 8,000 + 44.2% world knowledge.”