Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence

By Tal Schuster, Adam Fisch, Regina Barzilay. 2021

285 citations recorded by the source corpus.

Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness—improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Improving Wikipedia verifiability with AI (2022, background): “Finally, there exist a large body of research focused on fact-checking Wikipedia claims (Thorne & Vlachos, 2018; Thorne et al., 2018, 2019; Schuster et al., 2021; Trokhymovych & Saez-Trumper, 2022).”
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “, 2020), correct factual errors (Thorne and Vlachos, 2021b; Schuster et al., 2021) or perform rule discovery (Ahmadi et al.”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “Both SM and Meta are relatively well-calibrated for IMDB and VitaminC, which makes the threshold-based exit rule a competitive baseline.”
Entity-Based Knowledge Conflicts in Question Answering (2021, background): “More problematically, this behaviour inhibits the model’s ability to generalize to evolving knowledge and timedependent answers, not found in training (Guu et al., 2020; Schuster et al., 2021).”
Memory-Based Model Editing at Scale (2022, methodology): “We introduce the FC setting, building on the VitaminC fact verification dataset (Schuster et al., 2021), to assess an editor’s ability to update an out-of-date fact-checking model when presented with updated information about the world.”
Enabling Large Language Models to Generate Text with Citations (2023, methodology): “We use the version of TRUE model from https://huggingface.co/google/ t5_xxl_true_nli_mixture , which is trained on SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), Fever (Thorne et al., 2018), Sci-tail (Khot et al., 2018), PAWS (Zhang et al., 2019), and VitaminC (Schuster et al., 2021).”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, background): “Schuster et al. (2021) presents contrastive sentences based on Wikipedia revisions, showing how entailment decision can change over time.”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, methodology): “(2021) works analogously to the above metrics with a base model trained on both MNLI and Vitamin-C data (Schuster et al., 2021), consisting of closely-related contrastive entailment examples.”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, unknown): “16 Excluding VitaminC as it is much larger than other datasets and might therefore distort results.”
Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation (2021, methodology): “We vary knowledge retrieval setups as factual correctness in a response can vary depending on how accurate a knowledge retrieval model is (Schuster et al., 2021b).”
Unifying Language Learning Paradigms (2022, background): “…2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al., 2019), VitaminC (Schuster et al., 2021a), Civil Comments and Wikipedia Toxicity detection datasets (Borkan et al., 2019).”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, methodology): “…with attribution, we are inspired by recent successes in the fact-checking1 community where simple research-and-revise workflows are increasingly effective in attributing and/or correcting unattributable claims made by people (Thorne et al., 2018; Schuster et al., 2021; Thorne and Vlachos, 2021).”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, unknown): “For a fair comparison, we also report the average AUC-ROC (denoted as AVG-ZS) excluding datasets that the alignment function is trained on (PAWS, VitaminC and FEVER).”
Attributed Text Generation via Post-hoc Research and Revision (2022, methodology): “(4)We obtained a newer version of the E2E NLI model from the authors, which was trained on MNLI, SNLI, FEVER, PAWS, SciTail and VitaminC (Williams et al., 2018; Bowman et al., 2015; Thorne et al., 2018; Zhang et al., 2019; Khot et al., 2018; Schuster et al., 2021).”