FEVER: a Large-scale Dataset for Fact Extraction and VERification

By James Thorne, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal. 2018

2,295 citations recorded by the source corpus.

In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as Supported, Refuted or NotEnoughInfo by annotators achieving 0.6841 in Fleiss kappa. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Such tasks include open-domain question answering [5, 25], fact checking [50], fact completion [42], long-form question answering [12], Wikipedia article generation [32], dialogue [36, 59, 9, 13], translation [16], and language modeling [17, 23].”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Previous work has shown that retrieval improves performance across a variety of tasks such as question answering (Voorhees et al., 1999; Chen et al., 2017; Kwiatkowski et al., 2019), fact checking (Thorne et al., 2018), dialogue (Dinan et al., 2019) or citation recommendation (Petroni et al., 2022).”
ReAct: Synergizing Reasoning and Acting in Language Models (2022, positive): “Domains We consider two datasets challenging knowledge retrieval and reasoning: (1) HotPotQA (Yang et al., 2018), a multi-hop question answering benchmark that requires reasoning over two or more Wikipedia passages, and (2) FEVER (Thorne et al., 2018), a fact verification benchmark where each claim ”
Adversarial Training for Large Neural Language Models (2020, methodology): “(2019) to enable a head-to-head comparison, which combines four datasets (ANLI, MNLI, SNLI and FEVER (Thorne et al., 2018)) for fine-tuning.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “For the second round, we used a more powerful RoBERTa model (Liu et al., 2019b) trained on SNLI, MNLI, an NLI-version 2 of FEVER (Thorne et al., 2018), and the training data from the previous round (A1).”
Defending Against Neural Fake News (2019, background): “Future work should investigate integrating knowledge into the discriminator (e.g. for claim verification in FEVER; Thorne et al., 2018).”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “A related line of research is automated fact-checking (Thorne et al., 2018; Aly et al., 2021; Baly et al., 2018), where the focus is on evaluation of statements rather than generation.”
Distilling Knowledge from Reader to Retriever for Question Answering (2020, background): “Information retrieval is an important component for many natural language processing tasks, such as question answering (Voorhees et al., 1999) or fact checking (Thorne et al., 2018).”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “Mitigating hypothesis only bias in fact verification: We train models using sentence-pair instances extracted from the FEVER dataset released by Thorne et al. (2018). For the supported and refuted we use the evidence present in the dataset, and the NotEnoughInfo class, random”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, methodology): “Fact Checker We consider an off-the-shelf fact checker3 trained on the FEVER dataset (Thorne et al., 2018) which achieves the highest FEVER”
Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, background): “More broadly, extracting evidence can facilitate fact verification (Thorne et al., 2018) and debate.”
Reasoning Over Semantic-Level Graph for Fact Checking (2019, methodology): “We conduct experiments on FEVER (Thorne et al., 2018a), which is one of the most inﬂuen-tial benchmark datasets for fact checking.”
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale (2019, methodology): “The release of FEVER (Thorne et al., 2018) stimulates many recent developments (Nie et al., 2019; Yoneda et al., 2018; Hanselowski et al., 2018) on data-driven neural networks for automatic fact checking.”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “It is a combination of the two FEVER versions [Thorne et al., 2018b, 2019] omitting the N OT E NOUGH -”
Improving Wikipedia verifiability with AI (2022, background): “Finally, there exist a large body of research focused on fact-checking Wikipedia claims (Thorne & Vlachos, 2018; Thorne et al., 2018, 2019; Schuster et al., 2021; Trokhymovych & Saez-Trumper, 2022).”
Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, background): “, 2017)), FEVER (Thorne et al., 2018), and Wizard of Wikipedia (WoW) (Dinan et al.”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…with knowledge intensive language tasks (Levy et al., 2017; Dinan et al., 2019; Elsahar et al., 2018; Petroni et al., 2021), such as open-domain question answering (Voorhees et al., 1999; Joshi et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019) and fact checking (Thorne et al., 2018).”
Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “, 2021); and a fact-checking dataset: the single-hop FEVER (Thorne et al., 2018).”