FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information

By Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal. 2021

259 citations recorded by the source corpus.

Fact verification has attracted a lot of attention in the machine learning and natural language processing communities, as it is one of the key methods for detecting misinformation. Existing large-scale benchmarks for this task have focused mostly on textual sources, i.e. unstructured information, and thus ignored the wealth of information available in structured formats, such as tables. In this paper we introduce a novel dataset and benchmark, Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS), which consists of 87,026 verified claims. Each claim is annotated with evidence in the form of sentences and/or cells from tables in Wikipedia, as well as a label indicating whether this evidence supports, refutes, or does not provide enough information to reach a verdict. Furthermore, we detail our efforts to track and minimize the biases present in the dataset and could be exploited by models, e.g. being able to predict the label without using evidence. Finally, we develop a baseline for verifying claims against text and tables which predicts both the correct evidence and verdict for 18% of the claims.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “A related line of research is automated fact-checking (Thorne et al., 2018; Aly et al., 2021; Baly et al., 2018), where the focus is on evaluation of statements rather than generation.”
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023, methodology): “Generation Benefits Retrieval Adaptation To investigate how LLM outputs can be leveraged for retrieval adaptation, we experimented on Hot-PotQA and Feverous.”
DialFact: A Benchmark for Fact-Checking in Dialogue (2021, background): “(2018) and subsequent works (Wenhu Chen et al., 2020; Jiang et al., 2020; Nørregaard and Derczynski, 2021; Aly et al., 2021) introduced fact extraction and verification datasets verifiable against pieces of evidence from Wikipedia articles.”
A Survey on Automated Fact-Checking (2021, background): “Aly et al. (2021) further considered both text and table for factual verification, while explicitly requiring the retrieval of evidence.”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, background): “Due to the emergence of large language models (LLMs), the ”pre-train, prompt, predict” paradigm has revolutionized natural language processing (NLP) in real-world applications, such as open-domain question answering (O-QA), fact-checking (FC), and arithmetic reasoning (AR) (Chen et al. 2017; Asai et”