The Fact Extraction and VERification (FEVER) Shared Task

By James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, Arpit Mittal. 2018

272 citations recorded by the source corpus.

We present the results of the first Fact Extraction and VERification (FEVER) Shared Task. The task challenged participants to classify whether human-written factoid claims could be SUPPORTED or REFUTED using evidence retrieved from Wikipedia. We received entries from 23 competing teams, 19 of which scored higher than the previously published baseline. The best performing system achieved a FEVER score of 64.21%. In this paper, we present the results of the shared task and a summary of the systems, highlighting commonalities and innovations among participating systems.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “, 2017), and fact verification (Thorne et al., 2018) all first retrieve relevant documents for their later stage reranking, machine reading, and reasoning models.”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “It is a combination of the two FEVER versions [Thorne et al., 2018b, 2019] omitting the N OT E NOUGH -”
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “systems developed for the FEVER benchmark (Thorne et al., 2018b), more recent proposals use the evidence to generate explanations.”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, methodology): “While there are numerous datasets for knowledge-intensive tasks (e.g. Thorne et al., 2018a; Dinan et al., 2019; Kwiatkowski et al., 2019, to name just a few), it is difficult to answer the above questions generally across them.”
The FEVER2.0 Shared Task (2019, methodology): “…(FEVER) shared task (Thorne et al., 2018b) focused on building systems that predict whether a textual claim is S UPPORTED or R EFUTED given evidence (see (Thorne et al., 2018a) for a task description), or N OT E NOUGH I NFORMATION in case Wikipedia does not have appropriate evidence to verify it.”
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023, background): “An alternate approach is to leverage fact-verification approaches, where evidence is retrieved from an external database to assess the ve-racity of a claim (Thorne et al., 2018; Guo et al., 2022).”
The Limitations of Stylometry for Detecting Machine-Generated Fake News (2019, background): “This approach has recently gained increasing attention thanks to several synthetic (Thorne et al. 2018) and real-world data sets (Hanselowski et al.”
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020, methodology): “List of errors: errors in manually ’mutated’ versions of Wikipedia articles (Thorne et al., 2018a).”
WikiFactFind: Semi-automated fact-checking based on Wikipedia (2022, background): “Aggregate results and provide final verdict and its interpretation. for the FEVER Shared task [14].”
A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (2019, methodology): “This motivates the development of automated fact-checking pipelines (Thorne et al., 2018a; Popat et al., 2017; Hanselowski and Gurevych, 2017) consisting of several consecutive tasks.”
Language Models as Fact Checkers? (2020, methodology): “Also note that the Fever-baseline model introduced by the task organizers achieves accuracy score of 48.8% (Thorne et al., 2018b).”
Kernel Graph Attention Network for Fact Verification (2019, methodology): “In our experiments on FEVER (Thorne et al., 2018a), a large-scale fact veriﬁcation benchmark, KGAT achieves the 69.4% FEVER score, significantly outperforming previous BERT and GAT based approaches (Zhou et al., 2019).”
Adversarial attacks against Fact Extraction and VERification (2019, background): “The Fact Extraction and VERification (FEVER) shared task [29] invited participants to build systems to predict whether short factoid sentences (referred to as claims) – manually written using information derived from Wikipedia – are SUPPORTED or REFUTED by evidence or whether there is NOTENOUGHINFO ”
Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering (2020, background): “However, the FEVER task requires ﬁnding a context sentence that simply paraphrases the claim, rather than a reasoned-based explanation from more general statements - the aim of this work.”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “The resemblance of NLI to factual consistency evaluation has led to utilizing NLI models for measuring factual consistency (Thorne et al., 2018; Maynez et al., 2020; Dziri et al., 2021).”
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention (2020, methodology): “, 2018), and FEVER, the fact verification benchmark whose claims often require multiple pieces of evidence to support (Thorne et al., 2018).”