The FEVER2.0 Shared Task

By James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, Arpit Mittal. 2019

112 citations recorded by the source corpus.

We present the results of the second Fact Extraction and VERification (FEVER2.0) Shared Task. The task challenged participants to both build systems to verify factoid claims using evidence retrieved from Wikipedia and to generate adversarial attacks against other participant’s systems. The shared task had three phases: building, breaking and fixing. There were 8 systems in the builder’s round, three of which were new qualifying submissions for this shared task, and 5 adversaries generated instances designed to induce classification errors and one builder submitted a fixed system which had higher FEVER score and resilience than their first submission. All but one newly submitted systems attained FEVER scores higher than the best performing system from the first shared task and under adversarial evaluation, all systems exhibited losses in FEVER score. There was a great variety in adversarial attack types as well as the techniques used to generate the attacks, In this paper, we present the results of the shared task and a summary of the systems, highlighting commonalities and innovations among participating systems.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

How Decoding Strategies Affect the Verifiability of Generated Text (2019, methodology): “46% in the second FEVER shared task (Thorne et al., 2019).”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “It is a combination of the two FEVER versions [Thorne et al., 2018b, 2019] omitting the N OT E NOUGH -”
Improving Wikipedia verifiability with AI (2022, background): “Finally, there exist a large body of research focused on fact-checking Wikipedia claims (Thorne & Vlachos, 2018; Thorne et al., 2018, 2019; Schuster et al., 2021; Trokhymovych & Saez-Trumper, 2022).”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, neutral): “– KILT: FEVER (Thorne et al., 2018), AIDA (Yosef et al., 2011), WNED (Guo & Barbosa, 2014), T-REx (Guo & Barbosa, 2014), NQ (Kwiatkowski et al., 2019), HoPo (Yang et al., 2018), TQA (Joshi et al., 2017), ELI5 (Fan et al., 2019), WoW (Dinan et al., 2019).”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, background): “0 , Thorne et al., 2019) introduces a collection of adversarial instances.”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “Adversarial (Thorne et al., 2019c) Table 3: Factual revision ﬂagging scores for models aware of the full sentence-pair (full) and aware only of the modiﬁed words (diff).”
The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task (2021, background): “(Nakov et al., 2021), SemEval (Wang et al., 2021) and FEVER (Thorne et al., 2018b), as well as a number of datasets aimed at modelling parts of the task (Karadzhov et al., 2017; Wang, 2017; Augen-stein et al., 2019; Chen et al., 2020; Gupta et al., 2020).”
What Question Answering can Learn from Trivia Nerds (2019, methodology): “, 2016), as attempted by the 2019 iteration of FEVER (Thorne et al., 2019).”
Program Enhanced Fact Verification with Verbalization and Graph Attention Network (2020, background): “Recent proposed FEVER 2.0 (Thorne et al., 2019) has a more challenging dataset to verify factoid claims and an adversarial attack task.”
A Survey on Automated Fact-Checking (2021, background): “0 shared task explored how to generate adversarial claims and build systems resilient to such attacks (Thorne et al., 2019).”
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021, methodology): “0 (Thorne et al., 2019), crowdworkers were required to fool a fact-verification system trained on the FEVER (Thorne et al.”
Explainable Automated Fact-Checking: A Survey (2020, methodology): “Building upon these works in explainable NLI, there are a number of directions which we envision as future avenues of research in explainable fact-checking which could result in valuable gains.”
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020, background): “The FEVER workshops and shared tasks (Thorne et al., 2018b, 2019) asked participants to develop techniques to identify factual 1https://github.com/nlgcat/evaluating_ accuracy The Memphis Grizzlies (5-2) defeated the Phoenix Suns (3 - 2) Monday 102-91 at the Talking Stick Resort Arena in Phoenix.”