The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task
By Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal. 2021
The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are Supported or Refuted based on evidence retrieved from Wikipedia (or NotEnoughInfo if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018, FEVEROUS requires partial evidence to be returned for NotEnoughInfo claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.
Subfields: Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Evaluation & Benchmarks
- Andreas Vlachos
- James Thorne
- Caiming Xiong
- Wenhu Chen
- Tal Schuster
- Mohit Bansal
- Arpit Mittal
- Christos Christodoulopoulos
Citations in the corpus
- UL2: Unifying Language Learning Paradigms (2022, background): “We use several component tasks from UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Berant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
- UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “, 2020b) 1024 128 4 FEVEROUS (Aly et al., 2021) 1024 32 4”
- Large Language Models are few(1)-shot Table Reasoners (2022, methodology): “…2020c; Zhu et al., 2021; Chen et al., 2021b; Talmor et al., 2020; Chen et al., 2020a; Nan et al., 2022), table fact verifi-cation (Chen et al., 2019; Aly et al., 2021), table-based generation (Chen et al., 2020b; Parikh et al., 2020; Nan et al., 2021), and table-grounded conversation (Budzianowski…”
- Unifying Language Learning Paradigms (2022, methodology): “We use several component tasks from UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Be-rant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
- FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information (2021, methodology): “The FEVEROUS dataset was used for the shared task of the FEVER Workshop 2021 [Aly et al., 2021], with the same splits and evaluation as presented in this paper.”