MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

By Isabelle Augenstein, Christina Lioma, Lucas Chaves Lima, Casper Hansen, Christian Hansen, Jakob Grue Simonsen. 2019

319 citations recorded by the source corpus.

We contribute the largest publicly available dataset of naturally occurring factual claims for the purpose of automatic claim verification. It is collected from 26 fact checking websites in English, paired with textual sources and rich metadata, and labelled for veracity by human expert journalists. We present an in-depth analysis of the dataset, highlighting characteristics and challenges. Further, we present results for automatic veracity prediction, both with established baselines and with a novel method for joint ranking of evidence pages and predicting veracity that outperforms all baselines. Significant performance increases are achieved by encoding evidence, and by modelling metadata. Our best-performing model achieves a Macro F1 of 49.2%, showing that this is a challenging testbed for claim veracity prediction.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, background): “While LiarPlus (Alhindi et al., 2018) and MultiFC (Augenstein et al., 2019) both have a similar task signature with claims and additional information provided as text, the text is not evidential (i.e. not containing a premise that would support or refute the claim).”
Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “Augenstein et al. [19] and Fan et al. [20] make use of web data through commercial search engines as a part of building more diverse datasets for fact-checking.”
Explainable Automated Fact-Checking for Public Health Claims (2020, neutral): “This includes more accurate machine learning models for veracity prediction and datasets of both naturally occurring (Wang, 2017; Augenstein et al., 2019; Hanselowski et al., 2019) and human-crafted (Thorne et al., 2018) fact-checking claims, against which the models can be evaluated.”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “Examples include ﬂagging fake news about recent events that happen outside of the training period (Thorne and Vlachos, 2018; Zellers et al., 2019; Augenstein et al., 2019), forecasting stock prices from the latest news articles (Ding et al., 2015), and answering knowledge-intensive questions like…”
Generating Fact Checking Briefs (2020, methodology): “The claims for the dataset are sourced from existing fact checking datasets, specifically DATACOMMONS2 and MULTIFC (Augenstein et al., 2019).”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “Other recent datasets cover ver-iﬁcation against tables (Chen et al., 2020), relational databases (Jo et al., 2019), Wikipedia references (Sathe et al., 2020), multiple articles (Jiang et al., 2020), and search snippets (Augenstein et al., 2019).”
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims (2020, background): “Consequently, researchers [6] started to collect real-world claims from multiple fact-checking organizations along with evidence manually curated by human fact-checkers.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, methodology): “Fact-checking Most models for fact-checking perform the following task: given a (statement, evidence) pair as input, they output a label indicating whether the statement is supported or refuted by the evidence (Thorne et al., 2018; Wang, 2017; Karadzhov et al., 2017; Augenstein et al., 2019; Wadden ”
Automated Fact-Checking for Assisting Human Fact-Checkers (2021, methodology): “Similarly, simple claims can be verified with promising accuracy results when good evidence is available, for example for entities popular on the Web [Augenstein et al., 2019].”
Evidence-based Verification for Real World Information Needs (2021, background): “…in context. many datasets and task variants (Wang et al., 2017; Karadzhov et al., 2017; Baly et al., 2018; Thorne et al., 2018; Chen et al., 2019; Augenstein et al., 2019; Hanselowski et al., 2019; Chen et al., 2020; Wadden et al., 2020; Diggelmann et al., 2020; Schuster et al., 2021, inter alia…”
Attributed Text Generation via Post-hoc Research and Revision (2022, methodology): “…models for fact-checking perform the following task: given a (statement, evidence) pair as input, they output a label indicating whether the statement is supported or refuted by the evidence (Thorne et al., 2018; Wang, 2017; Karadzhov et al., 2017; Augenstein et al., 2019; Wadden et al., 2020).”
Pitfalls of Static Language Modelling (2021, background): “Examples of such tasks include ﬂagging the most recent batch of fake news (Thorne and Vlachos, 2018; Zellers et al., 2019; Augenstein et al., 2019, inter alia ), and answering questions like “How many people have been infected by COVID-19 worldwide?””
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “[51] Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Nevertheless, traditional fact-checking methodologies (Augenstein et al., 2019; Hanselowski et al., 2019; Atanasova et al., 2020) often incorporate simplified assumptions for practicality, leading to discrepancies when applied to complex real-world scenarios.”
A Comprehensive Overview of Large Language Models (2023, unknown): “SQuAD [336], BoolQ [337], SQUADv2 [338], DROP [339], RTE [340], WebQA [341], CMRC2017 [342], CMRC2018 [343], CMRC2019 [344], COTE-BD [345], COTE-DP [345], COTE-MFW [345], MultiRC [346], Natural Questions [347], CNSE [301], DRCD [348], DuReader [349], Dureader robust [350], DuReader-QG [349], SciQ [3”