A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

By Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, Iryna Gurevych. 2019

136 citations recorded by the source corpus.

Automated fact-checking based on machine learning is a promising approach to identify false information distributed on the web. In order to achieve satisfactory performance, machine learning methods require a large corpus with reliable annotations for the different tasks in the fact-checking process. Having analyzed existing fact-checking corpora, we found that none of them meets these criteria in full. They are either too small in size, do not provide detailed annotations, or are limited to a single domain. Motivated by this gap, we present a new substantially sized mixed-domain corpus with annotations of good quality for the core fact-checking tasks: document retrieval, evidence extraction, stance detection, and claim validation. To aid future corpus construction, we describe our methodology for corpus creation and annotation, and demonstrate that it results in substantial inter-annotator agreement. As baselines for future research, we perform experiments on our corpus with a number of model architectures that reach high performance in similar problem settings. Finally, to support the development of future models, we provide a detailed error analysis for each of the tasks. Our results show that the realistic, multi-domain setting defined by our data poses new challenges for the existing models, providing opportunities for considerable improvement by future systems.

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “Training datasets for evidence-based fact verification consist of instances containing a claim, a label indicating its veracity and the evidence, typically a set of sentences (Thorne et al., 2018a; Hanselowski et al., 2019; Wadden et al., 2020).”
Explainable Automated Fact-Checking for Public Health Claims (2020, neutral): “This includes more accurate machine learning models for veracity prediction and datasets of both naturally occurring (Wang, 2017; Augenstein et al., 2019; Hanselowski et al., 2019) and human-crafted (Thorne et al., 2018) fact-checking claims, against which the models can be evaluated.”
Generating Fact Checking Briefs (2020, methodology): “To address this Hanselowski et al. (2019) considered the claims fact checked by the website Snopes, but used the reports accompanying them as evidence instead of finding the evidence directly.”
Fact or Fiction: Verifying Scientific Claims (2020, background): “Fact-checking – a task in which the veracity of an input claim is verified against a corpus of documents that support or refute the claim – has been studied to combat the proliferation of misinformation in political news, social media, and on the web (Thorne et al., 2018; Hanselowski et al., 2019).”
On the Diversity and Limits of Human Explanations (2021, background): “(2018) and Hanselowski et al. (2019) ask almost the same questions) and ask people to select sentences instead of words.”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “…et al., 2018) verifying claims from text sentences crowd ∼ 136K ‡ H OTPOT QA (Yang et al., 2018) reading comprehension QA sentences crowd 112,779 Hanselowski et al. (2019) verifying claims from text sentences crowd 6,422 (varies) N ATURAL Q UESTIONS (Kwiatkowski et al., 2019) reading…”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Nevertheless, traditional fact-checking methodologies (Augenstein et al., 2019; Hanselowski et al., 2019; Atanasova et al., 2020) often incorporate simplified assumptions for practicality, leading to discrepancies when applied to complex real-world scenarios.”
Explainable Automated Fact-Checking: A Survey (2020, background): “There are also corpora which consist of real-world or naturally occurring claims, including claims from multiple domains, such as Mul-tiFC (Augenstein et al., 2019), FakeNewsNet (Shu et al., 2018), and a corpus with ﬁne-grained evi-1 dence (Hanselowski et al., 2019).”
FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information (2021, unknown): “…[Popat et al., 2016, Baly et al., 2018, Augenstein et al., 2019], or bypass the retrieval challenge entirely by extracting evidence directly from the fact checking articles [Alhindi et al., 2018, Hanselowski et al., 2019, Kotonya and Toni, 2020b] or scientiﬁc literature [Wadden et al., 2020].”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “[47]: A data archive is under Copyright.”
Evidence-based Verification for Real World Information Needs (2021, background): “…and task variants (Wang et al., 2017; Karadzhov et al., 2017; Baly et al., 2018; Thorne et al., 2018; Chen et al., 2019; Augenstein et al., 2019; Hanselowski et al., 2019; Chen et al., 2020; Wadden et al., 2020; Diggelmann et al., 2020; Schuster et al., 2021, inter alia ) that have enabled the…”