Fact Checking: Task definition and dataset construction

By Andreas Vlachos, Sebastian Riedel. 2014

473 citations recorded by the source corpus.

In this paper we introduce the task of fact checking, i.e. the assessment of the truthfulness of a claim. The task is commonly performed manually by journalists verifying the claims made by public figures. Furthermore, ordinary citizens need to assess the truthfulness of the increasing volume of statements they consume. Thus, developing fact checking systems is likely to be of use to various members of society. We first define the task and detail the construction of a publicly available dataset using statements fact-checked by journalists available online. Then, we discuss baseline approaches for the task and the challenges that need to be addressed. Finally, we discuss how fact checking relates to mainstream natural language processing tasks and can stimulate further research.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

FEVER: a Large-scale Dataset for Fact Extraction and VERification (2018, background): “2 Related Works Vlachos and Riedel (2014) constructed a dataset for claim veriﬁcation consisting of 106 claims, selecting data from fact-checking websites such as PolitiFact, taking advantage of the labelled claims available there.”
“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection (2017, background): “Vlachos and Riedel (2014) are the ﬁrst to construct fake news and fact-checking datasets.”
Generating Fact Checking Briefs (2020, background): “One path for scaling fact checking could be through full automation, taking a claim as input and producing a verdict (Vlachos and Riedel, 2014).”
UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF) (2018, background): “In this context, technology to automate fact-checking and source verification (Vlachos and Riedel, 2014) is of great interest to both media consumers and publishers.”
Automated Fact Checking: Task Formulations, Methods and Future Directions (2018, methodology): “Vlachos and Riedel (2014) collected and released 221 labeled claims in the political domain that had been fact checked by Politifact and Channel4.”
Fighting an Infodemic: COVID-19 Fake News Dataset (2020, background): “(Vlachos and Riedel, 2014) collected 221 labeled claims that were fact checked by journalists available online.”
Explainable Automated Fact-Checking: A Survey (2020, methodology): “Claims Sources Labels Evidence Explanation Decisions in dataset annotation inform formulations of the fact-checking task, e.g., the choice be-tween binary (P´erez-Rosas et al., 2018) and graded classiﬁcation (Vlachos and Riedel, 2014; Wang, 2017) informs the deﬁnition of veracity.”
A Survey on Natural Language Processing for Fake News Detection (2018, background): “Fact checking is the task of assessing the truth-fulness of claims made by public ﬁgures such as politicians, pundits, etc (Vlachos and Riedel, 2014).”
A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (2019, background): “PolitiFact14 Vlachos and Riedel (2014) As our analysis shows, while multiple fact-checking corpora are already available, no single existing resource provides full fact-checking sub-task coverage backed by a substantially-sized and validated dataset spanning across multiple domains.”
A Multi-Level Attention Model for Evidence-Based Fact Checking (2021, background): “The half-true label is deﬁned in some previous smaller datasets (Vlachos and Riedel, 2014; Wang, 2017), but not in the FEVER dataset.”
Fake news, disinformation and misinformation in social media: a review (2023, unknown): “Therefore, in the following, we specifically review examples of solutions that use fact-checking (Vlachos and Riedel 2014) to help build datasets that can be further used in the automatic detection of fake content.”
Improving Large-Scale Fact-Checking using Decomposable Attention Models and Lexical Tagging (2018, methodology): “Similar to the FakeNews, (Rashkin et al., 2017; Vlachos and Riedel, 2014) focused on political statements from Politifact.com to verify the degree of truthfulness.”
Emergent: a novel data-set for stance classification (2016, background): “Previous work by Vlachos and Riedel (2014) proposed using data from fact-checking websites such as Politifact1, but the labelling provided by the journalists is only the degree of truthfulness of the claims, without any machinereadable verdicts to supervise the various steps in deciding it.”
Toward Automated Fact-Checking: Detecting Check-worthy Factual Claims by ClaimBuster (2017, background): “Vlachos and Riedel [24] analyzed the tasks in fact”