Automated Fact Checking: Task Formulations, Methods and Future Directions

By James Thorne, Andreas Vlachos. 2018

330 citations recorded by the source corpus.

The recently increased focus on misinformation has stimulated research in fact checking, the task of assessing the truthfulness of a claim. Research in automating this task has been conducted in a variety of disciplines including natural language processing, machine learning, knowledge representation, databases, and journalism. While there has been substantial progress, relevant papers and articles have been published in research communities that are often unaware of each other and use inconsistent terminology, thus impeding understanding and further progress. In this paper we survey automated fact checking research stemming from natural language processing and related disciplines, unifying the task formulations and methodologies across papers and authors. Furthermore, we highlight the use of evidence as an important distinguishing factor among them cutting across task formulations and methods. We conclude with proposing avenues for future NLP research on automated fact checking.

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Revealing the Importance of Semantic Retrieval for Machine Reading at Scale (2019, methodology): “To the best of our knowledge, we are the ﬁrst to apply and optimize neural semantic retrieval at both paragraph and sentence levels for MRS. Automatic Fact Checking: Recent work (Thorne and Vlachos, 2018) formalized the task of automatic fact checking from the viewpoint of machine learning and NLP.”
Improving Wikipedia verifiability with AI (2022, background): “Finally, there exist a large body of research focused on fact-checking Wikipedia claims (Thorne & Vlachos, 2018; Thorne et al., 2018, 2019; Schuster et al., 2021; Trokhymovych & Saez-Trumper, 2022).”
Measuring Attribution in Natural Language Generation Models (2021, background): “A related field of study has dealt with the topic of fact or claim verification (Thorne et al., 2018; Thorne and Vlachos, 2018; Thorne et al., 2021, inter alia).”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “Examples include ﬂagging fake news about recent events that happen outside of the training period (Thorne and Vlachos, 2018; Zellers et al., 2019; Augenstein et al., 2019), forecasting stock prices from the latest news articles (Ding et al., 2015), and answering knowledge-intensive questions like…”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “Most of the fact verification literature focuses on resolving claims to be true or false (Popat et al., 2018; Thorne and Vlachos, 2018; Wang, 2017).”
Predicting Factuality of Reporting and Bias of News Media Sources (2018, background): “Another recent survey was run by Thorne and Vlachos (2018), which took a fact-checking perspective on “fake news” and related problems.”
Automatic Detection of Machine Generated Text: A Critical Survey (2020, background): “Hence, we call for more research on detectors that assess the veracity of machine generated text by consulting external sources, like knowledge bases (Thorne and Vlachos, 2018) and diffusion network (Vosoughi et al., 2018), instead of relying only on the source.”
Where is Your Evidence: Improving Fact-checking by Justification Modeling (2018, background): “Thorne and Vlachos (2018) provide a survey of current datasets and models for fact-checking (e.g., (Wang, 2017; Rashkin et al., 2017; Vlachos and Riedel, 2014; Thorne et al., 2018; Long et al., 2017; Potthast et al., 2018; Wang et al., 2018)).”
A Survey on Computational Propaganda Detection (2020, background): “Finally, it is worth noting that, even though there have been several recent surveys on fake news detection [Shu at al., 2017; Zhou et al., 2019], fact-checking [Thorne and Vlachos, 2018], and truth discovery [Li et al., 2016], none of them focuses on computational propaganda.”
Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document (2021, background): “Thus, automatic factchecking has been proposed as a possible alternative (Li et al. 2016; Shu et al. 2017; Rashkin et al. 2017; Hassan et al. 2017; Vo and Lee 2018; Lee, Wu, and Fung 2018; Li et al. 2018; Thorne and Vlachos 2018; Lazer et al. 2018; Vosoughi, Roy, and Aral 2018; Zhang et al. 2020b), ”
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2021, background): “, 2019) or fact checking (Thorne and Vlachos, 2018) also adopt this strategy.”
Evidence-based Verification for Real World Information Needs (2021, background): “…MultiFC (Augen-stein et al., 2019) (Derczynski et al., 2017) 330 yes no Twitter (Baly et al., 2018) 442 yes no News + fact checking websites (Thorne and Vlachos, 2018) 185 , 445 no yes Wikipedia (constructed claims) (Chen et al., 2019) 1 , 000 yes no Debate websites (Augenstein et al.,…”
SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles (2020, background): “Thorne and Vlachos (2018) surveyed fact-checking approaches to fake news and related problems, and Li et al. (2016) focused on truth discovery in general.”
FANG: Leveraging Social Context for Fake News Detection Using Graph Representation (2020, background): “In order to scale with the increasing amount of information, automated news verification systems consider external knowledge databases as evidence [13, 34, 42].”
FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information (2021, methodology): “…as the volume of potentially misleading and false claims rises [Graves, 2018], resulting in the development of both fully automated methods (see Thorne and Vlachos [2018], Zubiaga et al. [2018], Hardalov et al. [2021] for recent surveys) as well as technologies that can assist human…”
Pitfalls of Static Language Modelling (2021, background): “Examples of such tasks include ﬂagging the most recent batch of fake news (Thorne and Vlachos, 2018; Zellers et al., 2019; Augenstein et al., 2019, inter alia ), and answering questions like “How many people have been infected by COVID-19 worldwide?””