“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection

By William Yang Wang. 2017

1,697 citations recorded by the source corpus.

Automatic fake news detection is a challenging problem in deception detection, and it has tremendous real-world political and social impacts. However, statistical approaches to combating fake news has been dramatically limited by the lack of labeled benchmark datasets. In this paper, we present LIAR: a new, publicly available dataset for fake news detection. We collected a decade-long, 12.8K manually labeled short statements in various contexts from PolitiFact.com, which provides detailed analysis report and links to source documents for each case. This dataset can be used for fact-checking research as well. Notably, this new dataset is an order of magnitude larger than previously largest public fake news datasets of similar type. Empirically, we investigate automatic fake news detection based on surface-level linguistic patterns. We have designed a novel, hybrid convolutional neural network to integrate meta-data with text. We show that this hybrid approach can improve a text-only deep learning model.

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

GLTR: Statistical Detection and Visualization of Generated Text (2019, background): “Other scenarios include the possibility of generating false articles (Wang, 2017) or misleading reviews (Fornaciari and Poe-sio, 2014).”
Defending Against Neural Fake News (2019, background): “Efforts to automate fake news detection generally point out stylistic biases that exist in the text (Rashkin et al., 2017; Wang, 2017; Pérez-Rosas et al., 2018).”
Reasoning Over Semantic-Level Graph for Fact Checking (2019, background): “…investigated by following works, including Ferreira and Vlachos (2016) who build Emergent with 300 labeled rumors and about 2.6K news articles, Wang (2017) who builds LIAR with 12.8K annotated short statements and six ﬁne-grained labels, and Rashkin et al. (2017) who collect claims without…”
FEVER: a Large-scale Dataset for Fact Extraction and VERification (2018, background): “Wang (2017) extended this approach by including all 12.8K claims available by Politifact via its API, however the justiﬁcation and the evidence contained in it was ignored in the experiments as it was not machine-readable.”
Generating Fact Checking Briefs (2020, methodology): “It has been treated as a classification problem (Wang, 2017), often using stance detection (Riedel et al., 2017).”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “Most of the fact veriﬁcation literature focuses on resolving claims to be true or false (Popat et al., 2018; Thorne and Vlachos, 2018; Wang, 2017).”
RAFT: A Real-World Few-Shot Text Classification Benchmark (2021, methodology): “For example, we considered including the LIAR [35] dataset which includes Politifact statements and their veracity.”
DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (2018, methodology): “This is in contrast to prior works (Popat et al., 2017; Rashkin et al., 2017; Wang, 2017) that ignore either the article or the claim, and therefore miss out on this important interaction.”
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge (2021, neutral): “…employed in settings like generative question answering [Fan et al., 2019, Lewis et al., 2020] and fact veriﬁcation [Vlachos and Riedel, 2014, Wang, 2017, Thorne et al., 2018], they should exhibit not just common-sense about everyday scenarios (physical, social, etc.), but factual knowledge…”
Where is Your Evidence: Improving Fact-checking by Justification Modeling (2018, methodology): “The LIAR dataset introduced by (Wang, 2017) consists of 12,836 short statements taken from POLITIFACT and labeled by humans for truthfulness, subject, context/venue, speaker, state, party, and prior history.”