Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking

By Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, Yejin Choi. 2017

920 citations recorded by the source corpus.

We present an analytic study on the language of news media in the context of political fact-checking and fake news detection. We compare the language of real news with that of satire, hoaxes, and propaganda to find linguistic characteristics of untrustworthy text. To probe the feasibility of automatic political fact-checking, we also present a case study based on PolitiFact.com using their factuality judgments on a 6-point scale. Experiments show that while media fact-checking remains to be an open research question, stylistic cues can help determine the truthfulness of text.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Defending Against Neural Fake News (2019, methodology): “E↵orts to automate fake news detection generally point out stylistic biases that exist in the text (Rashkin et al., 2017; Wang, 2017; Pérez-Rosas et al., 1 We thank past work, such as OpenAI’s Staged Release Policy for GPT2 for drawing attention to neural disinformation, alongside other dual-use…”
Reasoning Over Semantic-Level Graph for Fact Checking (2019, background): “…Ferreira and Vlachos (2016) who build Emergent with 300 labeled rumors and about 2.6K news articles, Wang (2017) who builds LIAR with 12.8K annotated short statements and six ﬁne-grained labels, and Rashkin et al. (2017) who collect claims without meta-data while providing 74K news articles.”
Automatically Neutralizing Subjective Bias in Text (2019, methodology): “Several studies develop methods for highlighting subjective or persuasive frames in a text (Rashkin et al. 2017; Tsur, Calacci, and Lazer 2015), or detecting biased sentences (Hube and Fetahu 2018; Morstatter et al. 2018; Yang et al. 2017; Hube and Fetahu 2019) of which the most similar to ours is…”
Predicting Factuality of Reporting and Bias of News Media Sources (2018, background): “Rashkin et al. (2017) focused on the language used by “fake news” and compared the prevalence of several features in articles coming from trusted sources vs. hoaxes vs. satire vs. propaganda.”
Automated Fact Checking: Task Formulations, Methods and Future Directions (2018, methodology): “Rashkin et al. (2017) collated claims from Politifact without meta-data.”
Fake News Detection on Social Media using Geometric Deep Learning (2019, background): “Content-based approaches, which are used in the majority of works on fake news detection, rely on linguistic (lexical and syntactical) features that can capture deceptive cues or writing styles [1, 32, 30, 29, 28].”
Combating Fake News (2019, methodology): “d due to the its ability to effectively capture long range dependencies in the text, and has been applied to fake news detection, similar to the use of convolutional neural networks, in several works [91, 117]; whereas in another variant, LSTM has been applied to both, article headline and article t”
A Survey on Computational Propaganda Detection (2020, methodology): “Rashkin et al. [ 2017 ] released TSHP-17 , a balanced corpus with document-level annotation including four classes: trusted , satire , hoax , and propaganda .”
Program Enhanced Fact Verification with Verbalization and Graph Attention Network (2020, background): “…available on the Internet, fact veriﬁcation has become crucial for many applications such as detecting fake news, ru-mors, and political deception (Rashkin et al., 2017; Thorne et al., 2018; Goodrich et al., 2019; Vaibhav et al., 2019; Kry ´ sci ´ nski et al., 2019 and processing evidences from…”
Explainable Automated Fact-Checking: A Survey (2020, background): “…are the following: identifying the severity or nature of false claims, i.e., distinguishing between hoaxes, rumors and satire (Rubin et al., 2016; Rashkin et al., 2017); detecting check-worthy claims (Hassan et al., 2015; Hassan et al., 2017); distinguishing between hyper-partisan and mainstream…”
Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection (2019, background): “Rashkin et al. (2017) created a corpus of news articles labelled as propaganda, trusted, hoax, or satire.”
The Limitations of Stylometry for Detecting Machine-Generated Fake News (2019, background): “…and the increasing interest in fake news, recent studies applied stylometry on entire news articles (Pisarevskaya 2017), short news reports (Pérez-Rosas et al. 2018), fact and political statements (Nakashole and Mitchell 2014; Rashkin et al. 2017) and posts in social media (Volkova et al. 2017).”
DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (2018, methodology): “This is in contrast to prior works (Popat et al., 2017; Rashkin et al., 2017; Wang, 2017) that ignore either the article or the claim, and therefore miss out on this important interaction.”
Evidence-based Verification for Real World Information Needs (2021, background): “One potential risk of using scraped claims is that linguistic cues (Rashkin et al., 2017) provide indication to a claim’s veracity without the need to use textual evidence (Wang, 2017).”
Automated Fact-Checking for Assisting Human Fact-Checkers (2021, positive): “This further includes analysis of the language used in the claims based on lexicons such as LIWC [Rashkin et al., 2017], or using perplexity analysis [Lee et al.”
Improving Large-Scale Fact-Checking using Decomposable Attention Models and Lexical Tagging (2018, methodology): “Similar to the FakeNews, (Rashkin et al., 2017; Vlachos and Riedel, 2014) focused on political statements from Politifact.com to verify the degree of truthfulness.”