Predicting Factuality of Reporting and Bias of News Media Sources

By Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James R. Glass, Preslav Nakov. 2018

277 citations recorded by the source corpus.

We present a study on predicting the factuality of reporting and bias of news media. While previous work has focused on studying the veracity of claims or documents, here we are interested in characterizing entire news media. This is an under-studied, but arguably important research problem, both in its own right and as a prior for fact-checking systems. We experiment with a large list of news websites and with a rich set of features derived from (i) a sample of articles from the target news media, (ii) its Wikipedia page, (iii) its Twitter account, (iv) the structure of its URL, and (v) information about the Web traffic it attracts. The experimental results show sizable performance gains over the baseline, and reveal the importance of each feature type.

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “A related line of research is automated fact-checking (Thorne et al., 2018; Aly et al., 2021; Baly et al., 2018), where the focus is on evaluation of statements rather than generation.”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “Toxicity from Unreliable News Sites Gathering all documents in OWTC associated with a news site, and cross-referencing reliability ratings from Baly et al. (2018), we find that news reliability correlates negatively with the proportion of documents that are toxic (Spearman ρ = –0.”
Survey of Hallucination in Natural Language Generation (2022, background): “, web traffic, PageRank or URL structure) have been proposed to deal with this trustworthiness issue [7, 144, 145].”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, methodology): “, 2021) 1M (1M) BREAKING NEWS† 20K latest articles from 400 English news sites (Baly et al., 2018) 11M (1M) CONTRACTS† 500 commercial legal contracts (Hendrycks et al.”
Can Large Language Models Transform Computational Social Science? (2023, background): “For a variety of specific phenomena (Amber et al., 2013; Baly et al., 2018; Roy and Goldwasser, 2020; Luo et al., 2020b; Ziems and Yang, 2021), this takes the form of gathering articles from across the political spectrum, processing each one further for a phenomenon of interest, and evaluating the…”
Evaluating adversarial attacks against multiple fact verification systems (2019, background): “As this is a task with potentially sensitive applications like propaganda (Baly et al., 2018a) or biased news detection (Potthast et al.”
A Transformer-based Framework for Neutralizing and Reversing the Political Polarity of News Articles (2021, background): “Recent work tries to adopt more advanced network structures, since recurrent neural networks are known for having the limited capability of modelling long text such as full-length news articles [9].”
FANG: Leveraging Social Context for Fake News Detection Using Graph Representation (2020, background): “This allows the representation of heterogeneous entities to be dependent, leveraging not only fake news detection but also related social analysis tasks such as malicious user detection [7] and source factuality prediction [3].”
DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification (2020, background): “…Wu et al., 2020), emotions (Ajao et al., 2019), stances (Ma et al., 2018a; Kochkina et al., 2018; Wu et al., 2019), write styles (Potthast et al., 2018; Gr¨ondahl and Asokan, 2019), and source credibility (Popat et al., 2018; Baly et al., 2018a) from claims and relevant articles (or comments).”
A Survey of Fake News (2020, methodology): “Algorithms to identify Web spam thus can be classified into (i) content-based algorithms, which analyze web content features, such as word counts and content duplication [Baly et al. 2018; Fetterly et al. 2005; Ntoulas et al. 2006]; (ii) link-based algorithms, which detect web spam by utilizing grap”
SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums (2019, methodology): “, there has been work on checking the factuality/credibility of a claim, of a news article, or of an information source (Ba et al., 2016; Zubiaga et al., 2016; Ma et al., 2016; Castillo et al., 2011; Baly et al., 2018).”
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022, methodology): “Turning to the TOEFL exam responses, we would expect that if the ﬁlter agrees with essay 19 Baly et al. (2018) scores, higher scoring essays would receive higher quality scores.”