Language Models as Fact Checkers?

By Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, Madian Khabsa. 2020

86 citations recorded by the source corpus.

Recent work has suggested that language models (LMs) store both common-sense and factual knowledge learned from pre-training data. In this paper, we leverage this implicit knowledge to create an effective end-to-end fact checker using a solely a language model, without any external knowledge or explicit retrieval components. While previous work on extracting knowledge from LMs have focused on the task of open-domain question answering, to the best of our knowledge, this is the first work to examine the use of language models as fact checkers. In a closed-book setting, we show that our zero-shot LM approach outperforms a random baseline on the standard FEVER task, and that our finetuned LM compares favorably with standard baselines. Though we do not ultimately outperform methods which use explicit knowledge bases, we believe our exploration shows that this method is viable and has much room for exploration.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Language models (LMs) have been suggested as repositories of real-world knowledge (Petroni et al., 2019) and there is much interest in using them for tasks such as closed-book question answering (QA; Roberts et al., 2020), fact veriﬁcation (Lee et al., 2020) and dialogue (Adiwardana et al., 2020).”
Survey of Hallucination in Natural Language Generation (2022, background): “Most of the literature leverages Wikipedia as the knowledge source [72, 132], which is only a small part of world knowledge.”
Editing Factual Knowledge in Language Models (2021, methodology): “This is done without retrieving any evidence from a corpus, instead, just by relying on the knowledge accumulated during pre-training and encoded in its own parameters—this is similar to Lee et al. (2020) that investigate closed-book and zero-shot FC us-ing masked-LMs.”
Evidence-based Factual Error Correction (2020, background): “While such approaches have been employed for fact veriﬁcation (Lee et al., 2020), these approaches share the following limitations.”
A Survey on Automated Fact-Checking (2021, result): “This task was also explored by Vo and Lee (2020) from a multi-modal perspective, where social media claims about images were matched against previously fact-checked claims.”
GooAQ: Open Question Answering with Diverse Answer Types (2021, background): “, 2021), and so do the models designed for them (Roberts et al., 2020; Lewis et al., 2020b; Lee et al., 2020; Min et al., 2021; Lewis et al., 2021).”
Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022, methodology): “…2020), yielding competitive performance on knowledge probing (Petroni et al., 2019; Shin et al., 2020; Zhong et al., 2021) and other knowledge-intensive tasks such as question answering (Roberts et al., 2020) and fact checking (Lee et al., 2020), without resorting to any exter-nal knowledge source.”