Generating Fact Checking Briefs

By Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, Sebastian Riedel. 2020

66 citations recorded by the source corpus.

Fact checking at scale is difficult -- while the number of active fact checking websites is growing, it remains too small for the needs of the contemporary media ecosystem. However, despite good intentions, contributions from volunteers are often error-prone, and thus in practice restricted to claim detection. We investigate how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check, in the form of natural language briefs. We investigate passage-based briefs, containing a relevant passage from Wikipedia, entity-centric ones consisting of Wikipedia pages of mentioned entities, and Question-Answering Briefs, with questions decomposing the claim, and their answers. To produce QABriefs, we develop QABriefer, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers. To train its components, we introduce QABriefDataset which we collected via crowdsourcing. We show that fact checking with briefs -- in particular QABriefs -- increases the accuracy of crowdworkers by 10% while slightly decreasing the time taken. For volunteer (unpaid) fact checkers, QABriefs slightly increase accuracy and reduce the time required by around 20%.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “Augenstein et al. [19] and Fan et al. [20] make use of web data through commercial search engines as a part of building more diverse datasets for fact-checking.”
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “, 2021), answer claim related questions (Fan et al., 2020), or perform rule discovery (Ahmadi et al.”
Recursively Summarizing Books with Human Feedback (2021, methodology): “In Fan et al. (2020), models are used to search for evidence of facts, to help humans fact check faster and more accurately.”
Self-critiquing models for assisting human evaluators (2022, background): “For helping humans with evaluations, [FPP20] help humans fact-check claims faster and more accurately with natural language briefs.”
Internet-Augmented Dialogue Generation (2021, methodology): “We use the pre-trained DPR model from the KILT Benchmark (Petroni et al., 2020).”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “Generating texts that match given facts is a known challenge (Fan et al., 2020; Kryscinski et al., 2020; Lewis et al., 2020b; Parikh et al., 2020; Shah et al., 2020; Tian et al., 2020) as language models tend to degenerate and hallucinate (Holtzman et al., 2020; Schuster et al., 2020; Zhou et al.,…”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “In many real-world situations, relevant evidence for a statement is not readily available, so some works have explored how to automatically retrieve evidence that may help support or refute a statement (Fan et al., 2020; Piktus et al., 2021), just as we do.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “There also has been work that verifies a longer sentence or text through decomposition to atomic facts (Fan et al., 2020; Wright et al., 2022; Chen et al., 2022; Kamoi et al., 2023) from which we take inspiration.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “In many real-world situations, relevant evidence for a statement is not readily available, so some works have explored how to automatically retrieve evidence that may help support or refute a statement (Fan et al., 2020; Piktus et al., 2021), just as we do.”
Policy Compliance Detection via Expression Tree Inference (2022, methodology): “…generation (Heilman and Smith, 2010) of this form is typically employed as part of a pipeline for question-answering (Duan et al., 2017; Wang et al., 2020; Lewis et al., 2021) or for generating questions for fact-checking summary briefs (Hassan et al., 2017; Jimenez, 2017; Fan et al., 2020).”