Fact or Fiction: Verifying Scientific Claims

By David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, Hannaneh Hajishirzi. 2020

755 citations recorded by the source corpus.

We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that supports or refutes a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that these models benefit from combined training on a large dataset of claims about Wikipedia articles, together with the new SciFact data. We show that our claim verification system is able to identify plausible evidence for 23 / 36 claims relevant to COVID-19 on the CORD-19 corpus. Our results and experiments strongly suggest that our new task and data will support significant future research efforts.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “Training datasets for evidence-based fact verification consist of instances containing a claim, a label indicating its veracity and the evidence, typically a set of sentences (Thorne et al., 2018a; Hanselowski et al., 2019; Wadden et al., 2020).”
Explainable Automated Fact-Checking: A Survey (2020, background): “, in the scientific (Wadden et al., 2020) and health domains (Kotonya and Toni, 2020), which includes natural language explanations (see Table 1).”
MSˆ2: Multi-Document Summarization of Medical Studies (2021, background): “…Li et al., 2016), relation extraction (Jain et al., 2020; Luan et al., 2018; Kringelum et al., 2016), QA (Abacha et al., 2019), NLI (Romanov and Shivade, 2018; Khot et al., 2018), summarization (Cachola et al., 2020; Chandrasekaran et al., 2019), claim verification (Wadden et al., 2020), and more.”
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022, methodology): “…one IR dataset (MS MARCO (Nguyen et al., 2016)) to evaluate supervised adaptation capability, and 4 other datasets (CQADupStack, TREC-COVID, SCIDOCS, Sci-Fact (Hoogeveen et al., 2015; Voorhees et al., 2020; Cohan et al., 2020; Wadden et al., 2020)) to evaluate unsupervised adaptation capability.”
On the Diversity and Limits of Human Explanations (2021, background): “Wadden et al. (2020) Fact verifica- tion An evidence set is a collection of sentences from the abstract that provide support or contradiction for the given claim.”
Attributed Text Generation via Post-hoc Research and Revision (2022, methodology): “Fact-checking Most models for fact-checking perform the following task: given a (statement, evidence) pair as input, they output a label indicating whether the statement is supported or refuted by the evidence (Thorne et al., 2018; Wang, 2017; Karadzhov et al., 2017; Augenstein et al., 2019; Wadden ”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, neutral): “76 Scientific SciFact [68] 3 Binary 920 —- 300 5,183 1.”
DialFact: A Benchmark for Fact-Checking in Dialogue (2021, background): “Fact-108 checking has been explored in variety of medium109 such as Wikipedia based claims (Schuster et al.,110 2021), claims over tables (Aly et al., 2021), sci-111 entific claims (Wadden et al., 2020), and social112 media claims (Nakov et al., 2021).”
Can We Automate Scientific Reviewing? (2021, background): “(8) Factuality and Reliability : A generated review should be factually correct (Wadden et al., 2020) which also poses challenge to the current evaluation methodology.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, methodology): “Fact-checking Most models for fact-checking perform the following task: given a (statement, evidence) pair as input, they output a label indicating whether the statement is supported or refuted by the evidence (Thorne et al., 2018; Wang, 2017; Karadzhov et al., 2017; Augenstein et al., 2019; Wadden ”
Rational LAMOL: A Rationale-based Lifelong Learning Framework (2021, background): “• SciFact (Wadden et al., 2020): a dataset containing expert-written scientific claims coupled with evidence-containing abstracts.”
Cross-Domain Transfer of Generative Explanations Using Text-to-Text Models (2021, methodology): “These results are not directly comparable with [42], since we consider the subtask of label prediction and rationalization for the subset of refuted and supported claims.”
Evidence Selection as a Token-Level Prediction Task (2021, background): “In fact, the authors in (Wadden et al., 2020) also explored paragraph-level evidence selection using Longformer (Beltagy et al.”
Benchmarking Retrieval-Augmented Generation for Medicine (2024, background): “Fourth, while QA is the most commonly used task for evaluating biomedical LLMs, there are also other knowledge-intensive tasks that might benefit from M ED R AG , such as claim verification (Wadden et al., 2020; Liu et al., 2024).”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “[53] David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “Our work is closely related to prior work on fact verification (Thorne et al., 2018; Wadden et al., 2020) where claim sentences are automatically checked against a large knowledge source like Wikipedia or scientific literature.”