Reasoning Over Semantic-Level Graph for Fact Checking

By Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, M. Zhou, Jiahai Wang, Jian Yin. 2019

186 citations recorded by the source corpus.

Fact checking is a challenging task because verifying the truthfulness of a claim requires reasoning about multiple retrievable evidence. In this work, we present a method suitable for reasoning about the semantic-level structure of evidence. Unlike most previous works, which typically represent evidence sentences with either string concatenation or fusing the features of isolated evidence sentences, our approach operates on rich semantic structures of evidence obtained by semantic role labeling. We propose two mechanisms to exploit the structure of evidence while leveraging the advances of pre-trained models like BERT, GPT or XLNet. Specifically, using XLNet as the backbone, we first utilize the graph structure to re-define the relative distances of words, with the intuition that semantically related words should have short distances. Then, we adopt graph convolutional network and graph attention network to propagate and aggregate information from neighboring nodes on the graph. We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy. Our model is the state-of-the-art system in terms of both official evaluation metrics, namely claim verification accuracy and FEVER score.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “MS-MARCO SotA is [4], FEVER-3 is [68] and FEVER-2 is [57] *Uses gold context/evidence.”
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “ProoFVer outperforms current state-of-theart (SOTA) fact verification models (Liu et al., 2020; Ye et al., 2020; Zhong et al., 2020) in both performance and robustness.”
Explainable Automated Fact-Checking for Public Health Claims (2020, methodology): “As with sentence selection approaches from the fact-checking literature (Nie et al., 2019; Zhong et al., 2019), we choose k = 5 .”
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims (2020, methodology): “We reach a competitive label-accuracy of 77.68% on the fever dev-set using our end-to-end pipeline ( sota label-accuracy of 79.16% on the dev-set is reported by [10]).”
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023, background): “Existing fact-verification approaches follow a multi-stage pipeline of claim detection, evidence retrieval and verdict prediction (Guo et al., 2022; Zhong et al., 2020).”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Sentence-level graphs can be useful in dialog tracking [331], fact checking [332], etc. Document-level graphs include knowledge graphs [333], Table VIII: Vision-language models.”
A Multi-Level Attention Model for Evidence-Based Fact Checking (2021, methodology): “Zhong et al. (2020) use semantic role labeling (SRL) to build a graph structure, where a node can be a word or a phrase depending on the SRL’s outputs.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “[52] Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin.”
FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023, background): “1, the creation of the FEVER dataset (Thorne et al., 2018a) spawned models (Zhong et al., 2020; Kr-ishna et al., 2022) that determine whether a given ﬁne-grained claim made based on Wikipedia 1 articles is correct.”
Kernel Graph Attention Network for Fact Verification (2019, methodology): “(Zhong et al., 2019) further establishes a semantic-level graph for claim and evidence reasoning.”
Fine-grained Fact Verification with Kernel Graph Attention Network (2019, methodology): “Zhong et al. (2019) further employs XLNet (Yang et al., 2019) and establishes a semantic-level graph for reasoning for a better performance.”