Combining Fact Extraction and Verification with Neural Semantic Matching Networks

By Yixin Nie, Mohit Bansal. 2018

351 citations recorded by the source corpus.

The increasing concern with misinformation has stimulated research efforts on automatic fact checking. The recentlyreleased FEVER dataset introduced a benchmark factverification task in which a system is asked to verify a claim using evidential sentences from Wikipedia documents. In this paper, we present a connected system consisting of three homogeneous neural semantic matching models that conduct document retrieval, sentence selection, and claim verification jointly for fact extraction and verification. For evidence retrieval (document retrieval and sentence selection), unlike traditional vector space IR models in which queries and sources are matched in some pre-designed term vector space, we develop neural models to perform deep semantic matching from raw textual input, assuming no intermediate term representation and no access to structured external knowledge bases. We also show that Pageview frequency can also help improve the performance of evidence retrieval results, that later can be matched by using our neural semantic matching network. For claim verification, unlike previous approaches that simply feed upstream retrieved evidence and the claim to a natural language inference (NLI) model, we further enhance the NLI model by providing it with internal semantic relatedness scores (hence integrating it with the evidence retrieval modules) and ontological WordNet features. Experiments on the FEVER dataset indicate that (1) our neural semantic matching method outperforms popular TF-IDF and encoder models, by significant margins on all evidence retrieval metrics, (2) the additional relatedness score and WordNet features improve the NLI model via better semantic awareness, and (3) by formalizing all three subtasks as a similar semantic matching problem and improving on all three stages, the complete model is able to achieve the state-of-the-art results on the FEVER test set (two times greater than baseline results).1

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

Citations in the corpus

Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, background): “A common model for fact verification (Pomerleau and Rao, 2017; Thorne et al., 2018) is text-pair classification between a claim and evidence retrieved from a trusted source (Nie et al., 2018; Yoneda et al., 2018; Hanselowski et al., 2018, inter alia).”
Reasoning Over Semantic-Level Graph for Fact Checking (2019, methodology): “Top-ranked three systems aggregate pieces of evidence through concatenating evidence sentences into a single string (Nie et al., 2019), classifying each evidence-claim pair separately, merging the results (Yoneda et al., 2018), and encoding each evidence-claim pair followed by pooling operation…”
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale (2019, methodology): “The scores were proposed in Nie et al. (2019) and indicate the upperbound of ﬁnal FEVER Score at one intermediate layer assuming all downstream modules are perfect.”
Measuring Attribution in Natural Language Generation Models (2021, background): “Modeling approaches have overlapped with recent literature examining natural language inference (Nie et al., 2019).”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, methodology): “For fact checking, we consider NSMN (Nie et al., 2019), the highest scoring system from the FEVER shared task (Thorne et al., 2018b).”
The FEVER2.0 Shared Task (2019, methodology): “Team DOMLIN (Stammbach and Neumann, 2019) used the document retrieval module of Hanselowski et al. (2018) and a BERT model for two-staged sentence selection based on the work by (Nie et al., 2019).”
Explainable Automated Fact-Checking for Public Health Claims (2020, methodology): “As with sentence selection approaches from the fact-checking literature (Nie et al., 2019; Zhong et al., 2019), we choose k = 5 .”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, methodology): “…model (Liu et al., 2019) that has262 been finetuned on SNLI (Bowman et al., 2015),263 MNLI (Williams et al., 2018a), and FEVER-264 NLI (Nie et al., 2019).2 We use this training data265 as it provides us with an accurate initial model,266 and note that we collapse the neutral and…”
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims (2020, methodology): “We adopt a pipeline approach for our evidence retrieval and claim validation system similar to the baseline system proposed by fever [1] and similar to virtually all competing implementations that followed [7, 8, 9, 10, 11].”
Towards Debiasing Fact Verification Models (2019, methodology): “For instance, the leading evidence-aware system in the FEVER Shared Task, the NSMN classiﬁer by Nie et al. (2019) 2 , achieves only 58.7% accuracy on the symmetric test set compared to 81.8% on the original dataset.”
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification (2019, methodology): “…Malon (2018) ﬁne-tunes the generative pre-training transformer (GPT) (Radford et al., 2018) for FV. Based on the methods mentioned above, Nie et al. (2019) specially design the neural semantic matching network (NSMN), which is a modiﬁcation of ESIM and achieves the best results in the…”
Evidence Selection as a Token-Level Prediction Task (2021, background): “Multi-hop Retrieval: Some systems follow hyperlinks present in a Wikipedia article to explore additional documents (see e.g. Nie et al., 2018; Stammbach and Neumann, 2019).”
Explainable Automated Fact-Checking: A Survey (2020, methodology): “The only exception is Nie et al. (2019), whose method uses a DNN approach, i.e., neural semantic matching, for the entire pipeline (i.e., for all subtasks).”
Language Models as Fact Checkers? (2020, background): “Traditional pipeline Traditional fact-checking pipelines (Malon, 2018; Hanselowski et al., 2018; Yoneda et al., 2018; Lee et al., 2018; Nie et al., 2019) access knowledge within an external knowledge base (i.”
WikiFactFind: Semi-automated fact-checking based on Wikipedia (2022, methodology): “The UNC-NLP solution achieves 0.64 FEVER score comparing to 0.28 of baseline [11, 12].”
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021, methodology): “We use the NLI-version of FEVER, released by (Nie et al., 2019): claims are hypotheses; premises corresponding to “support” or “refute’ claims consist of ground truth textual evidence and some other randomly sampled evidence; premises for “not-enough-info” claims are the concatenation of all…”
Annotating and Modeling Fine-grained Factuality in Summarization (2021, background): “line of work has looked at fact verification (Thorne et al., 2018; Nie et al., 2019; Atanasova et al., 2020).”