TabFact: A Large-scale Dataset for Table-based Fact Verification

By Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, SHIYANG LI, Xiyou Zhou, William Yang Wang. 2019

700 citations recorded by the source corpus.

The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities. The data and code of the dataset are provided in \url{this https URL}.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

UL2: Unifying Language Learning Paradigms (2022, background): “We use several component tasks from UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Berant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “• Formal language to text translation describes formal language in natural language (Chen et al., 2020d).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…and Camacho-Collados 2019 superglue-wsc cls/other Levesque et al. 2012 swag qa/multiple-choice qa Zellers et al. 2018 tab_fact cls/fact checking Chen et al. 2020b trec cls/other Li and Roth 2002; Hovy et al. 2001 trec-finegrained cls/other Li and Roth 2002; Hovy et al. 2001 tweet_eval-emoji…”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “…ability for reasoning and numerical inference (Wiseman et al., 2017) and for building representations of structured objects (Liu et al., 2018), providing an interesting complement to tasks that test these aspects in the NLU setting (Pasupat and Liang, 2015; Chen et al., 2019; Dua et al., 2019).”
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data (2020, background): “Different datasets like WikiTableQuestions (Pa-supat and Liang, 2015), WikiSQL (Zhong et al., 2017), SPIDER (Yu et al., 2018), TabFact (Chen et al., 2020) have been proposed to build models which can interact with such structured information.”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, background): “English Wikipedia includes millions of tables with high lexical and domain diversity (Fetahu et al., 2019; Chen et al., 2020b; Gupta et al., 2020b; Talmor et al., 2021; Nan et al., 2021; Neeraja et al., 2021a).”
SpreadsheetCoder: Formula Prediction from Semi-structured Context (2021, methodology): “First, these models are designed for question answering (Herzig et al., 2020; Yin et al., 2020) or fact veriﬁcation (Chen et al., 2020), thus their inputs are the concatenation of a natural language question/statement and a table.”
BioTABQA: Instruction Learning for Biomedical Table Question Answering (2022, methodology): “To overcome this limitation, TabFact [17] is proposed which enables TQA when the answer is not explicitly available in the table.”
TaPas: Weakly Supervised Table Parsing via Pre-training (2020, methodology): “As for tables, Chen et al. (2019) experimented with rendering a table into natural language so that it can be handled with a pre-trained BERT model.”
StructGPT: A General Framework for Large Language Model to Reason over Structured Data (2023, methodology): “For Table based QA (TableQA), we adopt three widely-used datasets, weakly-supervised Wik-iSQL (WikiSQL) (Zhong et al., 2017), WikiTable-Questions (WTQ) (Pasupat and Liang, 2015), and TabFact (Chen et al., 2020).”
FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information (2021, methodology): “Furthermore, table-based datasets, e.g. Chen et al. [2020a], assume an unrealistic setting where an evidence table is provided, requiring extensions to evaluate retrieval [Schlichtkrull et al., 2020].”
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms (2020, methodology): “To remove incorrect logical forms, we ﬁrst execute the logical forms against their corresponding tables and then perform another round of semantic veriﬁcation. logical form Execution The functionality in our logical form is based on the ones used in (Chen et al., 2019b).”
Table Fact Verification with Structure-Aware Transformer (2020, methodology): “1627 Model Val Test Test(simple) Test(complex) LPA(Wenhu et al., 2020)† 65.”
Binding Language Models in Symbolic Languages (2022, methodology): “We demonstrate the effectiveness of the B INDER framework on W IKI T ABLE Q UESTIONS (W IKI TQ; Pasupat & Liang, 2015) T AB F ACT (Chen et al., 2019a), two structured knowledge grounding datasets that require complex reasoning on the tables.”
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning (2022, background): “Two recently created tabular Natural language Inference (NLI) datasets, TabFact (Chen et al., 2020b) on Wikipedia relational tables and INFOTABS (Gupta et al.”