HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
By Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, William Yang Wang. 2020
415 citations recorded by the source corpus.
Existing question answering datasets focus on dealing with homogeneous information, based either only on text or KB/Table information alone. However, as human knowledge is distributed over heterogeneous forms, using homogeneous information alone might lead to severe coverage problems. To fill in the gap, we present HybridQA, a new large-scale question-answering dataset that requires reasoning on heterogeneous information. Each question is aligned with a Wikipedia table and multiple free-form corpora linked with the entities in the table. The questions are designed to aggregate both tabular information and text information, i.e., lack of either form would render the question unanswerable. We test with three different models: 1) a table-only model. 2) text-only model. 3) a hybrid model that combines heterogeneous information to find the answer. The experimental results show that the EM scores obtained by two baselines are below 20%, while the hybrid model can achieve an EM over 40%. This gap suggests the necessity to aggregate heterogeneous information in HybridQA. However, the hybrid model’s score is still far behind human performance. Hence, HybridQA can serve as a challenging benchmark to study question answering with heterogeneous information.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Caiming Xiong
- Jianfeng Gao
- Percy Liang
- Hannaneh Hajishirzi
- Yejin Choi
- Graham Neubig
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Mohit Bansal
- Cordelia Schmid
- Jitendra Malik
- Ming-Hsuan Yang
In Evaluation & Benchmarks
- Hannaneh Hajishirzi
- Andreas Vlachos
- Ashish Sabharwal
- Mohit Bansal
- James Thorne
- Tushar Khot
- Caiming Xiong
- Jonathan Berant
In Question Answering & Reading Comprehension
- Ashish Sabharwal
- Tushar Khot
- Peter Clark
- Hannaneh Hajishirzi
- Jonathan Berant
- Harsh Trivedi
- William W. Cohen
- Luke S. Zettlemoyer
Citations in the corpus
- UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “• Formal language to text translation describes formal language in natural language (Chen et al., 2020d).”
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, neutral): “Each question in our dataset require 2-15 Wikipedia articles to answer, with the distribution of the percentage of dataset requiring different numbers of Wikipedia articles shown in Figure 2 (left).”
- On the Advance of Making Language Models Better Reasoners (2022, background): “…2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).”
- AgentBench: Evaluating LLMs as Agents (2023, methodology): “We acquire the source queries and databases via reusing and amalgamating several established datasets: WikiSQL [103], WikiTableQuestions [60], SQA [37], HybridaQA [13], and FeTaQA [53], ensuring the diversity of instructions and data.”
- FeTaQA: Free-form Table Question Answering (2021, methodology): “TabFact (Chen et al., 2020d) is composed of Wikipedia tables coupled with statements labeled as either “ENTAILED” or “REFUTED” by the table.”
- FinQA: A Dataset of Numerical Reasoning over Financial Data (2021, methodology): “As most existing QA datasets were constructed by MTurk workers (Yang et al., 2018; Dua et al., 2019; Chen et al., 2020c), it requires substantial domain-specific knowledge to compose meaningful questions that are hard for computers to answer.”
- UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering (2020, methodology): “We tried two types of tables linearization: ‘template’-like encoding used in recent literature (Chen et al., 2020a) and a simpler one which we find works the best in our experiments (see Table 4, bottom half).”
- FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information (2021, background): “Chen et al. [2020b] proposed HybridQA, a dataset consisting of multi-hop questions constructed by using Wikipedia tables and the introductory section of linked entities in the table, however, their dataset assumes the table as part of the input.”
- Unifying Language Learning Paradigms (2022, methodology): “…from UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Be-rant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al., 2017), MTOP (Li et al.,…”
- MultiModalQA: Complex Question Answering over Text, Tables and Images (2021, background): “Recently, Chen et al. (2020b) presented H YBRID QA, a dataset that requires reasoning over tabular and textual data.”
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2021, background): “Other multihop datasets (Khashabi et al., 2018; Dua et al., 2019, inter alia ) focus on different challenges such as multiple modalitites (Chen et al., 2020; Talmor et al., 2021), open-domain QA (Geva et al., 2021; Khot et al., 2020), fact verification (Jiang et al., 2020), science explanations…”
- Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning (2022, background): “Recent work aims at introducing datasets that require multi-hop reasoning between the textual and tabular data: Hy-bridQA (Chen et al., 2020b), OTTQA (Chen et al., 2020a), MultiModalQA (Talmor et al., 2020), AIT-QA (Katsis et al., 2021), and FeTaQA (Nan et al., 2022).”
- MATE: Multi-view Attention for Table Transformer Efficiency (2021, methodology): “While we apply our model to several such tasks (see section 6), HYBRIDQA (Chen et al., 2020b) is particularly interesting, as it requires processing tables jointly with long passages associated with entities mentioned in the table, yielding large documents that may not fit in standard Transformer…”
- Large Language Models are few(1)-shot Table Reasoners (2022, background): “Recently, a wide range of table-based tasks have been proposed like table question answering (Pasu-pat and Liang, 2015; Chen et al., 2020c; Zhu et al., 2021; Chen et al., 2021b; Talmor et al., 2020; Chen et al., 2020a; Nan et al., 2022), table fact verifi-cation (Chen et al., 2019; Aly et al.,…”
- Making Language Models Better Reasoners with Step-Aware Verifier (2022, background): “…2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).”
- Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021, background): “While QG/QA modeling per se is out of the scope of this paper, further improvements could be obtained thanks to advances in multimodal QA, a field particularly prolific (Chen et al., 2020; Herzig et al., 2020; Nan et al., 2021).”
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020, background): “Recently, Chen et al. (2020) introduced the HybridQA dataset—a multi-hop question answering over both tabular and textual data.”