TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

By Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-seng Chua. 2021

562 citations recorded by the source corpus.

Hybrid data combining both tabular and textual content (e.g., financial reports) are quite pervasive in the real world. However, Question Answering (QA) over such hybrid data is largely neglected in existing research. In this work, we extract samples from real financial reports to build a new large-scale QA dataset containing both Tabular And Textual data, named TAT-QA, where numerical reasoning is usually required to infer the answer, such as addition, subtraction, multiplication, division, counting, comparison/sorting, and the compositions. We further propose a novel QA model termed TAGOP, which is capable of reasoning over both tables and text. It adopts sequence tagging to extract relevant cells from the table along with relevant spans from the text to infer their semantics, and then applies symbolic reasoning over them with a set of aggregation operators to arrive at the final answer. TAGOP achieves 58.0% inF1, which is an 11.1% absolute increase over the previous best baseline model, according to our experiments on TAT-QA. But this result still lags far behind performance of expert human, i.e.90.8% in F1. It is demonstrated that our TAT-QA is very challenging and can serve as a benchmark for training and testing powerful QA models that address hybrid form data.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangnan He
2. Tat-seng Chua
3. Fuli Feng
4. Xiang Wang
5. Hanwang Zhang
6. Wenhu Chen
7. Pan Lu
8. Jian-Guang Lou

In Management Science and Operations Research

1. Xiangnan He
2. Tat-seng Chua
3. Fuli Feng
4. Ming Yin
5. Weizhu Chen
6. Zeqi Lin
7. Pan Lu
8. Wenhu Chen

Citations in the corpus

• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, methodology): “Besides MWP, some datasets also consider financial problems (Chen et al., 2021b, 2022; Zhu et al., 2021), where the intelligent systems need to answer math-driven financial questions.”
• Making Language Models Better Reasoners with Step-Aware Verifier (2022, background): “…2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).”
• Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, methodology): “Firstly, we decide to choose the approach of explicitly modeling discrete operations, since existing NDR solutions have demonstrated its superiority (Dua et al., 2019; Ran et al., 2019; Herzig et al., 2020; Zhu et al., 2021).”
• Large Language Models are few(1)-shot Table Reasoners (2022, background): “Recently, a wide range of table-based tasks have been proposed like table question answering (Pasu-pat and Liang, 2015; Chen et al., 2020c; Zhu et al., 2021; Chen et al., 2021b; Talmor et al., 2020; Chen et al., 2020a; Nan et al., 2022), table fact verifi-cation (Chen et al., 2019; Aly et al.,…”
• Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning (2022, unknown): “Note that 77.6% of questions in TAT-QA can be solvable without mathematical reasoning and 50.0% of questions in FinQA are not table-must to be answered.”
• TheoremQA: A Theorem-driven Question Answering dataset (2023, methodology): “…University GMAT/GRE NO MathQA (Amini et al., 2019) Arithm/Algebra University AQuA NO MathQA-Python (Austin et al., 2021) Arithm/Algebra University AQuA NO FinQA (Chen et al., 2021c) Finance University CrowdSource NO TAT-QA (Zhu et al., 2021) Finance University CrowdSource NO on our dataset.”
• What matters when building vision-language models? (2024, unknown): “TabMWP (Lu et al., 2023) 22,729 23,059 1,948,166 2.49% TAT-QA (Zhu et al., 2021) 2,199 13,215 283,776 2.18% HiTab (Cheng et al., 2022) 2,500 7,782 351,299 1.80% MultiHiertt (Zhao et al., 2022) 7,619 7,830 267,615 1.37% FinQA (Chen et al., 2021) 5,276 6,251 242,561 0.99% WikiSQL (Zhong et al., 2017) ”
• A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “, 2021c), TAT-QA (Zhu et al., 2021), and MultiHiertt (Zhao et al.”