Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks

By Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Singh Sachdeva, Chitta Baral. 2020

14 citations recorded by the source corpus.

Numerical reasoning is often important to accurately understand the world. Recently, several format-specific datasets have been proposed, such as numerical reasoning in the settings of Natural Language Inference (NLI), Reading Comprehension (RC), and Question Answering (QA). Several format-specific models and architectures in response to those datasets have also been proposed. However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain. In pursuit of this goal, we introduce NUMBERGAME, a multifaceted benchmark to evaluate model performance across numerical reasoning tasks of eight diverse formats. We add four existing question types in our compilation. Two of the new types we add are about questions that require external numerical knowledge, commonsense knowledge and domain knowledge. For building a more practical numerical reasoning system, NUMBERGAME demands four capabilities beyond numerical reasoning: (i) detecting question format directly from data (ii) finding intermediate common format to which every format can be converted (iii) incorporating commonsense knowledge (iv) handling data imbalance across formats. We build several baselines, including a new model based on knowledge hunting using a cheatsheet. However, all baselines perform poorly in contrast to the human baselines, indicating the hardness of our benchmark. Our work takes forward the recent progress in generic system development, demonstrating the scope of these under-explored tasks.

Subfields: Computational Chemistry & Drug Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “Multitask learning is a long-standing goal for AI (Caruana, 1997) and has led to successful models that can support a wider range of tasks (McCann et al., 2018; Raffel et al., 2020; Khashabi et al., 2020; Mishra et al., 2020; Aghajanyan et al., 2021; Ye et al., 2021).”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, background): “Recent studies have explored the numerical capabilities learned by neural networks trained on large amounts of texts (Talmor et al., 2019; Jiang et al., 2019; Naik et al., 2019; Wallace et al., 2019; Lin et al., 2020; Johnson et al., 2020; Mishra et al., 2020).”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Recent studies have explored the numerical capabilities learned by neural networks trained on large amounts of texts (Talmor et al., 2019; Jiang et al., 2019; Naik et al., 2019; Wallace et al., 2019; Lin et al., 2020; Johnson et al., 2020; Mishra et al., 2020).”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “Several other works have followed similar approach, for example, by converting tasks to reading comprehension (Mishra et al., 2020) and textual entailment format (Wang et al., 2021b).”
Representing Numbers in NLP: a Survey and a Vision (2021, methodology): “Mishra et al. (2020) ﬁnd the commonsense question answering to be one of the hardest among their Num-bergame challenge, using the NumNetv2 model (Ran et al., 2019) which is commonly used for DROP question answering.”