NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks

By Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Singh Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan. 2022

152 citations recorded by the source corpus.

Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4 %). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4 % on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.

Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

In Evaluation & Benchmarks

Citations in the corpus

Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “S T A N C E [ 293 ] , 20 M i n u t e n [ 108 ] , C o d e X G L U E [ 141 ] , N u m G L U E [ 162 ] C o s m o s Q A [ 90 ] , D R O P [ 56 ] , E ss e n t i a l -”
Is a Question Decomposition Unit All We Need? (2022, background): “2) language models by default struggle to do math tasks (Patel et al., 2021; Mishra et al., 2022), so the performance improvement seems more prominent there.”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “Several other works have followed similar approach, for example, by converting tasks to reading comprehension (Mishra et al., 2022) and textual entailment format (Wang et al.”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, background): “, 2022) and complex mathematical reasoning challenges (Mishra et al., 2022; Imani et al., 2023).”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “There is strong evidence that deep learning meth-ods for mathematical reasoning are not robust and susceptible to adversarial attacks (Lin et al., 2020; Patel et al., 2021; Mishra et al., 2022b,a; Welleck et al., 2022b).”
LILA: A Unified Benchmark for Mathematical Reasoning (2022, background): “NumGLUE (Mishra et al., 2022c) has been proposed as a multi-tasking numerical reasoning benchmark that contains 8 different tasks.”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “For the in-domain datasets, we consider GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), AQuA-RAT (Ling et al., 2017), and NumGLUE (Mishra et al., 2022b).”