Analysing Mathematical Reasoning Abilities of Neural Models

By David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli. 2019

524 citations recorded by the source corpus.

Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test splits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, positive): “Inspired by the “polynomial evaluation” subproblem in Saxton et al. (2019), we generate a dataset of polynomials of degree less than or equal to three, with integer coefﬁcients and inputs constrained to the range [ − 10 , 10] .”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Math questions (Saxton et al., 2019) For all tasks, our finetuning and evaluation code uses tensorflow datasets (TFDS) to load and process datasets.”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “(Bobrow, 1964; Char-niak, 1969; Shi et al., 2015; Ling et al., 2017; Amini et al., 2019; Parisotto et al., 2016; Wang et al., 2018; Zou and Lu, 2019) as well as datasets (Ling et al., 2017; Amini et al., 2019; Huang et al., 2016; Saxton et al., 2019) An interesting approach is proposed in (Rabe et…”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “It consists of data from 22 data sources, coarsely broken down into 5 categories: • Academic Writing: Pubmed Abstracts and PubMed Central, arXiv, FreeLaw,5 USPTO Backgrounds,6 PhilPapers,7 NIH Exporter8 • Web-scrapes and Internet Resources: CommonCrawl, OpenWebText2, StackExchange,9 Wikipedia (Engli”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, methodology): “Saxton et al. (2019) introduce the DeepMind Mathematics dataset, which consists of algorithmically generated plug-and-chug problems such as addition, list sorting, and function evaluation, as shown in Figure 1.”
Transformers as Soft Reasoners over Language (2020, background): “, 2019], integration [Lample and Charton, 2019], and math [Saxton et al., 2019].”
Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension (2019, background): “Other recent work has shown that even state-of-the-art neural models struggle with numerical operations and quantitative reasoning when trained in an end-to-end manner (Saxton et al., 2019; Ravichander et al., 2019).”
Scaling Laws for Autoregressive Generative Modeling (2020, methodology): “We studied this phenomenon in the fundamental ﬁgure 3, where we evaluate problem solving performance using a variety of test sets indexed by a numerical level, which corresponds to an ‘entropy’ used for generation [SGHK19].”
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, methodology): “Unfortunately, the compute requirements of training on the Mathematics Dataset currently makes using the full tensor product infeasible, unless the vector representations of symbols and roles are reduced to dimensions that proved to be too small for the task.”
Compositionality Decomposed: How do Neural Networks Generalise? (2019, background): “…the vast number of different tests that exist for compositionality [ Lake and Baroni, 2018; Hupkes et al. , 2018; Johnson et al. , 2017; Bahdanau et al. , 2018; Saxton et al. , 2019; Loula et al. , 2018; Dess`ı and Baroni, 2019; Liˇska et al. , 2018; Bowman et al. , 2015; Mul and Zuidema, 2019 ] .”
Generative Language Modeling for Automated Theorem Proving (2020, background): “Also, Saxton et al. [41] evaluated the Transformer architecture on a variety of mathematical problems.”
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (2022, background): “In Saxton et al. (2019) they study generalization on procedurally generated math problems such as arithmetic and differentiation, but for the most part these tasks are more involved than the simple binary op problems we have studied and as such do not lend themselves to observing the kinds of…”
Deep Learning for Symbolic Mathematics (2019, methodology): “Saxton et al. (2019) use LSTMs (Hochreiter & Schmidhuber, 1997) and transformers on a wide range of problems, from arithmetic to simpliﬁcation of formal expressions.”
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019, background): “Recent datasets such as DROP (Dua et al., 2019), EQUATE (Ravichander et al., 2019), or Mathe-matics Questions (Saxton et al., 2019) test numerical reasoning; they contain examples which require comparing, sorting, and adding numbers in natural language (e.g., Figure 2).”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “…Prompting has been widely applied to study the factual (Petroni et al., 2019), commonsense (Davison et al., 2019; Weir et al., 2020; Lin et al., 2020), mathematical (Saxton et al., 2019), and other NLP task-related (Rad-ford et al., 2019; Shin et al., 2020) knowledge LMs acquire during pretraining.”
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, methodology): “The mathematics dataset (Saxton et al., 2019) is a large, automatically generated set of 112M samples in 56 separated sub-tasks.”
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, background): “Recently, more challenging datasets (Dua et al., 2019; Saxton et al., 2019; Miao et al., 2020; Amini et al., 2019; Hendrycks et al., 2021; Patel et al., 2021) have been proposed to increase the difficulty, diversity or even adversarial robustness.”