Injecting Numerical Reasoning Skills into Language Models

By Mor Geva, Ankit Gupta, Jonathan Berant. 2020

248 citations recorded by the source corpus.

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 –> 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “(2021) for a survey), as well specialized methods to enhance their reasoning abilities (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Piękos et al., 2021; Lourie et al., 2021).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “(Geva et al., 2020) also proposes self-supervised training for improving mathematical abilities in language models.”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “Several other investigators have also pre-trained or fine-tuned neural models on synthetic data consisting of mathematical problems [3], [21], [22], although these approaches have generally used very large numbers of training examples.”
PAL: Program-aided Language Models (2022, background): “…decompose natural language problems into steps and perform simple arithmetic operations, their performance falls dramatically when dealing with complex arithmetic (Hendrycks et al., 2021; Madaan & Yazdanbakhsh, 2022) or large numbers (Geva et al., 2020; Nogueira et al., 2021; Qian et al., 2022).”
Measuring and Improving Consistency in Pretrained Language Models (2021, positive): “MLM Loss Since the consistency loss is different from the Cross-Entropy loss the PLM is trained on, we find it important to continue the MLM loss on text data, as was observed in previous work (Geva et al., 2020).”
Birds Have Four Legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models (2020, background): “Some works focus on incorporate algebra computation ability in PTLMs (Zou and Lu, 2019; Geva et al., 2020), thus making them able to answer math reasoning tasks such as MAWPS (Koncel-Kedziorski et al.”
Question Directed Graph Attention Network for Numerical Reasoning over Text (2020, methodology): “In addition to NAQANet, GenBERT (Geva et al., 2020) injects numerical skills into BERT by generating numerical data.”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, methodology): “Some alternatives to improve the extrapolation capabilities of neural models include augmenting pretraining corpora with numerical texts (Geva et al., 2020; Chu et al., 2020) or using scientiﬁc notation to represent numbers (Zhang et al., 2020).”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, methodology): “Similar to past work (Yogatama et al., 2019; Geva et al., 2020), to avoid “catastrophic forgetting” (Kirkpatrick et al.”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, background): “While this may yield noisy supervision, similar approaches have been successfully exploited to inject reasoning capability in neural models (Geva et al., 2020; Saxton et al., 2019); on a random sample of 100 such questions, a manual checking shows 24% noise with our heuristics.”
Representing Numbers in NLP: a Survey and a Vision (2021, methodology): “Character level tokenization outperforms subword level (Nogueira et al., 2021; Wallace et al., 2019; Geva et al., 2020).”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “…learning continues to revolutionize NLP tasks such as question answering and machine translation (Sutskever et al., 2014; Devlin et al., 2019), it has also made signiﬁcant strides in the ﬁeld of mathematical reasoning (Wang et al., 2017; Yang and Deng, 2019; Geva et al., 2020; Wei et al., 2022).”
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, methodology): “GenBERT (Geva et al., 2020) uses pre-trained language models to generate the numerical answer, which breaks the limitation of ﬁxed operators.”
GMAT: Global Memory Augmentation for Transformers (2020, methodology): “We create a pseudo-textual task that requires global mathematical reasoning by generating examples from a rule-based generator from [5].”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Some alternatives to improve the extrapolation capabilities of neural models include augmenting pretraining corpora with numerical texts (Geva et al., 2020; Chu et al., 2020) or using scientific notation to represent numbers (Zhang et al.”